Atlanta Local User Group 6/17/03

Testing PeopleSoft & Oracle Apps: Strategies and Techniques

Dan Downing, VP Testing Services

MENTORA GROUP
Atlanta Boston DC 404.250.6515 www.mentora.com

Objectives
Present unique challenges of testing major packaged apps, w/ examples from:
PeopleSoft7 load test project Oracle Apps 11i functional test project

Things you will learn:

Strategies and techniques for planning and executing functional and load testing projects with WinRunner, LoadRunner, TestDirector

Copyright 2003 Mentora Group Inc.

Application Challenges
Vendor reliance
You dont own the code, so you cant talk to the developers Often middleware configuration & tuning support is limited Big company Help Desk challenges (youre not on the latest release!)

Application complexity:
Numerous -- Financials: 5-10; Manufacturing: 12+; HR: 10+; CRM: 6-10 Your companys implementation & workflows are unique

Application Quality
Code is notoriously buggy Patches/upgrades can number into the hundreds per quarter!

Technology stack complexity

Multi-tiered, lots of middleware Often use proprietary protocols, specific client software

Copyright 2003 Mentora Group Inc.

Testing Challenges
Test Environment
Lots of hardware, storage, test server(s); multiple environments Often need a Citrix server to test the app as users see it Copy of production database often very large; takes time to create Need dual environments for upgrade testing

Test Planning/Development
Ensuring good test coverage requires experience Functional testing requires hundred of scripts Load testing requires understanding usage profile, calculating concurrent usage Upgrade testing requires 2 script versions when functionality changes Reliance on business users for workflow definition, execution review

Test Execution
Reliance on business users for manual testing Ultimately only they really know how they really use the applications Interest in cooperating wanes quickly Deviations need to be triaged with vendor, worked-around internally
Copyright 2003 Mentora Group Inc.

Strategies
Application Knowledge
Bring in application experts; business users or consultants Plan to comprehensively document user workflows

Test Environment
Consider outsourcing to a vendor partner that understands the app environment

Test Planning
Categorize your testing & develop methods & project plans for different types Categorize test cases into automated or manual Validate your projected usage load profiles

Script Development
Involve business users in defining & validating scripts Budget time and resources on scripting effort, infrastructure support

Test Execution
Refine your deviation management process to include triaging with the vendor, develop business workarounds
Copyright 2003 Mentora Group Inc.

Planning: Functional Test

General
Make sure testers have hefty PCs (PIII-400, 1GB RAM) Apply time/cost/user intervention criteria to decide what to automate Automate test cases that are going to be oft-retested, implicitly

New implementations
Inventory business functions, prioritize with users Automate stable workflows, not ones still in flux

Upgrades
Aggregate test cases into macro business functions; e.g., Procure-toPay, Order-to-Cash Map key test cases for these macro-functions & automate these first Project plan should include test on release x.x, then test on x.y, with time for script re-work

Copyright 2003 Mentora Group Inc.

Planning: Load Test

Estimating load
Profile users by time zone and peak hour within time zone Determine how many transactions by type need to be processed in peak hour Baseline single transaction time for each type Compute number of virtual users required to achieve target transaction throughput (by function) and use this as a starting point for your load scenarios

Secure Transactions
Baseline without https transactions first Be prepared for web server upgrades with security on

Technical Team
Monitor as many hardware & middleware resources on each server as possible Involve DBA, Systems Engineer, app server specialists

Copyright 2003 Mentora Group Inc.

Techniques: Functional Test

Oracle Apps WinRunner scripting (11.5.7 and above)
Ensure unique forms object names by configuring System Profile to expose Developer Name (Forms 6i v.6.0.8.8 or higher; see http://metalink.oracle.com
Doc ID: 200163.1 )

Script repository
Use TestDirector to maintain scripts Link TD with source code control to maintain multiple script versions in sync with OA revs

Test Execution
Document test execution to ID data prep, expected results, script steps Use TD sequencing via tables to pass data from one script to another to link multiple scripts for more automated business process testing

Copyright 2003 Mentora Group Inc.

Techniques: Load Test

PeopleSoft 8 LoadRunner scripting
No gotchas goes very smoothly PS8 pure http

Execution Monitoring
Configure LR monitors to monitor
Servers: cpu, memory, io, bytes sent/rcvd WebLogic: Web connections, sessions, memory, db connection pool Oracle: table scans, disk sorts, deadlocks, cache hit ratio

If dont have Oracle monitors, use perfmon (windows) or vmstat, iostat, 3d party Oracle monitors (Spotlight, Oracle Enterprise Monitor, own v$ scripts)

Test Execution
Establish performance baseline by function Ramp load by function with all hands watching monitors; look for missing indexes causing slow queries Test with https turned off first, to gauge impact of secure transactions

Copyright 2003 Mentora Group Inc.

PeopleSoft 8 Load Test

Customer
Major US wireless service provider

Application
Upgrading to PeopleSoft8 all web HRIS (from client-server v7)

Unknowns
New 3-tier platform & application server software (WebLogic) Performance hit of security implementation (https, firewalls, proxy servers)

Success Factors
2-week Open Enrollment for 30,000 employees to update their benefits Outsourced Call Center of 30 CSRs to support process

Copyright 2003 Mentora Group Inc.

Customer PeopleSoft8 Production Environment

All Hardware: HP9000-8xx/HP-UX11
Intranet Web App
Intranet

Web & App Server: WebLogic 5.1 DB Server: Oracle 8.1.6

DB App
Secure Zone 2 (DMZ 2)

Web
Secure Zone 1 (DMZ 1)

Internet
Unsecured (Internet)

Cisco Load Balancing CSS 1 https

Intranet WWW Server 1

Intranet Application Server 1

Primary DB Server

Internet Application Server 1

Intermet WWW Server 1

Single IP/DNS

https
lo Fai
http s

ver

http

https

ly On

Intranet

Single Logical DNS/IP Address

One Way Shareplex Replication

Internet Requests

https

https

https Cisco Load Balancing CSS 2 Failover

Single IP/DNS

Intranet WWW Server 2

Intranet Application Server 2

Replication/Failover DB

Internet Application Server 2

Internet WWW Server 2

Load Test System

Copyright 2003 Mentora Group Inc.

Load Sizing
Employees enrolling from Home over internet Hour PT ET CT MT PT 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 53 17:00 53 5 18:00 53 5 59 19:00 53 5 59 29 20:00 5 59 29 21:00 59 29 22:00 29 23:00 29 Employees enrolling from work over intranet ET CT MT PT 4,239 364 4,684 2,294 Total ALL TZ 4,239 364 4,684 2,294 53 58 116 145 92 87 29 29

Employees by Time Zone

Script Time for a single iteration (minutes) 5 8

Employees per Peak Hour

# of actual users in peak hour 4684 35 Iterations per hour Arrival Rate Minimum # of iterating concurrent virtual users to drive peak load 390 35

Enroll Medical, Dependents Ben Admin functions

12 7.5

Even distribution Constant

Concurrent Virtual Users

Copyright 2003 Mentora Group Inc.

System Capacity
Up to 100 users, scalability is linear, w/ pages responding in less than 5 seconds From 100 to 220 users, scalability degrades steadily, to 38% degradation at 190 users Above 220 users, response declines sharply and the system fails.

At 190 users the slowest pages begin to exceed the acceptable response of 30 seconds, thus determining the systems capacity.

190 concurrent users max per app server

Copyright 2003 Mentora Group Inc.

Oracle Apps 11i Upgrade Regression Test

Customer
Global materials manufacturer

Application
Upgrading from Oracle Apps 11.5.5 to 11.5.7, Financials, & Manufacturing (discrete and process)

Unknowns
Quality of 11.5.7 Robustness of the upgrade process

Success Factors
Complete upgrade over the Thanksgiving weekend No business interruption for any of their 8 US & European plants Convince business users that we are moving toward selfreliance

Copyright 2003 Mentora Group Inc.

Oracle Apps Architecture

Portal Servlet Engine

Browser JInitiator

Web Listener

Java Server Pages (JSP) PL/SQL Gateway Discoverer Reports Forms Data Server

Concurrent Manager
Copyright 2003 Mentora Group Inc.

Customer OA Production Environment

HW: Sun 420/4500; 100GB Oracle8i DB
StreamSrv v3.0 1 Merkur HEMSTREAM02 (SUN 220)

Remote access via Citrix servers

Arm Re-Rate HEMMAN04 (Terminal Services) IE & Jinit IE & Jinit ADI Xport Client HEM-CITRIX1 (CITRIX) Subnet 100 (100Mbps)

HEMWEB01 (SUN 220)

HEMDISCOVER (SUN 420)

HEMFORMS01 (SUN 420)

HEM-CITRIX2 (CITRIX) IE & Jinit

HEMA HEMT HEMB EMC Clarion (Disk Array)

HEMDB01 (SUN 4500) Web Connect Xport Svr Milemaker HEMMAN02 (SUN 420)

IE & Jinit ADI HEM-CITRIX3 (CITRIX) HEM-CITRIX4 (CITRIX)

IE & Jinit

IE & Jinit

HEM-CITRIX5 (CITRIX)

HEM-CITRIX6 (CITRIX)

Carrier Quantum DLT 8000 IE & Jinit HEMMAN03 (SUN 220) Xport Client HEM-CITRIX7 (CITRIX) HEM-CITRIX8 (CITRIX) IE & Jinit

Qualstar 46-120 AIT3

HEMBACK (SUN 420) = Fiber Channel (x2) = Oracle 11i = Manugistics Last Updated: 01/15/03

Europe / Hamina
StreamSrv v3.0 1 Merkur HEMSTREAM03 (SUN 220)

IE & Jinit

IE & Jinit

IE & Jinit

HEM-CITRIX9 (CITRIX)

HEM-CITRIX10 (CITRIX)

HEM-CITRIX11 (CITRIX)

Copyright 2003 Mentora Group Inc.

Test Environment

Oracle Apps Test Instance

Sun 4500, 8-cpu 8GB RAM Solaris 2.8 Oracle8i 8.7.3 Oracle Forms Server

Mercury Test Server

Test Workstation

Dell 1550 1x1.13Ghz/1GB RAM Windows2000 Server TestDirector Standard 7.5

Intel PC, 512MB RAM J-Initiator & ADI WinRunner 7.5

Copyright 2003 Mentora Group Inc.

Project Overview
Test scripts Project plan
Description 1 2 3 Initiation Coverage Discovery Development Design Discovery 11.5.5 Development User review and finalization 11.5.7 development Execution 11.5.5 11.5.7 Closeout & Reporting Dur 3d 2w 8w Module Purchasing OPM Logistics/Receiving Discrete (MRO) Inventory General Ledger Accounts Payable Fixed Assets Project Accounting Cash Management 2w Order Management OM Shipping Execution OPM Inventory 2d Quality Accounts Receivable Order Management Total Scripts 20 3 4 16 28 12 20 4 23 13 7 13 30 23 193

Deviations
Deviation Type Resolved Informational Performance Open Total Number 37 33 1 18 89

Surfaced Surfaceda a showstopper showstopper functional functionalchange change that thatjustified justifiedthe the entire entireproject! project!

Copyright 2003 Mentora Group Inc.

Other Resources
Oracle Apps Testing white paper
http://www-svca.mercuryinteractive.com/solutions/enterprise_applications/oracle/ http://www.peoplesoft.com/corp/en/products/edu_gov/univ_curr/on_campus/curr_resource.jsp http://www.oracle.com/appsnet/technology/architecture/content.html

PeopleSoft technology white paper Oracle technology white papers/presentations PeopleSoft load testing, Oracle Apps testing, load testing methodology, common bottlenecks
http://www.mentora.com/library.htm

Copyright 2003 Mentora Group Inc.

Questions?

ddowning@mentora.com www.mentora.com

Copyright 2003 Mentora Group Inc.