Table of Contents - Performance Testing Guide

Author: Sravan Tadakamalla

1. INTRODUCTION TO PERFORMANCE TESTING...........................................................................................3

1.1 OBJECTIVE................................................................................................................................................................3
1.2 TYPES OF PERFORMANCE TESTING ..............................................................................................................................3
1.3 RISKS ADDRESSED VIA PERFORMANCE TESTING ............................................................................................................4
1.4 UNDERSTANDING VOLUME .........................................................................................................................................4
1.5 TEST TOOLS..............................................................................................................................................................4
1.6 TOOL IDENTIFICATION & EVALUATION..........................................................................................................................4
1.7 WHAT KIND OF SKILLS ARE REQUIRED TO INVOLVE IN THE PERFORMANCE TESTING..............................................................5
2. PERFORMANCE TEST STRATEGY...................................................................................................................5
2.1 EVALUATE SYSTEMS TO IMPROVE PERFORMANCE TESTING EFFECTIVENESS ........................................................................5
2.2 DETERMINE PERFORMANCE TESTING OBJECTIVES, BUILDING THE TARGETS.......................................................................5
2.3 QUANTIFY END USERS RESPONSE TIME GOALS.............................................................................................................6
2.4 BASELINE APPLICATION PERFORMANCE ........................................................................................................................6
2.5 CONDUCT A PERFORMANCE TESTING RISK ASSESSMENT .................................................................................................6
3. ENVIRONMENT - PERFORMANCE TEST BEDS............................................................................................6
3.1 CREATING ................................................................................................................................................................6
3.2 TROUBLESHOOTING ...................................................................................................................................................6
3.3 REAL WORLD SIMULATION ..........................................................................................................................................7
3.4 SERVER CONFIGURATIONS............................................................................................................................................7
3.5 CLIENT CONFIGURATIONS............................................................................................................................................7
3.6 SCALABLE TEST ENVIRONMENTS...................................................................................................................................7
4. TEST EXECUTION.................................................................................................................................................8
4.1 TEST DESIGN AND EXECUTION....................................................................................................................................8
4.2 LOAD RUNNER EXECUTIONS.........................................................................................................................................8
4.3 REPORTING AND ANALYSIS..........................................................................................................................................8
4.4 DATA PRESENTATIONS.................................................................................................................................................8
4.4.1 Analysis Summary..........................................................................................................................................8
4.4.2 Statistics Summary.........................................................................................................................................9
4.4.3 Transaction Summary....................................................................................................................................9
4.4.4 HTTP Responses Summary............................................................................................................................9
4.5 STANDARD GRAPHS (PROVIDED BY DEFAULT) ..............................................................................................................9
5. DEFINING THE PERFORMANCE AND STRESS TESTING ..........................................................................9
5.1 MODELING THE USER EXPERIENCE...............................................................................................................................9
5.1.1 Simulate Realistic User Delays ....................................................................................................................9
5.1.2 Model Representative User Groups ............................................................................................................10
5.1.3 Simulate Realistic User Patterns ................................................................................................................10
5.1.4 How to use the transaction markers to evaluate performance....................................................................10
5.1.5 How to handle hidden fields........................................................................................................................10
 5.1.6 How to handle script and test failures.........................................................................................................10
5.1.7 Create Tests to Identify Points of Failure and Bottlenecks..........................................................................11
5.1.8 Create Tests to Optimize Critical User Actions...........................................................................................11
5.1.9 Performance Testing Across Load Balancers (clusters)..............................................................................11
5.1.10 Load Balancer - Why do I need load balancers?......................................................................................12
5.1.11 Scheduling and Balancing Methods...........................................................................................................12
5.1.12 Execute Tests to Tune Specific Components Across Network Tiers...........................................................12
6. HOW TO AVOID MISTAKES IN PERFORMANCE TESTING......................................................................12
6.1 UNSYSTEMATIC APPROACH.........................................................................................................................................12
6.2 PERFORMANCE TESTING BEYOND TEST EXECUTION.........................................................................................................13
6.3 TRANSLATE STAKEHOLDERS LANGUAGE INTO REAL PERFORMANCE GOALS AND REQUIREMENTS.........................................13
6.4 EMULATING PRODUCTION ENVIRONMENTS.....................................................................................................................13
6.5 HANDLE OUTLIERS IN PERFORMANCE TEST REPORTS....................................................................................................13
6.6 HANDLE PERFORMANCE DATA CORRECTLY AVOIDING OVER-AVERAGING.......................................................................13
7. STRESS TESTING WITH LOAD RUNNER......................................................................................................14
7.1 PERFORMANCE TESTING TAKING A PART IN SDLC........................................................................................................14
7.2 INSTALLING AND SET UP LOADRUNNER / BASICS .........................................................................................................15
7.3 CORE CONCEPTS.....................................................................................................................................................15
7.4 BENCHMARKING RUN / EXECUTION............................................................................................................................16
7.5 TEST DESIGN...........................................................................................................................................................16
7.6 RUNNING TEST........................................................................................................................................................16
7.7 HARDWARE SETUP...................................................................................................................................................17
7.8 PERFORMANCE ANALYSIS REPORT..............................................................................................................................17
7.9 PERFORMANCE COUNTERS..........................................................................................................................................17
7.10 PERFROMANCE METRICS.........................................................................................................................................19
8. DATA PRESENTATION........................................................................................................................................20
8.1 DATA PRESENTATION AT DIFFERENT LEVELS IN THE ORGANIZATION...................................................................................20
8.2 HOW TO ORGANIZE EFFICIENT DATA GRAPHS.................................................................................................................20
8.3 SUMMARIZE RESULTS ACROSS TESTS RUNS EFFICIENTLY ..............................................................................................20
8.4 USE DEGRADATION CURVES IN REPORTS ....................................................................................................................20
8.5 REPORT ABANDONMENT AND OTHER PERFORMANCE PROBLEMS ....................................................................................20
8.6 LOAD RUNNER SOLUTION AIDING WITH REPORTING ANALYSIS..........................................................................................20
9. PERFORMANCE TESTING FOR CAPACITY PLANNING AND SCALABILITY.....................................20
9.1 UNDERSTANDING THE ENVIRONMENT...........................................................................................................................20
9.2 PERFORMANCE TESTING TO AID IN CHECKING AVAILABILITY AND SUSTAINABILITY..............................................................21
9.3 EVALUATE THE TESTING INFRASTRUCTURE AGAINST THE FOOTPRINT..................................................................................22
9.4 MANUAL TESTING IS PROBLEMATIC............................................................................................................................22
10. BEST PRACTICES - PERFORMANCE TESTING ........................................................................................22
10.1 PERFORMANCE TESTING ACTIVITIES.........................................................................................................................23
10.2 IDENTIFY THE TEST ENVIRONMENT............................................................................................................................23
10.3 IDENTIFY PERFORMANCE ACCEPTANCE CRITERIA........................................................................................................23
10.4 PLAN AND DESIGN TESTS.........................................................................................................................................24
10.5 CONFIGURE THE TEST ENVIRONMENT.........................................................................................................................24
10.6 IMPLEMENT THE TEST DESIGN...................................................................................................................................24
10.7 EXECUTE THE TEST................................................................................................................................................24
10.8 ANALYZE RESULTS, REPORT, AND RETEST................................................................................................................24
1.INTRODUCTION TO PERFORMANCE TESTING

The performance testing is a measure of the performance characteristics of an application. The main
objective of a performance testing is to demonstrate that the system functions to specification with
acceptable response times while processing the required transaction volumes in real-time production
database. It’s defined as the technical investigation done to determine or validate the speed, scalability,
and/or stability characteristics of the product under test and also Performance-related activities, such as
testing and tuning, are concerned with achieving response times, throughput, and resource-utilization
levels that meet the performance objectives for the application under test.

1.1Objective

The objective of a performance test is to demonstrate that the system meets requirements for transaction
throughput and response times simultaneously.

The main deliverables from such a test, prior to execution, are automated test scripts and an
infrastructure to be used to execute automated tests for extended periods. This infrastructure is an asset
and an expensive one too, so it pays to make as much use of this infrastructure as possible. Fortunately,
this infrastructure is a test bed, which can be re-used for other tests with broader objectives. A
comprehensive test strategy would define a test infrastructure to enable all these objectives be met.

The performance testing goals are:

• End-to-end transaction response time measurements

• Measure Application Server components performance under various loads
• Measure database components performance under various loads
• Monitor system resources under various loads.
• Measure the network delay between the server and clients

1.2Types of Performance Testing

Performance Testing, Load Testing, Stress Testing, Spike Testing and Endurance Testing (Soak Testing)

Performance Testing is the process of determining the speed or effectiveness of a computer, network or
software program or device. This process can involve quantitative tests done in a lab, such as measuring
the response time or the number of MIPS (millions of instructions per second) at which a system
functions. Qualitative attributes such as reliability, scalability and interoperability may also be evaluated.
Performance testing is often done in conjunction with stress testing.

Stress testing is a form of testing that is used to determine the stability of a given system or entity. It
involves testing beyond normal operational capacity, often to a breaking point, in order to observe the
results. It refers to tests that put a greater emphasis on robustness, availability, and error handling under
a heavy load, rather than on what would be considered correct behavior under normal circumstances. In
particular, the goals of such tests may be to ensure the software doesn't crash in conditions of insufficient
computational resources (such as memory or disk space), unusually high concurrency, or denial of
service attacks.

Spike testing suggests to be done by spiking the number of users and understanding the behavior of the
application whether it will go down or will it is able to handle dramatic changes in load.

Endurance Testing (Soak Testing) is usually done to determine if the application can sustain the
continuous expected load. Generally this test is done to determine if there are any memory leaks in the
application.
1.3Risks Addressed via Performance Testing
Speed, Scalability, Availability and Recoverability, as they relate to Performance Testing

Scalability Testing is the testing of a software application for measuring its capability to scale up or scale
out in terms of any of its non-functional capability - be it the user load supported, the number of
transactions, the data volume etc

Availability Testing is the testing that need to check the application availability 24/7 at any point of time,
generally this will be done in production environment using tools called Site scope, BAC and Tivoli.

Recoverability Testing, is a disaster recovery testing majorly done in database perspective and in Load
balancing also. Recoverability means that committed transactions have not read data written by aborted
transactions (whose effects do not exist in the resulting database states).

1.4Understanding Volume
This section will discuss about the User Sessions, Session Duration, Transactions and User
Abandonment as they relate to volume in performance testing.
User Sessions :
For example a login session is the period of activity between a user logging in and logging out of a (multi-
user) system.

Session Duration is an Average amount of time that visitors spend on the site each time they visit the
pages in any web site.

Transaction is a movement carried out between separate entities or objects or functionalities, often
involving the exchange of items of value, such as information, services.

1.5Test tools
Load runner
Performance Center
WAPT
VSTS
Open STA
Rational Performance Tester
We have various tools available in industry to find out the causes for slow performance in the following
areas:
Application
Database
Network
Client side processing
Load balancer

1.6Tool Identification & Evaluation

The Performance testing tool identification and evaluation process would be carried out based on the
performance test requirements, protocols, software and hardware used in the application. A POC (Proof
Of Concept) would be carried out if required. The objective of this activity is to ensure that the tools
identified support all the applications used in the solution and helps in measuring the performance test
goals.

The three major components of functionality to execute a test are:

• Running of the Client application
• Actual Load Generation
• Resource Monitoring
 •

The Summary of Performance test tools identification and evaluation

Task Input Tools & Out Put

Techniques
Tools Identification • Test Requirement Proof Of Concept Tools identified
• Test Environment

• Application’s scripting.
• Tools vendor’s Information

1.7What kind of skills are required to involve in the Performance Testing.

Performance testing requires more skills than just knowledge about how to create a script for a particular
load testing tool.

What is going on within the system? Monitoring and Performance Analysis.

You got results. What if? Modeling and Capacity Planning.
We see the bottleneck. What to do? Tuning and System Performance Engineering.
And write, present, communicate, organize all the time.

2.PERFORMANCE TEST STRATEGY

2.1Evaluate Systems to Improve Performance Testing Effectiveness

While System Evaluation is a continual process throughout the performance testing effort, the bulk of the
evaluation process is most valuable if conducted very early in the effort. The evaluation can be thought of
as the evaluation of the project and system context. The intent is to collect information about the project
as a whole, the functions of the system, the expected user activities, the system architecture and any
other details that are helpful in creating a Performance Testing Strategy specific to the needs of the
particular project. Starting with this information, the performance goals and requirements can be more
efficiently collected and/or determined then validated with project stakeholders. The information collected
via System Evaluation is additionally invaluable when characterizing the workload and assessing project
and system risks. Simultaneously we need to learn techniques effectively and efficiently determine and
document the system's functions, document expected user activities and document the system's logical
and physical architecture.

2.2Determine Performance Testing Objectives, Building the Targets

Performance testing objectives are fairly easy to capture. The easiest way to capture performance testing
objectives is simply to ask each and every member of the team what value you can add for him or her
while you are doing performance testing. That value may be providing resource utilization data under
load, generating specific loads to assist with tuning an application server, or providing a report of the
number of objects requested by each web page. While collecting performance testing objectives early in
the project is a good habit to get into, so is periodically revisiting them and checking in with members of
the team to see if there are any new objectives that they would like to see added. And the majorly the
performance testing are as follows

• Establish valuable performance testing objectives at any point in the development lifecycle
• Communicate those objectives and the value they add to both team members and executives
• Establish technical, performance related targets (sometimes called performance budgets) that
can be validated independently from end-user goals and requirements
• Communicate those targets and the value that testing against them provides to both team
members and executives
2.3Quantify End Users Response Time Goals

Determining and quantifying application performance requirements and goals accurately is a critical
component of valuable performance testing. Successfully verbalizing our application’s performance
requirements and goals is the first and most important step in this process. Remember that when all is
said and done, there is only one performance requirement that really matters: those application users are
not annoyed or frustrated by poor performance. The users of your application don’t know or care what the
results of your performance tests are, how many seconds it takes something to display on the screen past
their threshold for "too long," or what your throughput is. The only thing application users know is that they
either notice that the application seems slow or they don't – and users will notice (or not) based on
anything from their mood to what they have become accustomed to. This How-To discusses methods for
converting these feelings into numbers, but never forgets to validate your quantification by putting the
application in front of real users. The major objectives in end user response time goals are follows

• Identify the difference between performance requirements and performance goals.

• Capture subjective performance requirements and goals.
• Quantify subjective performance requirements and goals.

2.4Baseline Application Performance

Base lining the application is where test execution actually begins. The intent is in twofold. First, all scripts
need to be executed, validated and debugged (if necessary). Second, various single and multi-user tests
are executed and recorded to provide a basis of comparison for all future testing. Initial baselines are
typically taken as soon as the test environment is available. Re-base lining occurs at each new release.
The following are the techniques to go forward in base line testing

• Design and execute single user baseline scenarios

• Design and execute multi-user baseline scenarios
• Report test results against baseline scenarios
• Determine when new baselines need to be taken

2.5Conduct a Performance Testing Risk Assessment

The performance tester is the only person on the project who has the information and experience to
assess project and business risks related to performance. Typical areas of risk include: unrealistic
acceptance criteria, unsuitable test environment, test schedule, test resources, limited access to real
users and a team-wide lack of understanding about when and how performance testing can add value to
the project. Identifying these risks early significantly increases the odds of success for the project by
allowing time for risk mitigation activities.
3.ENVIRONMENT - PERFORMANCE TEST BEDS
3.1Creating
Test Bed is an execution environment configured for software testing. It consists of specific Client/ Server
hardware, network topology, Client/Server Operating System, deployment of application under test with all
tiers, Installation of Performance Test tool (with valid license), other applications and other machines (if
required).
3.2Troubleshooting
Isolate the source of a problem and fix it, typically through a process of elimination whereby possible
sources of the problem are investigated and eliminated.

Troubleshooting involves the following:

1. Identify the symptom(s).

 • Is an error message displayed on your screen or written to the log?
• Is behavior abnormal?
• Is the symptom transient or persistent and can you reproduce it?

2. Locate the problem.

• Is it hardware or software?
• Can the problem source be identified easily?
• Can you bypass the problem?

3. Find the cause.

• Has your system recently been re-configured?

• Have you installed a new component (hardware or software)?
• Has the problem just occurred or has it existed for some time?

4. Prepare and apply a fix.

• Have you prepared the fix?

• Have you tested the fix?
• Can you rollback the fix?

3.3Real world simulation

Simulate Test Bed with real world environment.

It includes the following:

• Network bandwidth Usage (Customized / Maximum bandwidth)

• Normal / Peak User load
• Which business transaction needs to be included?
• Workload Models

3.4Server configurations
This Section includes the Hardware Configuration of Application Server / Web Server / Database Server.

Example:
• Processor
• Virtual Memory
• Physical Memory
• LAN Card

3.5Client Configurations

This Section includes the Hardware Configuration of Client machine where VUsers will be emulated (Load
Generator/ Controller)
3.6Scalable test environments

Once we’ve decided what servers and clients will be needed on your test network, next you need to
decide how many physical machines you need for the test lab. We can save money by creating multiple
servers on one physical machine, using virtualization software such as Microsoft Virtual PC/Virtual Server.
This is an especially scalable solution because it allows us to spend less money on hardware, and you
can add additional virtual servers by upgrading disk space and RAM, instead of buying complete
machines to emulate each new server that you add to your productivity network.

4.TEST EXECUTION
4.1Test Design and Execution
Based on the test strategy detailed test scenarios would be prepared. During the test design period the
following activities will be carried out:

• Scenario design
• Detailed test execution plan
• Dedicated test environment setup
• Script Recording/ Programming
• Script Customization (Delay, Checkpoints, Synchronizations points)
• Data Generation
• Parameterization/ Data pooling
4.2Load runner executions
When Scenario is designed that includes Business Transaction Script, Virtual User load, Load Generators
(if any), Ramp Up/ Ramp Down, Test Duration, Client/ Server Resource Measurements (Objects and
Counters), we can plan for test execution.

During Test Execution, monitor essential online graphs (Transaction Response Time, Hits per Second,
Throughput, Passed/Failed Transactions, Error messages (if any)).

4.3Reporting and Analysis

After Completion of Test Execution, Load Runner Analysis tool can display data points from a run as a
time series graph, for example Average Transaction Response Time. These graphs are one of the
methods the Analysis tool uses to summarize the vast amount of data collected during a test run. Creating
a summary in a standard form, the graph, removes the need to plough through all of the raw data.

The first step when the Analysis Tool time series graphs are being prepared is to generate the values that
can be seen in the Graph Data sheet. This is done by dividing the graph time span into slots and taking
the mean of the raw values falling within the slot as the Graph Data value for that slot. The duration of the
slots in the Graph Data is referred to as the Granularity of the graph. This can be set to be a number of
seconds, minutes or hours. Shorter periods provide more detail, longer ones more of an Overview.
4.4Data Presentations

Represent the Final Report (Word / HTML) with the following:

4.4.1Analysis Summary
4.4.1.1Header Time Range
This date is, by default, in the European format of dd/mm/yy (30/8/2004 for
August 30, 2004).
4.4.1.2Scenario Name:
The file path to the .lrs file
4.4.1.3Results in Session:
The file path to the .lrr file
 4.4.2Statistics Summary
4.4.2.1Maximum Running Vusers
This number is usually smaller than the number of VUsers specified in run-time
parameters because of ramp-up time and processing delays.
4.4.2.2Total Throughput (bytes):
Dividing this by the amount of time during the test run yields the next number:
4.4.2.3Average Throughput (bytes/second):
This could be shown as a straight horizontal line in the Throughput graph.
4.4.2.4Total Hits:
Dividing this by the amount of time during the test run yields the next number:
4.4.2.5Average Hits per Second:
This could be shown as a straight horizontal line in the Hits per Second graph.

4.4.3Transaction Summary
4.4.3.1Transactions
Total Passed is the total of the Pass column. The number of transactions
Passed and Failed is the total count of every action defined in the script,
multiplied by the number of VUsers, further multiplied by the number of
repetitions, and also multiplied by the number of iterations.

Transactions Min Avg Max

Transaction Name This is the Fastest This is the arithmetic This is the
time mean slowest time

4.4.4HTTP Responses Summary

HTTP 200 ("OK") is considered successful.
HTTP 302 highlights a redirection ("Moved Temporarily"), a normal event.
HTTP 404 is a "resource not found" error.
HTTP 500 is a "server busy" error
HTTP 503 is an authentication denial
4.5Standard Graphs (Provided by Default)
These graphs are sequenced according to the sequence in Controller menu option Graph
> Add Graph... During Analysis this order is automatically applied from our custom
Template.

• Vusers
• Transactions
• Web Resources
• System Resources

5.DEFINING THE PERFORMANCE AND STRESS TESTING

5.1Modeling the User experience
5.1.1Simulate Realistic User Delays

Application users think, read, and type at different speeds, and it's the performance tester's job to figure
out how to model and script those varying speeds as part the testing process. This How-To will explain all
of the necessary theory about determining and scripting realistic user delays.
 • Realistic user delays are important to test results
• Determine realistic durations and distribution patters for user delay times
• Incorporate realistic user delays into test designs and test scripts
5.1.2Model Representative User Groups
Modeling a user community has some special considerations in addition to those for modeling individual
users. This How-To demonstrates and how to develop user community models that realistically represent
the usage of the application by focusing on groups of users and how they interact from the perspective of
the application.

• Recognize and select representative groups of users

• Model groups of users with appropriate variance
• Consolidate groups of users into one or more production simulation models
• Identify and model special considerations when blending groups of users into single models
5.1.3Simulate Realistic User Patterns
Applications typically allow for many ways to accomplish a task or many methods to navigate the
application. While users think this is a good thing (at least most of the time), this complicates matters for
testers, requiring that we look, not only at what users do within an application but also at how they do it.
To effectively predict performance in production, these variants of individual user patterns must be
accounted for. This How-To discusses how to apply these individual user patterns to groups or
communities of users.

• Identify individual usage scenarios

• Account for variances in individual usage scenarios
• Incorporate individual usage scenarios and their variances into user groups
• Script individual usage scenarios with their variances.

5.1.4How to use the transaction markers to evaluate performance

It’s very much important about the inserted transactions and think times or timers and delays in between
the script. We should mimic the appropriate think and delays as per the end user experiences and
application behavior. Once completion of the executions we need to exclude the thinks and delays time
from respective response times.

• Identify individual transactions and should be proper naming convention

• Identify the think times and delay time in between the transactions and iterations.
• Incorporate all the think times and delays in individual usage scenarios and their variances into
user groups

5.1.5How to handle hidden fields

Generally, hidden values will generated in every application but some time it’s very important to handle
the hidden and dynamic values. For example in .net application “VIEW STATE” values and in Java
application “J SESSION” VALUES and so on.

• Identify the hidden values that may cause errors in scripts and executions.
• Incorporate proper functions as per the tools were it required to handle and hidden values.
• If required create re-usable actions or classes in virtual user generator scripts.

5.1.6How to handle script and test failures

There might be chances for errors in scripts after doing all required enhancements; we need to add the
error handling in all individual scripts. This is the best practice to trouble shoot and nail down the issue in
environment and application.
5.1.7Create Tests to Identify Points of Failure and Bottlenecks
There are at least as many ways of identifying bottleneck suspects as there are people who’ve ever
observed something being slow when working on a computer. It’s our job as performance testers to
identify as many of those suspects as possible and then sort, categorize, verify, test, exploit, and
potentially help resolve them. This How-To demonstrates some common methods to create tests that
identify bottlenecks.

• Design bottleneck exploiting tests by eliminating ancillary activities

• Design bottleneck exploiting tests by modifying test data
• Design bottleneck exploiting tests by adding related activities
• Design bottleneck exploiting tests by changing navigation paths

5.1.8Create Tests to Optimize Critical User Actions

We use tools to determine things like “The Submit loan Page takes a REALLY long time to load when we
do more than 500 submits in an hour.” While that is extremely valuable information, it’s only mildly useful
to the person who needs to resolve the issue. The person responsible for providing a resolution needs to
know what part of that page, or what process on a back end server, or which table in the database is the
cause of the issue before we can start trying to find a resolution for it.

• Capture metrics by tier

• Design tests to find breakpoints
• Design tests to find resource constraints
• Employ unit tests for performance optimization

5.1.9Performance Testing Across Load Balancers (clusters).

It’s a “FAIL OVER TESTING” across the Load balancers in real time environment. All expected load
generation will be generated through the load balancer. Load balancers and networks shouldn’t actually
be Causing performance problems or bottlenecks, but if they are, some configuration changes will usually
remedy the problem. Load balancers are conceptually quite simple. They take the incoming load of client
requests and distribute that load across multiple server resources. When configured correctly, a load
balancer rarely causes a performance problem. The only way to ensure that a load balancer is configured
properly is to test it under load before it’s put into use in the production system. The bottom line is that if
the load balancer isn’t speeding up our site or increasing the volume it can handle, it’s not doing its job
properly and needs to be reconfigured.
Physical architecture with load balancer

5.1.10Load Balancer - Why do I need load balancers?

Because of availability as well as performance, we want to achieve high-availability. Maintenance is no
longer a nightmare and does not require downtime.

5.1.11Scheduling and Balancing Methods

• Round Robin
• Weighted Round Robin
• Least Connection
• Weighted Least Connection
• Agent-based Adaptive
• Chained Failover (Fixed Weighting)
• Layer 7 Content Switching

5.1.12Execute Tests to Tune Specific Components Across Network Tiers

Use the third party tools to do network simulation testing while doing performance testing. We can do
performance across the network tiers by using different tool in industry.

6.HOW TO AVOID MISTAKES IN PERFORMANCE TESTING

6.1Unsystematic approach
• Decisions with out Running of the Client application
• Un acceptable Load Generation
• Not using Proper Monitoring and comparisons.
• No Goals
• No general purpose model
• Goals =>Techniques, Metrics, Workload
• Not trivial
• Biased Goals
• ‘To show that OUR system is better than THEIRS”
 • Analysts = Jury
• Unsystematic Approach
• Analysis without Understanding the Problem
• Incorrect Performance Metrics
• Unrepresentative Workload
• Wrong Evaluation Technique
• Overlook Important Parameters
• Ignore Significant Factors
• Inappropriate Experimental Design
• Inappropriate Level of Detail
• No Analysis
• Erroneous Analysis
• No Sensitivity Analysis
• Ignoring Errors in Input
• Improper Treatment of Outliers
• Assuming No Change in the Future
• Ignoring Variability
• Too Complex Analysis
• Improper Presentation of Results
• Ignoring Social Aspects
• Omitting Assumptions and Limitations

6.2Performance testing beyond test execution

We probably need to know about tuning and have understanding of how applications should be designed
to perform well (Software Performance Engineering). We don't need to be an expert, for example, in
database tuning – most companies have DBAs for that – but you need to speak to a DBA in his language
to coordinate efforts effectively. Or raise concerns about performance consequences of the current
application design. Unfortunately it is not easy – you need to know enough to understand what is going on
and communicate effectively.

6.3Translate Stakeholders Language into Real Performance Goals and Requirements

Translate the stake holder’s requirements in to SLAs example: - Response times

6.4Emulating production environments

Creating pilot environments as it is like the production environment. By doing the performance testing on
pilot environment will give us the results and expected behavior.

6.5Handle Outliers in Performance Test Reports

Handling outlier is an observation that is numerically distant from the rest of the data. Statistics derived
from data sets that include outliers may be misleading in terms of response times or 90% tile. Example: -
Generally if we have any failed transactions due to time outs errors, we need to find it out how this time
out occurred in our execution. Is it because of processor queue length? Or running out memory in web
server or application server? Especially we need to display graphs in our reports for these of issues in the
environment. Some time due to network outage we may see some spike in while executing the tests.
6.6Handle Performance Data Correctly Avoiding Over-Averaging
Base lining the application is where test execution actually begins. First, all scripts need to be executed,
validated and debugged (if necessary). Create a baseline for a single user scenario and for backend /
batch process scenarios. While executing tests, evaluate results, identify the baseline, exploit potential
vulnerabilities and performance threats. Perform an application performance walkthrough. Analyze first-
page load time to evaluate client-side performance. Perform network analysis for a single user and tune
the configuration against goals and constraints.

7.STRESS TESTING WITH LOAD RUNNER

7.1Performance testing taking a part in SDLC
7.2Installing and Set up Loadrunner / Basics

7.3Core Concepts
VU – generator - Virtual User Generator (Vugen) –records Vusers scripts that emulate the steps of real
users using the application.
Parameterization - Also known as Application Data and the data is resident in the application’s
database–Examples: ID numbers and passwords
Correlation – also known as user generated data or dynamic values. Originates with the user–Examples:
new unique ID or email address or session ids
Controllers - The Controller is an administrative center for creating, maintaining, and executing
Scenarios. The Controller assigns Vusers and load generators to Scenarios, starts and stops load tests,
and performs other administrative tasks.
Load generators - (also known as hosts) are used to run the Vusers that generate load on the
application under test.
LR Analysis - uses the load test results to create graphs and reports that are used to correlate system
information, identify bottlenecks, and performance issues.
Monitors –
7.4Benchmarking Run / Execution

To validate that there is enough test hardware available in the test environment, benchmark the business
processes against the testing hardware. Take a business process and execute a small number of users
against the application.

• Validates the functionality of the business process

• Potentially exposes unknown data dependencies

7.5Test design
Based on the test strategy detailed test scenarios would be prepared. During the test design period the
following activities will be carried out:

• Scenario design
• Detailed test execution plan
• Dedicated test environment setup
• Script Recording/ Programming
• Script Customization (Delay, Checkpoints, Synchronizations points)
• Data Generation
• Parameterization/ Data pooling

7.6Running Test

The test execution will follow the various types of test as identified in the test plan. All the scenarios
identified will be executed. Virtual user loads are simulated based on the usage pattern and load levels
applied as stated in the performance test strategy.

The following artifacts will be produced during test execution period:

• Test logs
 • Test Result

7.7Hardware Setup
Minimum requirements should be PIII machine and every virtual user will utilize 1 MB of memory.

7.8Performance Analysis Report

The test logs and results generated are analyzed based on Performance under various load,
Transaction/second, database throughput, Network throughput, Think time, Network delay, Resource
usage, Transaction Distribution and Data handling. Manual and automated results analysis methods can
be used for performance results analysis

The following performance test reports/ graphs can be generated as part of performance testing:-

• Transaction Response time

• Transactions per Second
• Transaction Summary graph
• Transaction performance Summary graph
• Transaction Response graph – Under load graph
• Virtual user Summary graph
• Error Statistics graph
• Hits per second graph
• Throughput graph
• Down load per second graph

Based on the Performance report analysis, suggestions on improvement or tuning will be provided to the
design team:

• Performance improvements to application software, middleware, database organization.

• Changes to server system parameters.
• Upgrades to client or server hardware, network capacity or routing.

7.9Performance counters

The following measurements are most commonly used when monitoring the Oracle server

Oracle server

Measurement Description

CPU used by this session
The amount of CPU time (in 10s of milliseconds) used by a session between the time a user call
started and ended. Some user calls can be completed within 10 milliseconds and, as a result, the
start and end-user call time can be the same. In this case, 0 milliseconds are added to the statistic. A
similar problem can exist in the operating system reporting, especially on systems that suffer from
many context switches.

Bytes received via The total number of bytes received from the client over Net8.
SQL*Net from client

Logons current The total number of current logons

Opens of replaced files The total number of files that needed to be reopened because they were no longer in the process file
 cache.

User calls Oracle allocates resources (Call State Objects) to keep track of relevant user call data structures
every time you log in, parse, or execute. When determining activity, the ratio of user calls to RPI calls
gives you an indication of how much internal work is generated as a result of the type of requests the
user is sending to Oracle.

SQL*Net roundtrips The total number of Net8 messages sent to, and received from, the client.
to/from client

Bytes sent via SQL*Net to The total number of bytes sent to the client from the foreground process(es).
client

Opened cursors current The total number of current open cursors.

DB block changes
Closely related to consistent changes, this statistic counts the total number of changes that were
made to all blocks in the SGA that were part of an update or delete operation. These are changes
that generate redo log entries and hence will cause permanent changes to the database if the
transaction is committed. This statistic is a rough indication of total database work and indicates
(possibly on a per-transaction level) the rate at which buffers are being dirtied.

Total file opens The total number of file opens being performed by the instance. Each process needs a number of
files (control file, log file, database file) to work against the database.

Web Server Metrics

1. Request Execution Time (ASP.NET)

2. Request Wait Time (ASP .NET)
3. Requests Current (ASP.NET)
4. Requests Queued (ASP.NET)
5. Request Bytes In Total (ASP.NET Applications__Total__)
6. Request Bytes Out Total (ASP.NET Applications__Total__)
7. Requests Executing (ASP.NET Applications__Total__)
8. Requests In Application Queue (ASP.NET Applications__Total__)
9. Available MBytes (Memory)
10. Page Faults/sec (Memory)
11. Pages/sec (Memory)
12. Pool Nonpaged Bytes (Memory)
13. Threads (Objects)
14. %Disk Time (PhysicalDisk_Total)
15. Page File Bytes (Process_Total)
16. Private Bytes (Process_Total)
17. Virtual Bytes (Process_Total)
18. *%Interrupt Time (Processor_Total)
19. *%Privileged Time (Processor_Total)
20. %Processor Time (Processor_Total)
21. Interrupts/sec (Processor_Total)
22. Pool Nonpaged Failures (Server)
23. File Data Operations/sec (System)
24. Processor Queue Length (System)
25. Context Switches/sec (Thread_Total)
26. %Disk Time (PhysicalDisk_Total)
27. %Processor Time (Processor_Total)
28. File Data Operations/sec (System)
29. Interrupts/sec (Processor_Total)
30. Page Faults/sec (Memory)
 31. Pages/sec (Memory)
32. Pool Nonpaged Bytes (Memory)
33. Private Bytes (Process_Total)
34. Processor Queue Length (System)
35. Threads (Objects)

Database Server Metrics

1. Concurrency Wait Time
2. Consistent Gets
3. Commit SCN cached
4. CPU Used By This Session (Sesstat)
5. CPU Used When Call Started
6. Db Block Changes
7. Db Block Gets
8. Db Block Gets from cache
9. Dirty buffers inspected
10. Logons Current
11. Opened Cursors Cumulative
12. Opened Cursors Current
13. Background Timeouts
14. Buffer Is Pinned Count
15. Buffer Is Not Pinned Count
16. Parse Count(Hard)
17. Parse Count(Total)
18. Parse Time Cpu
19. Parse Time Elapsed
20. Physical Reads
21. Physical Read IO Request
22. Physical Writes
23. Physical Write IO Request
24. Queries Parallelised
25. Redo Size
26. Sorts (Memory)
27. Sorts (Rows)
28. Table Fetch By Rowid
29. Table Fetch Continued Row
30. User Calls
31. User Commits
32. Work Area Memory Allocated
33. Available MBytes (Memory)

7.10Perfromance Metrics
The Common Metrics selected /used during the performance testing is as below Response time

Turnaround time = the time between the submission of a batch job and the completion of its output.

Stretch Factor The ratio of the response time with single user to that of concurrent users.

Throughput Rate (requests per unit of time) Examples:

• Jobs per second
• Requests per second
• Millions of Instructions Per Second (MIPS)
• Millions of Floating Point Operations Per Second (MFLOPS)
• Packets per Second (PPS)
• Bits per second (bps)
 • Transactions per Second (TPS)

Capacity:

• Nominal Capacity: Maximum achievable throughput under ideal workload conditions. E.g.,
bandwidth in bits per second. The response time at maximum throughput is too high.
• Usable capacity: Maximum throughput achievable without exceeding a pre-specified response-
time limit
• Efficiency: Ratio usable capacity to nominal capacity. Or, the ratio of the performance of an n-
processor system to that of a one-processor system is its efficiency.
• Utilization: The fraction of time the resource is busy servicing requests.
• Average Fraction used for memory.

8.DATA PRESENTATION
8.1Data Presentation at different levels in the organization
Data presentation will get differ when we are participating in the execution meeting. The execution data /
analysis data should be with low granularity and with out think times or delay times. The data report
should talk about the individual transactions counts and window resource utilization from the entire web,
app and if we have any external service related and shared services data.
8.2How to organize efficient data graphs
It’s a very good to organize the data that need to be very easy to understand, so we need to correlate the
graphs properly with required graphs information’s. Example: - Through put, hits per second, average
transaction response times and if we have any spike in the environment.
8.3Summarize Results Across Tests Runs efficiently
Need to provide the brief summary for the all graph after every execution, so it will be easy to compare
the results with previous executions. The best practice is to create reports in form of “TEMPLATE” and
then apply the “TEAMPLATES” when ever we require after every execution.
8.4Use Degradation Curves in Reports
It’s very important, our data graphs should display the degradations curve graphs in proper manner and
need to provide exact summary for the respective spikes. If we have degradation in the graphs then we
need to do the root cause analysis and help the developer where exactly the problem occurs.
8.5Report Abandonment and Other Performance Problems
Performance problem might be differ from execution to execution because there might be a lot number of
issue across environment in terms of issues in web server and application server configuration file,
network, virtual IP’s, load balancers, external services, shared services and fire wall.

8.6Load Runner solution aiding with reporting analysis

Load runner will provide full fledged reporting for analysis after every execution either in crystal reports,
HTML reports or Microsoft word reports.

9.PERFORMANCE TESTING FOR CAPACITY PLANNING AND SCALABILITY

9.1Understanding the environment

9.2Performance testing to aid in checking Availability and sustainability
• Allows data to be written, read, and destroyed without affecting production users
• Allows test system to be rebooted during test runs without affecting production users
• Should mirror the production system.
• Needs business processes that are functioning correctly
• Should include Benchmark runs
• Must contain sufficient hardware to generate the load test.
9.3Evaluate the testing infrastructure against the footprint.
• Do I have enough hardware to generate the user load?
• Do I have enough memory?
• Do I have enough CPUs?
9.4Manual Testing is Problematic

10.BEST PRACTICES - PERFORMANCE TESTING

Figure: - Performance Best Practices
10.1Performance Testing Activities
• Identify the test environment
• Identify performance acceptance criteria
• Plan and design tests
• Configure the test environment
• Implement the test design
• Execute the test
• Analyze results, report, and retest

10.2Identify the test environment

• Identify the physical test environment and the production environment as well as the tools and
resources available to the test team.
• The physical environment includes hardware, software, and network configurations.
• Having a thorough understanding of the entire test environment at the outset enables more
efficient test design and planning and helps you identify testing challenges early in the project.
• In some situations, this process must be revisited periodically throughout the project’s life cycle.

10.3Identify Performance Acceptance Criteria

• Identify the response time, throughput, and resource utilization goals and constraints.
• In general, response time is a user concern, throughput is a business concern, and resource
utilization is a system concern.
• Additionally, identify project success criteria that may not be captured by those goals and
constraints; for example, using performance tests to evaluate what combination of configuration
settings will result in the most desirable performance characteristics.
10.4Plan and design tests
• Identify key scenarios, determine variability among representative users and how to simulate that
variability, define test data, and establish metrics to be collected.
• Consolidate this information into one or more models of system usage to be implemented,
executed, and analyzed.

10.5Configure the test environment

• Prepare the test environment, tools, and resources necessary to execute each strategy as
features and components become available for test.
• Ensure that the test environment is instrumented for resource monitoring as necessary.
10.6Implement the test design
• Develop the performance tests in accordance with the test design.
10.7Execute the Test
• Run and monitor your tests.
• Validate the tests, test data, and results collection.
• Execute validated tests for analysis while monitoring the test and the test environment.
10.8Analyze Results, Report, and Retest
• Consolidate and share results data.
• Analyze the data both individually and as a cross functional team.
• Reprioritize the remaining tests and re-execute them as needed.
• When all of the metric values are within accepted limits, none of the set thresholds have been
violated, and all of the desired information has been collected, you have finished testing that
particular scenario on that particular configuration.