Understanding the Impact of

Running WAN Emulation with Load

Testing

A Shunra Software White Paper

July 2, 2008
 Understanding the Impact of Running WAN Emulation with Load Testing
.

Introduction

Investment in pre-deployment performance testing has become a widely adopted standard

used to verify the performance and reliability characteristics of core business applications.
As reflected by the robust capabilities of industry-standard load testing tools such as HP
LoadRunner, the proper setup and execution of a load test is a complex undertaking.
Significant effort goes into each load testing project. Organizations that are disciplined
enough to properly implement performance testing into their application development
lifecycle have gained significant business benefits from finding issues and properly sizing
capacity prior to application roll-out.

Load testing has traditionally been focused on identification of software or hardware

capacity problems, without taking into account the impact of the network on application
performance. In some cases the network is not an impediment to application deployment.
Unfortunately, in many cases applications fail to perform to expected levels, despite load
testing prior to roll-out.

A load test environment that does not reflect all relevant real-world conditions can not
provide an accurate projection of how the application will perform once deployed. When
this happens, the analysis and decisions leading up to approval to deploy the application may
be misguided. Increasingly, many pre-deployment testing labs have found that it is critical to
incorporate the impact of WAN on application load testing. This white paper describes the
importance of assessing network impact on application performance and the value that is
obtained by integrating WAN emulation into load testing.

Organizations typically have used one of four approaches to assessing the impact of the
network:

Approach 1 – Ignore network impact entirely

Focus on the scalability of the datacenter infrastructure and target response time and
throughput targets based solely on local performance at the datacenter, ignoring the WAN.

Approach 2 – Deploy agents to generate load

Incorporate network impact on the application by deploying distributed agents for load
generation.

Approach 3 – Estimate impact of the network

Separately estimate the impact of the network on an application being deployed and
manually factor that into result reports and deployment readiness findings

Approach 4 – Incorporate WAN emulation

Incorporate WAN emulation as an integral part of application deployment automated testing

Page 2 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

There are problems inherent in the first three approaches, and benefits for many
organizations that are using the 4th approach, which includes a WAN emulation solution
within their load testing environment.

Weighing the Pro’s & Con’s

Approach 1 – Ignore network impact entirely

This has been a common practice, despite shortcomings that this paper highlights. Ignoring
the impact of the network poses a substantial risk. Applications may be deployed without
fully understanding their performance characteristics. Deployments may fail due to
unidentified scalability issues that were entirely overlooked by load testing focused on the
server infrastructure.

Approach 2 – Deploy agents to generate load

To accurately assess end user transaction response times, it seems clear that some
incorporation of network performance needs to be included into test results. However,
without a WAN emulation solution such as Shunra Virtual Enterprise (VE), the only effective
way to accomplish this is to place physical load generators in remote end-user branch
locations. This is both costly and inaccurate. First, the cost of deploying, maintaining, and
administering remote generators is an obvious negative. Second, since a distributed load test
is typically run during off-hours when the network is lightly loaded by other traffic, it is
unlikely that the test will appropriately reflect performance that actual users will experience
during working hours. Additionally, running a load test over the production network runs
the risk of impacting other production applications and users. Further, in most cases it is
not possible to test applications delivered over the internet to business partners and external
customers using this approach. As a result, few organizations have chosen to select this
approach.

Approach 3 – Estimate impact of the network

This approach is based on testing a single user running transactions at each remote site and
measuring the overall performance. This provides partial insight into the impact of the
network, but totally ignores the interrelationships that network performance has on the
application logic. In addition, since this approach entails a separate activity from load
testing, it is typically a manual process that is labor intensive and not repeatable. Whenever
the application logic or infrastructure configuration changes, there is an ongoing risk of
encountering an unanticipated adverse impact and unhappy users.

Approach 4 – Incorporate WAN emulation

This approach incorporates WAN emulation as an integral part of application deployment
automated testing. Shunra’s customers have adopted this approach and are vocal advocates
for the resulting benefits they have provided to their companies. With the most recent
versions of Shunra VE, integration into load testing scenarios has been made even easier to
accomplish.

Page 3 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

Manage Your Risk with WAN Emulation

Rigorous performance testing and optimization is a critical factor in the successful delivery
of key business applications. Yet frequently the performance of deployed applications
doesn’t live up to business requirements or end-user expectations despite load testing. One
reason behind these unpleasant “surprises” is the fact that most performance staging labs
only test the application with local users (in a local area network (LAN) environment), while
the fully deployed application is used by a variety of end-users, some local and others
accessing the application remotely over different network links. The different network
conditions that exist between end-users and application servers have a tremendous effect on
the overall performance that remote end-users experience. This deviation from performance
test results obtained in the lab is further exacerbated in the case of SOA composite
applications where various services may reside in different geographical locations, each
posing their own unique set of network conditions.

Ensuring Application Performance to Remote End-Users

A simple example of such a deviation can be seen from the results of the following test
conducted at a major North American financial institution. This test included both local
users, (i.e. users interacting with the database over a local, unrestricted link), as well as
remote end-users located in San Antonio, connecting to the datacenter in Tuttle, Ohio over
a wide area network link. The wide area network and end-user load were created using
emulation technologies. The emulated wide area link was a T1 connection, which
accommodated the latency (geographic distance) between Columbus, OH and San Antonio,
TX as well as the expected packet loss. Furthermore, the link was 30% utilized before the
test in order to replicate the exact conditions in the production environment where the same
network link supported other applications. This left 70% of the network link available for
our financial application in the test below.

Location Max Throughput (kbps) & Scripts

Bandwidth Utilization (%)
In Out Name Max Transaction
Load Response
Time
Local users in Tuttle, OH 3838.29 658.84 (0%) Financial 25 50.2
with local datacenter (0%) Transaction A
(LAN)
Remote users in San 1168.66 656.86 (80%) Financial 25 80.48
Antonio, TX (100%) Transaction A
Figure 1

Page 4 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

The results in Figure 1 show a substantial deviation from the performance of the local end-
users on the local area network to that experienced by end-users at the remote location on
the wide area network link. While increasing the load of local end-users did not seem to
strain the application server or backbone network, the same gradual load increase from the
remote locations negatively affected performance due to overwhelming strain on the
network and the application servers as well.

Now let’s drill down further to understand the end-user behavior at each location and the
causes of performance problems over the wide area network. The graph in Figure 2 below
shows the performance of end-user transactions over time (red line), over a local network
link, as the number of end-user transaction scripts running increases. In this case, the
transaction response time over the local network increases only slightly, reaching 60 seconds
at 25 end-user transactions (i.e. 33% increase from the starting point of 7 users at 45
seconds). This means that while the users are running over a local link, the system can scale
and support additional end-user load.

Figure 2. Results from testing end- user load over a local network.

However, when we ran the same test over an emulated wide area network the results were
drastically different.

Page 5 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

The graph in Figure 3 below correlates end-user load with wide area network utilization. In
this case, we can see that with as few as 16 end-user transactions (red line) running over an
emulated wide area network, the available network bandwidth (light orange line) is already
90% utilized. Furthermore, at 25 end-user transactions the response time is up to 100
seconds (an increase of 66% over the response time over the local network link).
Furthermore, nine minutes into the test, the application started generating errors (this is not
shown on the chart). Obviously, this application is not ready for deployment to remote end-
users over the wide area network.

Figure 3. Results from testing end-user load over a wide area network.

Page 6 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

The graphs in Figure 4a and 4b below show how HP Web Page Diagnostics help pinpoint
the extent to which particular web components are impacted by the network.

Figure 4a - Web Page Diagnostics with 50ms of latency

Figure 4b - Web Page Diagnostics with 100ms of latency

The graph in Figure 5 (next page) shows Shunra VE can deliver a breakdown of transaction
performance from a single Virtual User into Client, Network, and Server tiers during a load
test. Like the Web Page Diagnostics, Shunra analysis can be used to analyze web
applications. However Shunra’s technology also enables analysis of applications using other
protocols such as SQL and RPC. Targeted analysis and tuning can be quickly accomplished
for transactions that perform poorly over the WAN.

Page 7 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

Figure 5. Shunra analysis for an application’s Customer Summary transaction

By incorporating automatic analysis into load test scripts, Shunra makes it easy to see the
relative impact of the network and server on transaction performance.

It’s Not Just the Network

The tests above clearly show that production wide area network conditions can significantly
impact the performance of distributed applications, but it doesn’t tell the whole story. In
addition to the delay from the constrained network resources, the stress on the servers also
increases since more resources are needed to support remote end-users. Sessions are open
longer, OS resources are occupied for more time and more concurrent threads are needed.
None of these factors are taken into account when testing and optimizing the performance
only for local end-users on the local area network. Thus, remote business users are often
unpleasantly “surprised” by poorly performing application despite load testing efforts that
failed to address the impact of the WAN.

Shunra’s best practices implementation guidelines for effective application deployment is

based on a methodology that incorporates the following items in order to deliver
comprehensive performance testing capabilities that incorporate the key impact of the
network during application deployment:

Latency impact on the datacenter – Latency causes the transactions and sessions to stay
open longer than they would on the LAN. Without incorporating the WAN effects of delay
into a load test, memory usage, thread usage, connection pool utilization, network stack, and
other critical server resources can be significantly understated.

Page 8 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

Bandwidth limitations – Affects transactions “in-flight,” causing them to take longer to

complete their transmission, also impacting important server resources.

Network impact on response time – Where specific response time goals are being
targeted, it is important to incorporate the effect of the network into testing to examine the
impact on total response time. Shunra enables you to accurately assess total response time
for remote users prior to deployment, and can analyze the breakdown of total response time
into the client, network, and server components to assist in diagnosing and resolving
transactions that will not consistently meet your response time goals.
Site capacity limitations – the number of concurrent users that can use an application at a
remote site is highly dependant on the network capacity to and from the remote site.
Incorporating bandwidth limitations into the load test allows accurate prediction and
planning for the needed capacity to support an application at a remote site.

Figure 6 shows an example of how Shunra can be used to enable LoadRunner to determine
the actual response times by transaction for key remote locations. Realistic expectations can
be set for the end user experience as the application is deployed. If an application is not
going to perform acceptably for some remote users, Shunra VE’s analysis helps to rapidly
determine the underlying cause of the problem.

Page 9 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

Figure 7 shows an example of how Shunra can be used to enable LoadRunner to determine
the bandwidth capacity that will be needed to support expected usage for key remote
locations.

Two of the primary causes of application performance shortfalls are insufficient resource
capacity and shared resource contention. Capacity planning which is performed without
considering the network and remote end-users lacks vital information and may result in
misleading or incorrect recommendations.

Is the Network too Complex to Accurately Emulate?

Many load testing organizations are not aware that network-aware load testing is readily
attainable.

There are 4 common misconceptions about network-aware application testing:

1. Emulating the live network is not realistic

2. I need to have a network expert involved in my testing
3. Network emulation is a costly, time-consuming process
4. Network-aware testing rarely needs to be a high priority in test planning

Page 10 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

Here are the realities:

1. Emulating the live network is not realistic

Shunra’s customers have consistently found that it is very easy to test with enough
accuracy to properly anticipate performance as the application is rolled into
production

To completely model the network is truly a daunting task. There are many variables
including the potential interdependencies between multiple hops of network links, router
memory and configuration, changes to network routes, capacity, competing traffic, and QoS
amongst others.

Fortunately, you don’t need to model the entire network to perform network-aware load testing.
You don’t need to gain access to network devices or to install special agents or probes to get
the needed parameters for testing.

In many cases latency is the key factor that will influence application performance over the
WAN. Without network aware testing the critical impact of network latency is completely
ignored. The reality is that it is easy to gather the end to end latency information needed for
emulation testing. This provides a huge value to enabling accurate load testing with minimal
effort.

2. I need to have a network expert involved in my testing

The great thing about emulation is that the VUs will experience performance just as
if they were physically at a remote location.

There is no longer a need to deploy remote load generators to include the network into load
testing. Often even just a simple ping to each remote location is adequate for setting the
correct latency. Network Catcher lite is a simple (free) tool from Shunra that can
immediately give you the data needed.

For more advanced testing that includes bandwidth constraints and other factors, Shunra
offers a more advanced version of Network Catcher that allows you to monitor network
performance over a longer period of time with greater detail. Shunra also provides training,
professional services, and best practices to help users properly characterize and emulate the
network.

Shunra’s reporting includes intuitive reports that allow analysis of application behavior
without entailing the need for network expertise.

Page 11 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

3. Network emulation is a costly, time-consuming process

Shunra customers have seen significant and ongoing payback from their investment
in WAN emulation.

Shunra’s powerful technology is designed to make it quick and easy to setup and run a load
test that includes network emulation. By adding our VE Desktop solution to LoadRunner
you can install Shunra at 9AM and be performing network-aware load testing by noon

4. Network-aware testing rarely needs to be a high priority in test planning

Shunra has evidence from many customers across a wide range of industries that
prove that the network is often a critical component of application performance.

Many applications are still tested only under LAN conditions and then deployed to the
WAN. The cost of one failed deployment is a risk not worth taking, now that an integrated
testing solution is available.

I Don’t Control the Network – Why Should I Care?

Although responsibility for the network typically falls outside the responsibility of the
performance testing team, even if infinite bandwidth was provisioned with no packet loss
some application transactions would still fail to perform optimally for remote end-users. Add
the impact of real-world provisioned networks which have capacity constraints and periodic
packet loss and any transaction problems for remote users become magnified.

If an application is certified for deployment yet can't realistically handle the conditions on
the corporate network, there is very little anyone can do on the network side to fix it. From
the business owner’s vantage point as well as the CIO, it is really the responsibility of the
entire development team, and QA in particular, to verify that an application can cope with
the actual conditions that will be encountered on the corporate network.

Can I Use My Load Tool's "Think Time" and Bandwidth Option to

Create the Network Effects?

WAN Emulation lets you test the actual behavior of an application as it will perform over
the WAN. There are many underlying factors that can influence how an application will
perform under real-world latency and bandwidth that prevent a simplified approach such as
adding "think time" to the end-user scripts from being effective. All applications are
impacted by how chatty they are on the network. Chattiness is defined as the number of
application turns or times that the user needs to wait for the response to arrive back from
data transmitted to a remote application service.

Page 12 of 13

© 2008 Shunra Software Ltd.

 Understanding the Impact of Running WAN Emulation with Load Testing
.

In many cases the majority of the sensitivity of an application’s performance comes from the
latency, not a bandwidth constraint. The bandwidth throttling built into LoadRunner only
restricts outbound total usage, not the return path. Typically web applications send small
requests to the server and larger content such as images and tables are returned. This limits
the value of the throttling provided by LoadRunner.

With Shunra VE and LoadRunner users get a complete and accurate test that reflects how
the application will behave under real-world latency and bi-directional bandwidth
constraints.

Conclusion – Comprehensive testing can be done within a local

Performance Testing Lab

Shunra Virtual Enterprise is tightly integrated with leading tools such as HP LoadRunner.
Shunra’s training, implementation guidelines, and professional services have allowed
many leading organizations to rapidly achieve the benefits of comprehensive testing that
includes the WAN as an integral part of their load testing efforts. Shunra Virtual
Enterprise provides an accurate replica of the production network environment within the
test lab. With the Shunra Virtual Enterprise, users can quickly and easily find and resolve
problems during testing, not after rollout.

About Shunra
Shunra is the pioneer and market leader in predicting how business applications and network services will
perform for remote end-users - before deployment. It is the only company to provide both hardware and
software WAN emulation products that address the needs of all IT groups throughout the application
development lifecycle. Shunra’s headquarters are in Philadelphia, Pennsylvania and the R&D center is in
Kfar Saba Israel. The company also has local offices in the New York and the UK and is also supported
through a global network of channel partners. For more information, visit www.shunra.com or contact
Shunra at 1-877-474-8672 (U.S.) or +44 (0)207 153 9835 (outside U.S.).

Page 13 of 13

© 2008 Shunra Software Ltd.