Performance Management in Continuous Integration

This article provides a short implementation guide on how to integrate performance management
into development - or more specifically Continuous Integration.

The Basic Idea

Before going into the details I will start explaining the basic idea behind the approach. Let us
start with how a software system works from the performance perspective. The model we are
using here are queuing networks. A queuing network is a collection of resources (nodes) with
queues for processes waiting to get a hold on those resources. The resources are connected to
each other forming a network.

A Queuing Network Model of a Simple Application

We see all our major resources modeled which are CPU, Memory, Disk and other limited
resources like threads in thread pools, database connections or the network. How we use
these resources affects performance and scalability. If we - for example - execute too many
database statements, we will run into performance problem when load increases, as more
requests have to wait for a database connection to become available. Tuning approaches like
wait-based-tuning rely on these increasing wait times to identify performance problems.

If we run short of resources, we try to increase the amount of resources i.e. adding more CPUs
or memory. This is referred to as vertical scaling if it is done within one system or horizontal
scaling if is done by adding more nodes. See my post on Performance vs. Scalability for more
information and links on this topic. From a scalability perspective CPU and Memory are
scalable by hardware. Network and sequential shared resource access can only be scaled by
changing the application behavior and architecture. Architectural changes often involve a lot
of work and cost time and money. So our primary goal must be to discover architectural
problems as early as possible. Based on my own experience and also other people I have talked
to about 50 percent of later performance problems can already be found in development.

In addition to architectural problems the detection of performance regressions is our second

major concern. Especially in large applications which are continuously improved - so all serious
enterprise applications - changes get introduced by implementing a new feature which affect
the performance of other features. This is then very often discovered late during final load tests.
This is however not a developer fault. Very often a developer does not understand all
interdependencies of a complex application. Respectively side effects are not obvious.
Performance regression analysis should discover exactly these regression problems. This can
happen as easy as introducing an additional field in a data transfer object, eventually resulting in
additional megabytes being transferred over the wire in certain usage scenarios.
Bottom line Continuous Integration Performance Management avoids introducing
architectural problems and performance regression. This process also must be automated
as it has to be part of the Continuous Integration process.

Implementing Performance Management in Continuous Integration

Now as we are clear what we want to achieve we look at how to implement performance testing
and management and integrate it into our existing processes.