A Strategy for Designing
and Executing an
Effective Regression
Test Program
1. Introduction
This paper focuses on the area of
regression testing. An industry-wide
definition of regression test is “retest-
ing of a previously tested program fol-
lowing modification to ensure that
faults have not been introduced or
uncovered as a result of the changes
made.”
We have adapted this slightly for the
type of testing we perform to give the
following definition: “a test designed to
confirm that an existing feature still
performs correctly after any software
modifications.”
Both these definitions provide the read-
er with a sense of the importance of
regression testing. It is of no benefit
providing your valued customer with
lots of new and exciting features if you
manage to break features they have
been using successfully for the last
number of years. In fact, some feedback
received from a customer several years
back indicated that they expect to have
problems with new features when they
first get a new release of software, but
September/December 2001 http://www.testinginstitute.com
that they cannot tolerate existing fea-
tures not working on the new release.
So, regression testing is recognised as
being very important, yet it often
remains the test phase given the least
attention in the planning of test activi-
ties. Too often, organizations take the
easy way out and regression test phases
simply consist of reruns of existing test
suites in the hope that these will catch
all the problems. This approach has a
number of potential flaws:
• If you are dealing with a large com-
plex system, you can end up with
thousands upon thousands of regres-
sion tests as you continue to add fea-
ture test suites from previous releas-
es to the regression program for your
current release. This is all well and
good if you have a fully automated
and stable test environment, which
allows you to run all these tests
quickly without using a lot of
resources.
• You may not have the required cov-
erage with your existing test suites to
find problems being introduced as
Journal of Software Testing Professionals
new features or other code changes
are being added.
• Although you may build up a library
of regression tests over time, and this
library can be quite extensive, they
can also prove troublesome as they
contain both redundant test suites &
test cases which themselves contain
errors. There can be a huge overhead
in maintaining this library of regres-
sion suites and test cases.
So, an unplanned approach to regres-
sion testing can lead to inefficient use of
resources. If you take the time to plan
your regression test program, there are a
number of questions you need to ask:
• What do I need to test?
• How long is it going to take?
• What resources are required?
• What tools are required?
• Are there any other special
requirements?
The first question on this list is the most
important one to answer, as it will help
answer the remaining questions. Apart
1
from being the most impor-
tant question it is also the
most difficult to answer.
2. Understanding Our
Problem
We are involved in sub-sys-
tem and system testing of
software releases for a
Mobile Switching Centre
(MSC). We test in both an
MSC-only configuration and
also in a full cellular system
configuration with other net-
work elements. Over the past
3 years the complexity of the
features being developed for
the MSC has increased
tremendously as we move away from
MSC-only features to features that span
several network elements. Our regres-
sion testing is a mixture of automated
(30%), semi-automated (30%) and
manual (40%).
On our current release of MSC soft-
ware, we decided to spend some time
analysing the way we approached our
regression testing. In general we had
been taking an ad-hoc approach to plan-
ning our regression test program for
previous releases and our analysis
uncovered the following facts:
• We are dealing with a legacy system;
there has been a lack of consistency
in the way features developed over
the years have been documented and
controlled. This has led to a lack of
traceability from existing test
cases/suites to old feature/functional
requirements.
• Consistently on all our releases, our
Problem Report (PR) arrival rate
showed a straight-line increase with
no levelling off towards the end of
our cycle as shown in Figure 1. A
‘normal’ PR arrival rate is one where
you are finding the majority of your
problems early in your cycle, thus
September/December 2001 http://www.testinginstitute.com
allowing them to be fixed early. This
then reduces the risk of impacting
quality with fixes having to be made
to the release late in the cycle prior to
customer deployment.
• We had an increasing number of
regression tests per release, as we
simply added new tests to cover new
features developed on the previous
release we had tested. This increas-
ing number of tests had turned our
regression cycle into a 12-week
interval on average and this was
increasing with every release. In fact,
over a 6-year period our regression
cycle had increase from 6 weeks to
the current average of 12 weeks.
• There was a lot of frustration among
test engineers, as they were simply
running regression tests for no appar-
ent reason except we did it on the last
release so must do it again! More
often or not, these tests would not
even find any problems, so the frus-
tration increased.
• Over the years we have built up an
extensive library of regression tests
through the addition of tests to cover
new functionality. Even with this
library of tests, we can never be
Journal of Software Testing Professionals
100% sure that we have
full coverage, as the
method of selecting the
regression tests was
purely to try and cover
the basic functionality of
the MSC.
We have been perform-
ing Escaped Defect
Analysis & Root Cause
Analysis on problems
that escape to our cus-
tomers, and this showed
us a number of things in
relation to our regression
testing:
• We were good at testing new feature
and enhancements but problems
escaped in other areas that should
have been covered by our regression
testing.
• By simply re-running existing
regression suites, we could not hope
to catch most of the problems that
were escaping. Regression suites
often covered the basic functionality
of a particular feature, so if a code
change resulted in an impact outside
of this basic functionality, we did not
have the required coverage to catch
the problem.
• Quite often, the solution to the
Escaped Defect Analysis was just to
add a test to provide coverage for the
problem that escaped, thus increas-
ing the number of regression tests
even further. Clearly it would be bet-
ter to try and detect the problem in
the first place rather than adding a
test just in case it re-occurred.
3. A Time For Change
Once we had done the analysis, it was
time for some change. Quite simply we
could not continue the way we were
going.
29
Due to the effort involved
for older features and func-
tionality, we could not go
back and resolve the prob-
lem of poorly documented
and controlled requirements
and the lack of traceability,
so we had to work around
this in some way.
Why did our typical prob-
lem arrival rate show a
straight line for all our
releases? Some of it was
due to churn on the release,
i.e., features being added
late in our cycle causing
new problems to be intro-
duced and found late in our
cycle or sometimes not at
all. However, the major
finding was that we simply
had no strategy in terms of the way we
executed our regression tests—engi-
neers simply took their suites, went into
our labs and started executing. We
decided that we needed a way of priori-
tising our tests and then to execute
based on the priority.
While looking at the issue of the
increasing number of tests per release, it
was clear that we had no regression
selection criteria, except that we were
trying to show that existing features and
functionality still worked. We decided
we needed some way of reducing the
amount of tests we were running by
identifying the tests that were actually
necessary to run. When testing new fea-
tures you have requirements from
which to work from in developing tests
and have a methodology that can be fol-
lowed to create these tests. For regres-
sion we had nothing, so there was a
need to create some form of require-
ments, which we could then use to iden-
tify the required regression testing. To
do this we needed specific information
from the development organization on
the changes they were implementing in
30 Journal of Software Testing Professionals
the software, both for new features/
enhancements and any other changes
they were making (problem fixes, re-
engineering work etc.).
We used 2 ways of collecting this infor-
mation:
3.1 A Regression Impacts Matrix.
We created a simple Regression
Impacts Matrix that lists all existing
features and areas of functionality. Each
development team working on a new
feature or enhancement was then asked
to complete the matrix indicating the
features and functionality both directly
and indirectly impacted by their code
changes. Table 1 shows a sample
Regression Impacts Matrix.
It is quite simple in format and lists all
existing features and functionality
along the top row with all new features
on the release being listed in the first
column. Where a new feature has an
explicit impact on an existing feature,
an ‘X’ is used to indicate the impact.
Where the impact is not explicit, an ‘R’
is used to indicate a related impact.
http://www.testinginstitute.com
Related impacts can be consid-
ered to be of lower priority and
tests for these features can be
included in the Secondary
phase (refer to Section 4.).
Where there are no impacts,
the cell can be left blank.
3.2 A Test Impacts Checklist
We used an existing Test
Impacts Checklist. The devel-
opment engineers are required
to complete this checklist for
every submission of code they
make to the software release.
The checklist asks specific
questions that the developer
has to answer, and the test
engineers can then use this
information to identify regres-
sion impacts due to code
changes not associated with a
new feature or enhancement
(i.e. fixes, re-engineering). In our case,
the checklist was broken down into sev-
eral areas as follows:
Area of Impact: We were able to identi-
fy 8 main functional areas (including
call-processing, Billing, Fault Man-
agement & Statistics), and the develop-
er first has to indicate which area was
being impacted by their code change.
This will then give the testers an idea of
the type of regression test they might
require.
Description of Impact: They are then
asked to describe the impact in their
own words. This information helps fur-
ther define the regression test that might
be required.
Call Scenarios: Because a lot of our
testing involves call processing, devel-
opers are asked to identify specific call-
scenarios that are impacted. Example
scenarios include: three-party confer-
encing, call forwarding, and hand-offs.
Knowing the call-scenarios impacted
allows the tester to select these scenar-
ios from our existing suites of regres-
September/December 2001
sion tests if they exist, or be in a posi-
tion to create new regression tests if
required.
Area of Code Change: Developers are
asked whether the code change is in a
frequently executed leg of code, i.e.,
would every call-scenario be affected.
This gives testers an idea of the likely
complexity of the required regression
test, as a code change in a less frequent-
ly executed leg of code may require
additional effort to create the regression
test.
Recommended Regression Test: The
developer is asked to recommend what
regression testing would be required to
exercise both the code change itself and
also functionality around the code
change. Depending on the developer’s
experience, this information can again
be very useful to the tester.
Features Impacted by Code Change:
Similar to the Regression Impacts
Matrix, the developer must indicate
what features are impacted by the code
change. This will allow the tester to
cross-check this with the Regression
Impacts Matrix and they can determine
if additional feature-specific regression
testing is required.
External Message Changes: As we deal
with a product that interfaces to a large
number of other products, there can be
external message changes involved in
the code change. If so, this can often
add complexity to the regression testing
required, as it can either mean that:
a) Another product will also need to be
modified and we may need that prod-
uct modification to fully regression
test, or
b) We will need to make modifications
to our test tool in order to update it to
include the new external message
change.
September/December 2001 http://www.testinginstitute.com
By using all the information provided
by both the checklist and the matrix,
test engineers can then decide on the
regression testing required and allowed
them to identify any required test envi-
ronment changes. This gives them a
sense of ownership and purpose, and a
chance to help design the regression
program and make decisions them-
selves rather than being simply told
what regression suites to execute.
4. The Regression Strategy
A strategy was needed to pull every-
thing together so that:
a) We were consistent in our approach,
b) It was repeatable, and
c) It led to focused regression testing
taking place.
We came up with the following
approach based on grouping our suites
into 3 main types and then executing
our regression tests in 3 phases:
4.1 Primary
The Primary group consists of suites
and test cases that are selected for exe-
cution based on the following criteria:
• Providing coverage for areas directly
impacted by new features and other
known impacted code changes – all
the information provided in the
Regression Impacts Matrix and Test
Impacts Checklist is used to identify
either the individual regression tests
or suites of regression tests required
to test the impacted areas.
• A small subset of tests covering basic
functionality for all features – here
we simply select smaller subsets of
suites from our regression library. It
is much easier to then maintain this
smaller subset.
The idea behind identifying Primary
test suites is that these will focus on the
impacted areas of the software and so
Journal of Software Testing Professionals
will increase the possibility of detecting
problems due to those impacts. These
suites would also cover overall basic
feature functionality thus giving a high
degree of confidence in the overall
quality of the software release.
4.2 Secondary
The Secondary group consists of suites
and test cases that are selected for exe-
cution based on the following criteria:
• Suites that have repeatedly found
problems on previous releases – this
information was readily available
from our Final Test Reports for pre-
vious releases.
• Suites that cover customer-sensitive
functionality such as billing and sta-
tistics.
4.3 Tertiary
The Tertiary group consists of suites
and test cases that are selected as fol-
lows:
• Hot-spot suites – these are selected
based on problems found by earlier
Primary and Secondary testing, i.e.
you might wish to increase test cov-
erage due to large number of prob-
lems found in specific areas.
• Additional testing due to late code
additions – once again we would use
the information provided in the Test
Impacts Checklist to identify the
tests required.
• Clean-run/Sanity testing prior to
deployment of software release to
customers.
Primary and Secondary phase testing
can be identified early in the release
cycle, and can therefore be planned for.
Tertiary tests are more ad-hoc and
unplanned, but we allow time for them
in the schedule when Primary and
Secondary testing is complete. We can
31
then plan and report on their status on a
short-term basis.
5. The Benefits
This new approach to regression testing
has just been used on our current release
of software, and we have seen the fol-
lowing benefits:
• We have reduced the number of tests
in our regression program to 2000
from a figure of approximately 6000
on previous releases. While this is a
large decrease in the number of tests
being executed, we were confident
that there would not be an impact to
quality as the strategy allowed us to
focus on covering all basic function-
ality, plus the areas where changes
were being made. This was clearly a
big improvement on simply rerun-
ning tests from previous releases
without any sound reason.
• Our problem arrival rate was less lin-
ear - Figure 2 shows our Current
Release Problem Report Arrival
Rate. For this release we had a total
regression test interval of 28 weeks,
and the graph shows that we opened
30 PRs during the first half of this
interval. This is 65% of the total
number of PRs we opened during
regression test. This compares to
approximately 30% of PRs being
opened during the first half of our
interval on previous releases. This
difference can be attributed to testing
of impacted areas earlier in our cycle.
• Test engineers have increased satis-
faction because they see a purpose to
what they are doing as they are
actively involved in designing the
regression program. They also get a
chance to learn more about the fea-
ture and functionality they are test-
ing, as they are not executing a suite
simply because they were told to do
it.
32 Journal of Software Testing Professionals
• We are focusing on high-risk areas
earlier, increasing the chances of
finding the more serious problems
earlier in the cycle.
• The strategy is flexible and there is
scope for additional testing if
required.
• There is an increased level of aware-
ness from the development commu-
nity with regards to the impacts their
code changes can have. We have also
increased their interaction with
testers in helping to define the
regression program.
• It had led to improved maintenance
of existing regression suites given
that impacts and feature interactions
are identified earlier and changes/
updates to suites can then be planned
during the test development phase of
the release.
The current release has just been
deployed on a customer site for the first
time. When we compare the level of
escaped defects during this deployment
phase to the previous release, we see a
reduction in the amount of problems
found, from 21 to 8. This reduction can-
not however be directly attributed to
our new regression strategy as other
defect reduction activities have also
http://www.testinginstitute.com
taken place on this release earlier in the
lifecycle. These activities focused on
improvements to the formal inspection
process including: improved inspection
checklists and training, along with reg-
ular analysis of inspection data, leading
to reinspection of out-of-bound inspec-
tions.
6. Issues
While we have seen many benefits of
this new strategy, there are some issues
we have noted:
• Depending on the number of test
engineers available, it can be difficult
to schedule execution of the test
suites in each phase so that each
phase completes fully before the
other begins. It is recommended that
test managers aim to complete a cer-
tain percentage of the Primary phase
before beginning the Secondary
phase and similarly with the
Secondary phase before moving onto
the Tertiary phase. It is suggested
that between 80 and 90% of testing
in each phase should be completed
before starting the next phase.
• There is effort involved in planning
and executing this regression strate-
gy, and this effort needs to be esti-
mated, resourced and scheduled in
the project plan accordingly. It needs
September/December 2001
 to be seen as an integral part of the
project.
• The benefits of this strategy do need
to be explained clearly to the devel-
opment community so that they can
see the importance of their contribu-
tions. We have been moving to an
organizational structure where devel-
opers and testers belong to the same
team. For those organizations where
an independent test team exists, there
may be some friendly persuasion
required!
7. Summary & Conclusion
Regression suites can often contain
thousands and thousands of tests as they
are built up over releases of software,
especially when dealing with a legacy
product. When looking to focus regres-
sion testing, requirements are needed
along with a strategy that is consistent,
repeatable and flexible. Development
and Test Engineers need to be actively
involved in defining the requirements
for the regression program and then
designing the program that will test
these requirements. This strategy does
September/December 2001 http://www.testinginstitute.com
not provide for 100% coverage of the
system being tested. What it does is
identify what regression tests need to be
executed based on the ‘requirements’
we have defined using the strategy, i.e.,
the tests in the Primary, Secondary and
Tertiary Phases. Given the complexity
of the system we are testing, we don’t
know for sure that we have exact cover-
age but we are sure that we are covering
all basic feature functionality, plus
focusing on areas where changes are
being made.
For operational reasons, my organiza-
tion is moving towards a combined
development/test team. As a result of
this we are going through some ‘test re-
invention’ at the moment. As we have
successfully reduced the amount of sub-
system & system regression testing we
need by using this strategy, increased
focus is now being placed on lower
level testing such as unit & integration
testing, and the idea of regression test-
ing is now being applied to our integra-
tion testing, where we will now regres-
sion test individual low-level functional
areas of the MSC. We will now be using
Journal of Software Testing Professionals
this strategy for our low-level function-
al area regression testing. We have seen
several benefits on the first release
where we used this strategy to design
our regression program and feel it can
be adapted to suit different applications
and products.
About the Author
Martin Gilleran graduated with a B.Sc.
in Electrical/Electronic Engineering
from Trinity College, Dublin in 1986
and has spent the last 13 years working
in industry. He spent 4.5 years with
Measurex, Inc. as a software develop-
ment engineer responsible for develop-
ing, testing and installing measurement
and control systems for the paper
industry. Mr. Gilleran has spent 8.5
years working with Motorola in differ-
ent test roles. He has 4 years experience
working as a test project. He then spent
2 years as a Test Manager responsible
for testing a range of Motorola's
Network Products including Mobile
Switching Centre (MSC) and Home
Location Register/Visitor Location
Register (HLR/VLR).
33