Professional Documents
Culture Documents
Image Area
www.infosys.com
Introduction
Quality is an increasingly critical factor for software products as customers become more sophisticated, technology becomes
more complex, and the software business becomes extremely competitive. Software quality may look like a simple concept, at
least in the literature. But, Software quality is not so straightforward in practice: where requirements change so rapidly, projects
are continually understaffed and behind schedule.
And definitely, one of the most important criteria for success of any product is to release the right product at the right time.
As Ed Kit has rightly said - It is fundamental to delivering quality software on time and within budget.
The quality of a software system is mainly determined by the quality of software process that produced it. Similarly, the quality
and effectiveness of software testing are largely determined by the quality of the test processes used.
We have to admit the fact that it may not be practicable to test a product fully. The test coverage provided to a product is limited
by the size of your test bank supplemented by some amount of ad-hoc testing. But, due to the increased product complexities in
todays world, the test bank sizes have become huge along with an extensive list of supported hardware-software configurations.
The challenge of providing coverage to the supported configuration matrix for the product has become equally important as
providing functional coverage. Having said this, we have to be with the fact that we have limited amount of resources/time
available for providing this coverage. Hence, the challenge will now transform into: we need to provide an optimum level of test
coverage to the product. That is where effective test management comes in to picture.
There are a number of essential questions for testing questions about product quality, risk management, release criteria,
the effectiveness of the testing process and when to stop testing. Measurement provides the answers to these questions. But
once we start to think about what can be measured, its easy to be overwhelmed with the fact that we could measure almost
anything. However, this is not practical and we have to create priorities for measurement based on what measures are critical
and will actually be used once we have them.
As stated by Albert Einstein Not everything that counts can be counted, and not everything that can be
counted counts
Quite often in the world of software development, testing remains a low focus area until software implementation has been
almost completed. Obviously, this approach to testing is inadequate in light of the increasingly high demands for software
quality and shorter release cycles. As a result, the place of testing in the software lifecycle has expanded.
This paper explores the challenges faced during the functional testing for a series of products from a worlds leading Identity
and Access Management solution provider along with the practices, which were adopted to cope up with such challenges.
The paper also provides an insight in to a comprehensive measurement program established for the project, leading the team
towards continuing operational excellence.
02 | Infosys
Test Requirements Gathering: Define clear, complete requirements that are testable. Requirements management plays a
critical role in the testing of software.
Test Planning: Identify test-creation standards and guidelines, identify hardware/software for the test environment, assign
roles and responsibilities, define test schedule and set procedures for executing, controlling and measuring the testing
process.
Test Strategizing: Devise plans for best possible utilization of the resources allocated for the test cycles ensuring optimum
test coverage.
Test Execution: Devise an efficient test execution flow/mechanism with the institutionalization of various tools, reusable
artifacts.
Defect Management: Associated with Test execution is effective defect management. As todays systems become more
complex, so does the severity of the defects. A well-defined method for defect management will benefit more than just the
testing team.
Test Results Reporting: With increased application complexity and significance, more and more people are interested in
the quality of a given product/application. By providing visibility in to a products health, large sets of stakeholders are able
to satisfy themselves as to the expected quality of the product. In addition, senior management and executives are able to
easily grasp and act upon critical quality information, acting on issues/exceptions before they turn into real problems. This
visibility is only useful if it is easy to find, easy to comprehend and personalized for the individual.
Test Metrics Collection, Analysis and Improvement: Institutionalize an effective metrics model to gauge the testing
processs health and take improvement actions on a continuous basis.
Infosys | 03
Since the PRS (Product Requirements Specification) and functional specifications for various
product features were the only inputs to the test requirements gathering process, following
were the major challenges for the testing team during this phase:
Challenges:
Best Practices:
Get the traceability matrices reviewed by development team for completeness and clarity
In case of any requirement changes, make corresponding modifications to traceability matrices at both levels
Traceability matrices to serve as a starting point for the new testers deployed for a features testing
In summary, collaborate with development team throughout the testing life cycle starting with test requirements gathering stage till
product release (Refer figure A-3 below for the collaborative model, which was followed by the project)
Product Name
Requirement Id
(as per PRS)
Requirement
Description
Feature
Name(s)
1.1
<This is my first
requirement>
<First Req>
First ReqTraceabilityMatrix-1.2
First ReqTestPlan-1.2
..............
..............
..............
..............
..............
..............
..............
..............
..............
..............
04 | Infosys
Feature Name
Sr. No.
Functional
Area
Sub-Functional
Area
1.1
<This is my first
functional area>
..............
..............
..............
..............
..............
..............
Functional
Operation
Test Case Id
Remarks
<First subfunctional
operation>
Strategy for
testing First
sub-functional
operation
1.1-A
If any
..............
..............
..............
..............
..............
..............
..............
..............
..............
..............
Benefits:
Gaps, if any in the traceability matrices closed after review by development team
Reduction in learning time for each feature by the testers as the traceability matrices are quick and easy to go through
The collaborative approach for QA as indicated in figure A-3 below proved to be really beneficial resulting in requirements clarity and
completeness, gap minimization and early gap closure. In reality, the collaborative approach with QA and Development teams working
in partnership has been found to be the Mantra for successful execution of testing projects
Infosys | 05
As per the earlier practice, the preparation of test plans was being carried out directly on
the basis of feature functional specifications, which led to the following challenges for the
testing team during this phase:
B) Test Planning
Challenges:
Best Practices:
Have the reviewed/approved traceability matrices as indicated in figure A-2 above serve as the basis for subsequent test plan generation
Benefits:
Due to completeness of traceability matrices, functional gaps in the test plans are reduced to a minimum
Reduction in time required for test plan creation as the same are now created on the basis of finalized traceability matrices
C) Test Strategizing
The product to be functionally tested was really complex one with a large feature set and a
number of supported hardware-software configurations. On the other hand, the time/effort
available for testing was limited.
As James Bach rightly said The test manager who refuses to face the fact that exhaustive testing is impossible chooses instead to seek an
impossible level of testing.
Hence, we as test managers ought to understand that an optimum level of functional test coverage needs to be provided to the product
within available resources.
Following were the major challenges for the test team during this phase:
Challenges:
How to utilize the available limited resources/time to provide optimum test coverage to the product functionality while covering various
supported configurations
06| Infosys
Best Practices:
Preparation of a consolidated functional test coverage matrix for various test cycles to be executed (Refer figure C-1 below for a snapshot
of consolidated functional coverage matrix)
Preparation of a consolidated configurations coverage matrix for various test cycles (Refer figure C-2 below for a snapshot of consolidated
configurations coverage matrix)
Utilize an extensive decision model while deciding on the test matrix for each cycle (Refer figure C-3 below for a snapshot of the QA
decision model in place)
Prepare a detailed test matrix for each test cycle to monitor the details for test environment to a minute level and get the same reviewed
by the development team before proceeding with the testing (Refer figure C-4 below for a snapshot of detailed component and
functional test matrix for a test cycle)
Infosys | 07
Benefits:
Starting early with test automation helped the team in catching defects early in the product implementation phase
Institutionalization of daily test harness helped quick detection of defects and regressions caused by recent code check-ins
Portability of test harness enabled its execution with various supported hardware-software configurations with little or no effort thereby
resulting in maximum utilization of test harness
Inclusion of automated test runs in to the test matrix enabled the test team to reduce the effort for testing radically
The practice of having a detailed component and functional test matrix and its subsequent review by development team prevented any
ambiguities in setting up the test environment and set an agreement between both development and test teams before going in to
actual testing
08 | Infosys
D) Test Execution
This has been one of the most critical phases of the functional testing life cycle. Once we
have decided on the overall plan and strategy for test execution, this is the phase when the
test team will really come in to action and will make an optimum utilization of the time and resources allocated for testing. In spite of doing
a detailed and effective planning for your test cycles, sometimes test teams fail to accomplish the planned amount of testing during this
phase while juggling with various issues related to test environment setup, test case understanding and moreover product areas, which are
not testable due to certain blocking functional issues.
Following were the challenges, which the test team had to cope up during this phase:
Challenges:
Testers struggling while learning the product functionality by using the actual product build for the first time and due to mismatch
between test plans and actual functionality
A large amount of time going towards test environment setup due to product complexity and an extensive set of supported
configurations, which need to be setup
Recurring issues related to test environment setup and test execution, which take enormous time in resolving for most testers especially
who are new entrants to the test team
Blocking issues found in the product functional areas lead to extensive re-planning putting the initial overall plan at a stake
Best Practices:
Set up a process to get the intermediate product builds well before the formal QA hand-off. This process is also known as in-process
testing (as indicated in figure A-3 above)
Institutionalize usage of re-usable test environments through various tools viz. VMware images, Ghost images etc
Establish a knowledge repository of various problems encountered during test environment setup/test execution and their possible
solutions. This repository can be searched quickly especially by novice testers instead of re-inventing the wheel. Anyone in the team,
whenever solves a particular problem logs the problem and the corresponding solution in to this repository. Moreover, an email is
generated at the same time to the whole team containing the problem and its solution
Benefits:
Starting with the in-process QA well before the start of formal test cycles enabled the test team to learn the functionality by playing
around with the product
Functional gaps in the actual product and the test plan were closed early before going in to formal functional testing
Test team was able to identify and report defects for the blocking functional areas well before going in to formal functional test cycles
resulting in effective utilization of the time allocated for formal test cycles and well-planned testing
An overall reduction of approx. 25% was realized in the time going towards test environment setup due to the reusability achieved with
the usage of VMware/Ghost images. The test environment, setup by one tester can now be reused by multiple other testers. Moreover,
the setups for third party servers/components can be preserved as VMware images for usage during subsequent test runs
The institutionalization of Problems/Solutions knowledge repository resulted in approx. 30% reduction in the time going towards issue
resolution. Any tester now facing a problem has to search the repository for a solution before going ahead with spending time on
resolving the same
Infosys | 09
E) Defect Management
Effective defect tracking, analysis and reporting are critical steps in the testing process. A
well-defined method for defect management will benefit more than just the testing team.
The defect statistics are a key parameter for evaluating the health of the product at any stage during test execution.
Extra time spent on preparing a defect profile and its history often benefits through easier analysis, shorter resolution times, and better
product quality. This not only includes the new defects reported against the product, but also the defects reported during earlier test cycles
and the defects to be verified.
Following were the challenges, which were faced by the test team while managing the defects for the product:
Challenges:
Incomplete and ambiguous defect reporting resulting in too many defects coming back for Needs More Information from developments
end
Testers opening new defects for the problems already reported by other testers as defects during previous test cycles
Too many defects being marked as Not-A-Defect by development due to Operator errors
Inappropriate verification procedure for the fixed defects, which come for verification
Inappropriate tracking of defect trend resulting in lack of insight into current health of the product
Best Practices:
Institutionalize standard templates for defect profile preparation and defect verification profile preparation having comprehensive
details for each defect (Refer figures E-1 (a) and E-1 (b) below for snapshots of these templates)
Establish clear definitions for Defect Severity and Priority. Train test team members on the same and have all of the defects go through
a thorough review by the test lead before reporting
Maintain an updated list of the various defects logged against the product till date containing appropriate details for these defects,
which testers can refer to as and when they encounter a defect (Refer figure E-2 for snapshot of sample defects list for a product)
Coordinate and discuss with development team regarding the suspicious defects before reporting
Establish a process, wherein the development team specifies the procedure of verification for all the fixed defects in the defect tracking
system. Also, attach the verification profile for all the verified defects in the defect tracking system
Institutionalize various defect metrics, which are tracked, analyzed and reported on a continuous basis to various stakeholders (Refer
figure E-3 for snapshots of a few defect metrics, which were tracked)
10 | Infosys
Figure E-2 List of Defects Logged against the Product A Running List
Benefits:
Standard templates for defect profile and defect verification profile resulted in consistent and comprehensive defects being logged by
various test team members
Assignment of appropriate Severity and Priority to various defects logged enabled development team to focus on these defects in a
well-organized manner
Referring to the list of the defects logged against the product till date enabled the test team to avoid any Duplicate defects as well as
to track every defect to a logical closure
Coordination with development team on suspicious defects brought about a considerable reduction in the number of defects being
marked as Operator Errors
Specifying the verification procedure in the defect tracking system enabled the development team to validate the verification procedure
followed by QA for any defect resulting in an increased consistency in the verification procedure
Institutionalization of various defect metrics resulted in an accurate and quick insight in to the products health and focus efforts in the
right product areas
Infosys | 11
Figure E-2 List of Defects Logged against the Product A Running List
12 | Infosys
Once the test team is done with the test cycle or even at intermediate stages, it is desirable to
have a visibility into quality of the product. With the increase in both product complexity and
significance, more and more people are interested in the quality of a given software product. By providing this visibility, large sets of
stakeholders are able to assure themselves as to the anticipated quality of an application. In addition, senior management and executives
are able to easily grasp and take action on critical quality information, acting on exceptions before they turn into tribulations.
Following were the challenges, which were faced by the test team during the test results reporting phase:
Challenges:
The test results report should be easy to comprehend and should be useful for all the stakeholders starting with the testers in the team
to the senior most executive having an interest in the product
It should provide a status of the feature-wise health of the product along with feature-wise defects data for an easy decision-making
towards Go/No-Go for the product
Best Practices:
Simple metrics, charts and graphs are often preferable to text, as they are easy to comprehend, and because they highlight exceptions.
A comprehensive test results report template was prepared in a spreadsheet format, which was then reviewed and approved by the
customer containing the following details:
o Component Version Details: This sheet provides the details of versions for various hardware-software used for functional testing
o Test Matrix: This sheet provides the actual test matrix used for testing
(Refer figure C-4 above for a snapshot of detailed component and functional test matrices for the test cycle)
o Result Summary report: This sheet provides the overall summary of the test cases executed and effort spent during the testing for each
feature
o New Defect Details: This sheet provides the details for the new defects found during the current test cycle
o Old Defect Details: This sheet provides the details for the defects, which were filed during some earlier test cycles for the product and have
also been observed during the current test cycle
o Feature-Risk Analysis: This sheet provides the risk associated with each feature on a scale of High (red), Medium (yellow) and low (green)
based on the test cases which have failed, are blocked or were not executed for the feature
o Feature-Test Case Percentage-Chart-<Platform>: This chart provides the distribution of percentage of test cases passed, failed and not
executed for each feature on <Platform> platform. There will be one such sheet for each platform tested
o Feature-Test Case Count-Chart-<Platform>: This chart provides the distribution of number of test cases passed, failed and not executed for
each feature on <Platform> platform. There will be one such sheet for each platform tested
o Defect Severity Distribution: This chart provides the distribution of defects based on Severity
o Defect Priority Distribution: This chart provides the distribution of defects based on Priority
(Refer figure E-3 above for snapshots of defect severity/priority distribution)
o Effort-Feature-Chart-<Platform>: This chart provides the effort distribution per feature basis executed on <Platform> platform. There will
be one such sheet for each platform tested
(Refer figure F-1 below for snapshots of sample graphs/charts included in the detailed test summary report)
Infosys | 13
Provides a risk assessment for each feature from release readiness point-of-view
Benefits:
14 | Infosys
Once the test team is done with the test cycle or even at intermediate stages, it is desirable to
have a visibility into quality of the product. With the increase in both product complexity and
Challenges:
The need for a mechanism for measuring the efficiency, effectiveness and quality of the testing process so as to identify the areas of
improvement
Best Practices:
Identify a set of process/product metrics to be tracked on a continuous basis. Refer Figure G-1 below for a summary of the various
metrics institutionalized:
Refer Figures G-2, G-3 and G-4 below for snapshots of test metrics adopted for test process efficiency, effectiveness and quality respectively.
Refer Figures G-5 and G-6 below for snapshots of test metrics institutionalized for Product/Program Health and Defect Tracking respectively.
Develop dashboards for these metrics and share these dashboards with the customer at 2 levels: at test program level, at regular intervals
Benefits:
The dashboards enabled integrated, accurate and actual reporting, with a holistic view of metrics and graphical charts to facilitate
decision making
Analysis of the metrics provides an insight in to the process maturity and the areas, where improvements can be targeted
Customer gets a formalized mechanism to assess the effectiveness and efficiency of the QA process
A useful way to educate the team on the importance of various process parameters and to eliminate the problem areas in a planned
manner
Infosys | 15
b)
Cost of Testing
Shows phase-wise effort distribution Depicts intensive effort areas to focus for improvement
16 | Infosys
c)
d)
e)
Infosys | 17
b)
c)
18 | Infosys
d)
e)
Depicts the quality of functional test teams work. Having more operator
errors is an area of concern and needs to be looked into
Infosys | 19
Feature Sensitivity
Shows the % of failed test cases for various product features over
multiple test cycles Depicts sensitive features
b)
Feature Sensitivity
20 | Infosys
Shows new defects logged during various test cycles. Depicts how
the product quality faired over multiple test cycles
Depicts (new + old) defects logged during various test cycles. The old
defects are the ones, which we reported during an earlier test cycle
and are detected during the current test cycle also.
Infosys | 21
Continuous improvement is the key to success of any process. Having illustrated the metrics model in place, we will have to
continuously enhance the metrics model to strive for continuous improvement.
As H. James Harrington truly said - The journey towards excellence is a never ending job.
References:
Software Engineering: A Practitioners Approach, 6/e by Roger S Pressman, R.S. Pressman and
Associates, ISBN: 0072853182, Copyright year: 2005
A Framework for Good Enough Testing by James Bach, Reliable Software Technologies
Software Testing in the Real World Improving the Process, by Edward Kit
22 | Infosys
Appendix
Common Terms Used in This Document:
QA Quality Assurance
IT Information Technology
Infosys | 23
About Infosys
Infosys partners with global enterprises to drive their innovation-led growth.
That's why Forbes ranked Infosys 19 among the top 100 most innovative
companies. As a leading provider of next-generation consulting, technology
and outsourcing solutions, Infosys helps clients in more than 30 countries
realize their goals. Visit www.infosys.com and see how Infosys (NYSE: INFY),
with its 150,000+ people, is Building Tomorrow's Enterprise today.
www.infosys.com
2013 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change without notice.
Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Except as expressly permitted,
neither this documentation nor any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing, photocopying, recording or
otherwise, without the prior permission of Infosys Limited and/ or any named intellectual property rights holders under this document.