SQS Group Limited

Vodafone
Automation HealthCheck
Report

Darryl Kennedy
6
th
February 2013
Executive Summary
Background
Vodafone have an existing test automation solution created using QC 10 and QTP 11
The automated regression testing solution includes both ECC (GUI) and SRM (Portal)
The existing solution has been in existence since its creation in December 2011
The existing solution is maintained in Pune by VISPL Automation Team
Test execution and reporting in carried out in Pune by VISPL Automation Team

Purpose of the Automation HealthCheck
In order to maximise the benefits of automation, SQS have been asked to undertake an
independent review (HealthCheck) of the current test automation solution to ensure that
Industry best practice has been employed
The solution is extensible and provides the necessary portability and robustness
The test assets created are re useable and easily maintainable
Tests execute unattended, results and evidence are clear and easily interpretable
SQS have been asked to provide any recommendations for improvement along with
supporting benefits of implementing any recommendations
Work Carried out
SQS provided Vodafone with a test automation specialist who
Reviewed a cross section of the QC and QTP assets available
Assessed the approach to automation and the automation framework deployed
Assessed the versatility, robustness and extensibility of the automation framework
Assessed the approach to script development including test data
Assessed error / exception triggers and recovery
Assessed automation synchronisation and execution performance
Assessed ability to execute unattended and the existing infrastructure

High Level HealthCheck Finding
The QC and QTP solution adheres to certain design principles necessary for a
maintainable component driven framework but lacks necessary robustness
The existing solution performs reasonably but essential attributes are missing or poorly
implemented which are necessary for enterprise strength automation
Important strategic changes must be implemented if the true benefits of test automation are
to be realised before significantly extending the scale of the existing solution
Executive Summary
Automation Solution Summary
A good technical basis for an automated testing solution incorporating some features
provided by the HP Tool set. Consideration has been made to ensure the framework is
extensible and maintainable with minimal effort.
Some fundamental characteristics of test automation have been introduced such as shared
object repositories, data abstraction and the use of external function libraries to provide a
modular design.
Web and SAP automation present a number of challenges and although innovation has
been applied in the framework to provide solutions above what QC and QTP provide out of
the box some basic principles have not been adhered to:

Modular design of the solution and use of external data and function library files
Custom HTML reporting easily interpretable by non technical users
XML Configuration files for multiple environment execution & Country specific test data
Lacks an overall framework engine with the appropriate flow control
Synchronisation and error handling needs a lot of work so able to run unattended
No validation included into the functions which make up the Test Scenarios

The VISPL team were knowledgeable of their subject and extremely collaborative.
Executive Summary
Recommendation Priority & Implementation Complexity
The table below summarises all the recommendations made in this report into the priority
for implementation.






The table below summarises all the recommendations made in this report into the
complexity of the implementation.


Executive Summary
Implementation Priority Total Recommendations
High 36
Medium 13
Low 1
Implementation Complexity Total Recommendations
High 12
Medium 10
Low 28
Automation HealthCheck
The HealthCheck focussed upon
Adherence to test automation best practice
Ease and skill requirements of execution and maintenance
Future proofing

The HealthCheck did not
Test existing automation (any issues identified during the HealthCheck will be reported)
Review the automated scripts for test coverage, accuracy to reflect business processes
or ability to reflect business testing needs
Review the scope or prioritisation of test scenarios being automated
Estimate the cost to run and maintain the scripts

Basis for the HealthCheck
The findings are based on virtual meetings that took place on 4
th
and 5
th
February
Thank you to the VISPL Automation Team for the help and support they provided
Raghavendar Bandaru, Ganesh Kaushik and Nagesh Battula
Automation HealthCheck Scope
Framework
Component
Criteria Rating Comments
Scalability
Modularity Green
A good approach has been applied to ensure modular/component
design of the code for reusability.
Concurrent Use Amber
The modular design and abstraction layers allow for some concurrent
use. Large function library files and global objects create contention.
Reusability Amber
Some good solutions for reusability including different data tables for
country specific data. The configuration limitations make this difficult
during execution.
Suitability Amber
A good overall methodology adhering to modular design but there are
some limitations which must be addressed before further development
(see detailed recommendations).
Reusability
Ease Of Use Amber
A clean solution using much of the standard QC and QTP
functionality. Easy to understand but lacks manual steps for end user.
Reliability Red
Standard QTP object timeouts may work well with SAP GUI but the
same approach can not be used for the Web Portal.
Maintainability Amber
Modular design and abstraction layers for code, test data, objects and
the use of external functions libraries and XML configurations.
Limitations with large function library files.
Reporting Red
HTML reporting introduced in addition to the standard QTP reports but
additional coding steps needed and no validation included.
Error Handling Red
Detection, reporting and handling of errors is poor. This will also
occur more often due to the lack of dynamic synchronisation.
Test Automation Score Card
Framework
Component
Criteria Rating Comments
Audit Trail
Naming
Conventions
Amber
Naming conventions exist for Test Scenarios and Functions which
are adhered to. Additional detail should be applied to make it easier
for end users to interpret these easier i.e. SAP T Code.
Unique
Identifiers
Amber
Test Scenarios and functions are distinctly named but do not contain
a unique identifier.
Mapping to
Requirements
Amber
No requirements exist in QC. A matrix of requirements is
maintained by the functional test team using Excel. This makes
traceability time consuming.
Mapping to
Manual Tests
Red
Manual and technical assets are duplicated in QC so traceability is
difficult and time consuming. Extra actions must be performed by
the end user to see the manual steps. Manual and technical assets
may get out of synchronisation with each other.
Coverage
Requirements Red
No requirements exist in QC so there is no technical solution
available to ascertain coverage of the technical assets.
Test Automation Score Card
Day 1
Initiation meeting with Tom Sewell to agree scope of the HeathCheck
Review (SQS & VISPL) of the QC and QTP framework implementation
Review (SQS & VISPL) of the Scenarios, Objects, Reporting and Error Handling
First day wrap up meeting with Tom Sewell
Day 2
Review (SQS & VISPL) of the Data, Config. File, Reporting, Execution & Development
Execution (SQS) of Portal and SAP Test Scenario
Initial Report Creation
Walkthrough of the findings report structure with Tom Sewell
Day 3
Report Finalised
Assessment of the automation
Recommendations and examples of findings
Issues identified during the HealthCheck
Walkthrough of findings (Vodafone & SQS)
Schedule of Activities
In order to conduct the HealthCheck, SQS requested access to the following
Access to all test assets
Access to the system under test and existing manual test scripts
Access to working end-to-end automation test scripts
Access to a sample of custom built drivers for the various automation tools
Access to sample reports generated
A demonstration of the test execution:
Against the applications under review
Using manual run
Via build triggered run if applicable
Availability of the VISPL Automation Team
Environment support during demonstration and test execution of the solution
Reasonable turn around times on queries raised whether against the VISPL
Automation Team or external (for example a manual test or solution design query).

All of the above was provided as requested by the VISPL Automation Team.
Pre-requisites and Support Criteria
Detailed Findings &
Recommendations
Recommendations
Recommendation Guidelines
Each of the recommendations made has a unique identifier
Each of the recommendations is clearly defined
Each of the recommendations is supported by a benefit
Each of the recommendations has an implementation priority assigned
Each of the recommendations has an implementation complexity assigned

Ordering of Recommendations in Table format
The unique identifier does not reflect an implementation order or importance
Many of the recommendations are linked and must be considered as a whole
Recommendations for Quality Center
ID Recommendation Benefit Priority Complexity
QC01
Implement the usage of the Detail field
in QC for each of the assets stored
Provides the end user with a clear
overview of the purpose of the test
High Low
QC02
Implement the usage of the Status field
in QC for each of the assets stored
Creates a development workflow and
improves visibility of the overall
progress
High Low
QC03
Implement the usage of the Design
Steps in QC for each of the assets
stored
Ensures the manual tests assets and
the technical test assets do not verge
apart from each other. Provides the
technical team a clear and easy way of
working
High Low
QC04
Store the requirements in QC which
are currently tracked using Excel
Provides dynamic traceability of
progress and enables tracking of
issues found
High Low
QC05
Move the Test Resources outside of
QC to enable the implementation of an
enterprise strength version control
solution (unless the upgraded QC
version control provides the necessary
functionality)
QC provides a basic level of version
control but not sufficient for
development of automated test suites
on an enterprise level with many
people working concurrently
Medium High
Recommendations for QTP Scripts
ID Recommendation Benefit Priority Complexity
SC01
Naming Convention with unique
identifiers applied to all assets
Simplifies the traceability of all assets
and improves the reporting in QC
High Low
SC02
Detailed headers and script comments
to be implemented to all assets
Improves the ease of use of the
technical assets for collaboration
Medium Low
SC03
Code repeated in each test which
should be a framework function i.e. Env
Variables
Simplifies and speeds up maintenance
activities required on all tests
Medium Low
SC04
All tests must be suitable for
environment portability. Load XML
environment variables from a central
file dynamically
Supersedes XML loaded manually to
every test which is a slow and
cumbersome process prone to error
High Low
SC05
Start up and teardown functions to be
applied to all assets.
Allows for quick changes to be made
globally to all test assets
High Low
SC06
Break away from functions and create
QC assets for each business
component
Will introduce overall framework and
flow control and ability to execute tests
from a failure point
Medium High
SC07
Use the QTP object to dynamically load
run-time setting at the beginning of
each test. Screen prints Never/Always
and global object timeouts
Applies consistency to every test case
running in the suite and removes the
need to execute VBS manually
High Low
Recommendations for QTP Objects
ID Recommendation Benefit Priority Complexity
OR01
Remove version numbers from current
OR in use
Removes the requirement to manually
associate this with every test using it
High Low
OR02
Changes the object description, dont
just use the SAP generated name
Makes scripts readable, intuitive and
allows other users to understand them
High Low
OR03
Object Windows must be distinct,
merge _2 and _3 as appropriate
Keeps OR tidy and reduces the
number of windows to maintain
High Medium
OR04
Objects must be distinct, merge _2 and
_3 as appropriate
Keeps OR tidy and reduces the
number of objects to maintain
High Medium
OR05
Object property and value pairs must
be improved and regular expressions
applied where appropriate
Clearer object definition and reduces
fails during execution
Medium Medium
OR06
ORs for SAP and Web should be
broken up into smaller files
Reduces the change of corruption and
allows concurrent development
Medium Low
OR07
Portal OR cleaned up to remove many
instances of Browser object
Keeps OR tidy and reduces the
number of objects to maintain
High Medium
OR08
Remove all local objects once merged
into the shared OR
Prevents tests failing on shared and
local conflicts
High Low
OR09
Store shared OR in a central location
and not on a C:\
Easily accessible by all developers High Low
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL01
Large single files over 12,000 lines of
code must be broken down into smaller
libraries of technical functions and QC
assets for business components
Collaborative development is
impossible and contention exists.
Using QC assets will improve the
traceability and maintainability
High High
FL02
Remove the version numbers from the
names of function libraries
Reduces the manual overhead of
continually associating to every test
High Low
FL03
Library files and individual functions
must have detailed headers which
include modification history
Provides a level of self documenting
solution and improves collaboration
High Low
FL04
Naming conventions must be improved
and include SAP TCode where
appropriate
Improved script readability and ease of
debugging when errors occur
High Medium
FL05
A Version Control solution must be
adopted, taking code out of QC if
necessary
Improves concurrent development,
traceability, multiple environment
execution and reversion where
necessary
High High
FL06
Create traceability with the manual
assets stored in QC by creating
technical assets in the same QC entity
Improves the overall knowledge of the
team. Keeps manual and technical
assets in sync and improves
traceability
High High
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL07
Dynamic synchronisation must be
applied to all business components and
all hard coded waits removed
Currently there is limited or none. The
standard QTP object synchronisation
points are not robust enough for
unattended execution and porting the
scripts across different test
environments
High Medium
FL08
Login functions must handle multiple
OPCO logins for all authorisation levels
These should be stored in an external
file
Medium Low
FL09
An overall flow control must be
implemented into every business
component. Currently QTP simply
executes linear with no checks to
ensure events are happening as
expected to allow control to pass back
to the calling test for error reporting and
handling
This will prevent tests from failing
unexpectedly and provide a much
greater visibility of what caused the
failure in the QTP execution report
High High
FL10
Evaluate all reporter message to
ensure failing steps are reported
correctly.
Found an example of code which fails
but report message contained a pass.
An If / Else statement was completely
bypassed as SAP was on the incorrect
screen
High Low
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL11
Remove the dependency on the QTP
Recovery Manager. This should be
used as a catch all not an every
instance solution. These should be
handled in the code itself
Much greater control of the error
handling and provides far more detail
in the QTP execution report as to the
cause of the failure. Can then also
incorporate screen captures
High High
FL12
Validation and checkpoints must be
applied to all business components
This provides detailed feedback as to
why tests failed and allows tests to run
in batches unattended without the
need for constant monitoring
High Medium
FL13
Error detection, error reporting, error
handling and recovery must be applied
to all business components.
QTP Run settings updated with a VBS
fail to suggest Test Stops on error, this
is only part of the necessary error
handling. Tests must also reset
conditions i.e. close browser and then
continue to the next test cleanly
High High
FL14
An overarching function is missing to
handle and recover from failure
Inbuilt QTP setting used to capture
screenshot on failure. This is typically
unreliable and should only be
incorporated in the code where failures
have been anticipated to occur
High Medium
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
FL15
Improve the robustness of the business
components by adding additional
validation to the objects before use.
Creating overload functions in QTP
could include this and many other
checks i.e. Enabled, Visible etc. As
they are overloaded no change to the
end user so scripts can still be
recorded in QTP.
High High
FL16
Function to create HTML report
requires lots of additional code in the
functions and should be included in the
overloaded functions.
This will simplify the script creation and
ensure consistency throughout over
business component
Medium High
FL17
HTML creation functions must also
upload to QC RUN table.
Reduce manual overheads High High
FL18
HTML functions for automated
reporting must be included in
overarching function to handle errors
Ensures these are created and
uploaded during test runs which
encounter errors
High Low
Recommendations for Test Data
ID Recommendation Benefit Priority Complexity
TD01
OPCO User Accounts are stored on a
central file governed by environment
and country
Easy for the end user to maintain and
provides versatility for multiple
environment execution
Medium Low
TD02
Different tabs for OPCOs should be
amalgamated to single sheet and QTP
function created to select appropriate
row
Reduces overall maintenance as there
are less Excel sheets to manage
Medium Medium
TD03
Needs to have a unique identifier for
each Excel sheet which map to the test
Improves visibility and traceability High Low
TD04
Additional iterations of data should be
incorporated into the Excel data sheets
to test more data permutations
Incorporating additional data sets
provides greater coverage easily with
applying a unique identifier to the data
row and mapping to test name.
Current constraint in the framework
does not allow this
Medium Low
Recommendations for Test Execution
ID Recommendation Benefit Priority Complexity
TE01
6 to 10 dedicated QTP servers hosted
virtually so accessible from anywhere
load balanced using QC host groups
instead of the 4 machines currently in
use
Utilises more of the 20 QTP licenses
available and provides greater
flexibility from the 4 machines used
currently which are also the day to day
machines of the team. Tests can then
be developed and executed
concurrently

Removes the dependency that the
user must be present to monitor the
execution

Removed the dependency of the
Developer running their own script
High Low
TE02
Hardcoded C:\ path in the HTML
reports must be removed
This must be addressed to allow many
machines to run unattended and see
results from anywhere
High Low
TE03
HTML report is uploaded dynamically
to QC in the same way manual tests
do, these should be uploaded to the
RUN table / Execution history
Ensures results can be accessed from
anywhere and not just the machine
they were executed on. Provides clear
and concise audit trail
High Medium
Recommendations for General Adoption
ID Recommendation Benefit Priority Complexity
GA01
QTP Patching level should be brought
up to the latest version
Issues already identified by HP will be
addressed
Low Low
GA02
Development work items must be
visible to everyone so its clear who is
working on what changes
Greater visibility of the changes
planned and progress tracked against
each. Allows work to be decided on a
priority basis
Medium Low
GA03
Solution documentation must be
created for the framework
Allows up skilling of anyone who
needs to develop, maintain or execute
tests
Medium Low
GA04
Solid Version Control solution required
which may include moving outside of
QC
Unable to develop and execute tests
concurrently so different versions of
QTP code base must be maintained.
QTP Environments in General Options
along with dynamic library loading in
QTP 11 allow the developer to quickly
change environments and versions
High High
Implementation Plan
Implementation Plan To be followed as a pilot phase
Implementation Area Detailed Description
Build a Core Framework
1) Object Identification Mechanism to be streamlined
2) Object Repository to be re-structured
3) Custom functions to be built to replace QTP inbuilt methods
4) Incorporate Error handling, Object Validation, Screen Capture for evidence, Object sync
and Checkpoints in the scripts
5) Reduce the large Function Library by moving the "Application Specific" functions to
modular QTP scripts
Test Plan
1) Convert the Application Specific functions along with the new framework into individual,
modular, robust QTP scripts
2) Include the design step for each T-code - for better readability and review by Non-
Technical/ Business Users
Test Lab
1) Update the scenario details, description, Objective, Pre-requisite
2) Build the end-to-end scenario by calling the modular QTP scripts into Test Lab and
setting the execution flow
3) Incorporate custom reporting to generate a report at the end of each scenario execution
4) Datasheet consolidation
Scheduling/ Host Management
1) Get the physical assets in place (20 Concurrent user license for QTP available)
2) Prepare a scheduling strategy to run the scenarios overnight where possible to better
manage the execution timelines
SQS Group Limited
Darryl Kennedy
Principal Consultant
Email : Darryl.Kennedy@sqs.com
Tel: 07950 128 060

SQS Group Limited
82 King Street,
Manchester, M2 4WQ

Internet: www.sqs-uk.com
Internet: www.sqs-group.com