Professional Documents
Culture Documents
W
PR dow
BE CTIC & .N
in
A s
ST ES ET
:
VOLUME 5 • ISSUE 10 • OCTOBER 2008 • $8.95 • www.stpmag.com
www.empirix.com/freedom
VOLUME 5 • ISSUE 10 • OCTOBER 2008
Contents A Publication
14
C OV E R S T ORY
‘Till Hacks Do Us Part’; Keep
Security and Quality Together
Minimize the risk of security vulnerabilities with security testing throughout
development–from grave-digging to lightning strike. By Danny Allan
Depar t ments
29
Picking Out
7 • Editorial
If only a tester’s conference were like a soft-
ware application.
24 Why Automation
Projects Fail
How to keep clear of the obstacles that
33
Two Cents
13 • ST&Pedia
Industry lingo that gets you up to speed.
CHRISTOPHER JUDD
FREE ADMISSION
DWIGHT DEUGO W-4 Develop Better Java EE Applications With
Eclipse Web Tools Platform
TO THE EXHIBIT HALL
W1 Eclipse 101 for Java Developers Visit the EclipseWorld Exhibit Hall for a floor full of demos!
106 Beat Those Java Dependencies: Extend the Find out the latest plug-ins, who is offering what, and
101 First Steps for Building and Deploying Web Tools Platform With Facets
become a better Java developer.
Eclipse RCP Applications
206 Interacting With Relational Databases
EXHIBIT HALL HOURS:
OVER 60 CLASSES TO CHOOSE FROM! Wednesday, October 29 2:30 pm – 7:00 pm
Thursday, October 30 10:00 am – 1:15 pm
DOWNLOAD THE COURSE CATALOG TODAY!
www.eclipseworld.net
PLATINUM SPONSORS SILVER SPONSOR MEDIA SPONSORS PRODUCED BY
www.code-magazine.com
GOLD SPONSORS
Ed Notes
Copy Desk
Adam LoBelia
Diana Scheben
alan@bzmedia.com
Contributing Editors
Matt Heusser
Chris McMahon
Joel Shore
Booooston?
ART & PRODUCTION As I write this, BZ Media’s have the ability to test
Art Director conference group and I your code from the very
LuAnn T. Palazzo
lpalazzo@bzmedia.com
are gearing up for the first character of the very
Software Test & Perform- first method (not that you
SALES & MARKETING ance Conference, which is would, of course).
Publisher taking place Sept. 24-26 at As a tester, you could
Ted Bahr the Marriott Copley Place even be involved in the
+1-631-421-4158 x101
ted@bzmedia.com
in Boston. I’m excited to requirements, design and
have the conference right initial development phas-
Associate Publisher in the city rather than in es. And you should. Be-
David Karp
+1-631-421-4158 x102
Cambridge, as in prior cause that’s when many of
years. Not that there’s any- Edward J. Correia the tester’s concerns can
dkarp@bzmedia.com
thing with Cambridge; it’s a beautiful best be addressed. Particularly those of
Advertising Traffic Reprints
Liz Franklin Lisa Abelson
place and our conference venue was security.
+1-631-421-4158 x103 +1-516-379-7097 nestled along the Charles River. Very And that’s the subject of our lead fea-
lfranklin@bzmedia.com labelson@bzmedia.com scenic and tranquil. But being right ture by security expert Danny Allan, direc-
inside the city, I think, will give the tor of security research at IBM Rational.
List Services Accounting
Lisa Fiske Viena Ludewig conference a joie de Danny provides real-
+1-631-479-2977 +1-631-421-4158 x110 vivre that was absent world methods of inte-
•
lfiske@bzmedia.com vludewig@bzmedia.com across the river. grating the testing
As we prepare to effort with every aspect
READER SERVICE
Director of Circulation Customer Service/
produce the confer- of development, from
Agnes Vanek Subscriptions ence in a new venue, requirements through
+1-631-443-4158
avanek@bzmedia.com
+1-847-763-9692
stpmag@halldata.com
it occurs to me that As a member of a to post-deployment
there’s no real way to maintenance.
Cover Photograph Courtesy of Sideshow Collectibles,
test any of the hotel’s development team, you By the time you
Thousand Oaks, CA facilities in advance read this, the confer-
—to test our confer-
ence “software” and
have the ability to test ence will be over, so I
can’t invite you to sit
processes before de-
ployment, as it were.
your code from the in on a class by Danny
Allan. He’s scheduled
Sure, we can tour the to teach two one-hour
facilities, see and
very first character of sessions on security
President BZ Media LLC
Ted Bahr 7 High Street, Suite 407 hear from the host, vulnerabilities and
Executive Vice President Huntington, NY 11743
talk about what they
the very first method. Web 2.0 concerns.
Alan Zeichick +1-631-421-4158
fax +1-631-421-4130 offer in terms of Perhaps you hap-
www.bzmedia.com
info@bzmedia.com
I was surprised you did not include I’m always looking forward to new interesting issues of Software
the test automation tool Watir in Test & Performance.
your article. Watir has been down-
loaded more than 85,000 times and Vladimir Belorusets, PhD
is easily one of the most widely used SQA Manager, Xerox
tools for automating functional tests Palo Alto, CA
of browser-based applications.
Pete Dignan to write documents and not take offense. Maybe you can even
and e-mails for address in a future issue whether we
Thanks for the information on the various clients that were easy to understand and should draw the line between QA, test-
tools. Providing the original web site for reduced confusion," she says. ing, and good old-fashioned copy editing
the source of your searching is helpful. I Of course, as anyone with a basic grasp and proofreading (and no, I do not think
found your brief description of the various of grammar should know, it would be even that running Spell-Check suffices,
tools useful enough for me to consider nicer if there were a way to write documents although it’s better than nothing).
investigating some of them. My company and e- mails for clients that were easy to Anne Simoncelli
has adopted TestLink as a test case repos- understand and reduced confusion. Relex Software, Greensberg, PA
itory and execution tracking tool. We have Ah, the subjunctive - For most people,
adapted to the quirks of the tool and I have it would seem, it's an inflammation of Regarding “Open-Source Tools: Asked
found it to be useful but it could use some the clear lymphoid tissue membranes that and Answered,” Test & QA Report, Aug.
more enhancements that are currently in cover the white part of the eyes and line 5, 2008, (www.sdtimes.com/link /32664):
the queue waiting for the funding that is the inside of the eyelids. As someone mentioned WatiR we should
necessary to make the changes. I now have Reg Blank now also mention WatIN (watin.source-
another web site to add to my list of useful forge.net), which is a Web application
places to visit. "And good written communication skills testing tool for .NET languages, and of
Thanks again. start with a command of the language." course it is free. It works similar as
Gretchen Henrich Is that a sentence? Good written? Funny RANOREX (www.ranorex.com) which
article.... the article is about writing well, was once also free but now they detected
You missed the eValid solution, which but is poorly written. Good one. they've built something that is worth some
goes them all one better by putting ALL Randall Karl money. WatiN also provides a recorder.
of the test functions inside a browser. Even though we all know that cap-
www.e-valid.com CATCHING UP ture/replay has little to do with mature
Edward Miller, President Regarding “Playing Catch-Up With This test automation, recorders are still appre-
www.e-valid.co, San Francisco, CA Week's Report,” Test & QA Report, August ciated and useful if you use it to help you
26, 2008, (www.sdtimes.com/link/32764): identify how objects are addressed.
ME RITE GUD, GET JOB,YAH? Maybe it’s not too late to edit your article Cheers.
Regarding “Good Communication Starts – it’s “pore” over, not “pour” over. (That is, T.J. Zelger
with Good English,” Test & QA Report, unless you are actually talking about spilling Audatex Systems, Switzerland
Sept. 2, 2008, (www.sdtimes.com/link/327 your energy drink/beer/wine /whatever
97): So, the woman who wants to explain over your technical manuscripts…) FEEDBACK: Letters should include the writer’s
the importance of the use of correct Call me fussy. I like to think that prop- name, city, state, company affiliation, e-mail
grammar to all and sundry doesn't know er word choice, spelling, grammar, and address and daytime phone number. Send your
the difference between a conditional and punctuation fall under the Testing head- thoughts to feedback@bzmedia.com. Letters
become the property of BZ Media and may be
a hypothetical, eh? ing, and so they’re fair game. Hopefully
edited for space and style.
"It would be nice if there was a way you will find my nitpicking instructive,
Application Security
Neither of us is a professional security tester, birth, or social security number. To do this,
but we’re both enthusiastic amateur security the criminal sets up a realistic-looking website,
testers. Security testing is a field both broad similar to a legitimate site, then attempts to
and deep, so this issue of STPedia we focus on lure prey to the site with e-mail “alerts.” The
the security of computers exposed to the gen- victim sees the alert, clicks and logs into the
eral Internet in three ways –possible exploits, fake site using their personal information The
Web technologies and the security techniques criminal saves that information and can then
used to prevent them. Matt Heusser and gain access to the real site to transfer balances,
Chris McMahon arrange loans, and so on.
Possible Exploits
BLACK HAT SCAN (PORT SCAN)
Person responsible for malicious attacks on
computers and networks. Also known as a
Q: What would your
answers be?
The use of a script to quickly identify all open
ports on a system. Many intrusion detection
‘Cracker.’ Did you exhaustively test systems are sensitive to port scans. Many black
this? Are we doing an SVT hat tools are designed to avoid intrusion
CRACK after our BVT? Does the detection systems.
Software that tries to log in using dictionary performance testing pass?
entries. The popularity of such software in the What are your equivalence SQL INJECTION
1980s gave rise to password security policies classes? Which heuristics
SQL is a database language; SQL injection is
requiring capitalization, numbers, and non- an attempt to gain access to a Website by mod-
are you using?
word characters. ifying submit fields. The most common
method is to change the username and pass-
CROSS SITE SCRIPTING (XSS)
Some Web sites, such as forums, allow users to
A: ST&Pedia will help
you answer questions
like these and earn the
word fields to something like:
write text for others to view. A black hat might respect you deserve. uid’ or ‘hello=hello
include some JavaScript in a forum post that
takes control of your browser to download a Upcoming topics: If the SQL is generated dynamically, it may
virus, re-direct to a specific website, or be used as return TRUE and log the user in, despite hav-
a pawn for a denial of service attack. (See below) November ing an invalid password.
Testers Choice Awards
DENIAL OF SERVICE (DOS) Web Technologies
December
To make a large number of independent serv-
Test Automation
PORT
ice requests simultaneously in an attempt to Internet protocols agree to communicate,
disrupt the service for legitimate users. A January 2009 with each element of the communication
black hat may attempt to gain control of mul- Change Management, ALM assigned a number between 0 and 65535. That
tiple independent systems and do a synchro- number is a port. Some ports are reserved for
nized DOS attack; we refer to this as a distrib- February certain uses, such as port 80 for HTTP. The
uted denial of service, or DDOS. Tuning SOA Perfomance SQL Slammer worm exploited an error in
Microsoft SQL Server communicating over
MAN IN THE MIDDLE (MITM) March port 1434. Exposing one’s database to the
Web Perf. Management
One type of attack is to intercept communica- Internet is in itself a security risk (see firewall.)
tion - such as a username and password – April
between a sender and a recipient. Unlike Security & Vuln. Testing ROOT
phishing, this attack does not seek to imper- The root user on Unix-like systems controls
sonate either sender or recipient, but seeks the entire machine. The phrase “to get root”
Matt Heusser and Chris McMahon
only to capture and exploit private informa- is the goal of a class of security attacks regard-
are career software developers,
tion. testers and bloggers. They’re col- less of operating system.
leagues at Socialtext, where they
PHISHING perform testing and quality assur- Prevention
A type of fraud, phishing is a process by which ance for the company’s Web- CERTIFICATE
a criminal attempts to obtain sensitive infor- based collaboration software. An independent, third-party guarantee that
mation such as a user login, password, date of continued on page 37 >
OCTOBER 2008
understand the fundamentals of building functional,
secure and compliant software and then be responsible If Hackers
for regular testing in the quality assurance process. Any
emergent issues can be considered software bugs and
defects, and require testing by the quality assurance
Come At Your
team.
and the ways they can be addressed, it for the most common and risky securi-
TABLE 1: FIRST STEPS
can start to think about how to effec- ty issues, it frees them to focus on the
Phase Action tively introduce security into its inter- more obscure while maintaining an
1 Employ external security consult-
nal software development lifecycle. audit of the common issues.
ants to do security assessment
There are two key concepts when The second concept to consider is
2 Security team implements pre- implementing this process: increment iteration. It is not enough to roll out
deployment application vulnerabili- and iteration. these tasks as a one-time action within
ty scanning tool and begins com- Security awareness and testing can be the SDLC. The most powerful mecha-
municating results
effectively addressed through an incre- nism for increasing security is to
Security team begins logging secu- mental introduction to the security ensure that these actions are consis-
3
rity defects into defect tracking process and testing strategy. The phased tently repeated within the process. By
system approach follows the five steps in Table 1. enabling security testing in the func-
As the organization reaches the tional or performance test plans of the
4 Quality assurance and security later phases, it should continue to quality assurance team, the repetitive
team collaborate to run automated employ external security consultants nature of this work ensures that appli-
security tests within scheduled
for security assessments and penetra- cations are repeatedly tested for secu-
regressions
tion testing of more sensitive and reg- rity issues before being deployed.
5 Quality assurance team configures ulated applications. It might also It is also important to consider some
and runs application security tests, choose to alter the role of the security essential foundations that enable this
passing security issues and fix rec- team to perform random audits of the incremental and iterative security initia-
ommendations into the defect applications in the pipeline to ensure tive to be rolled out together: education,
tracking system
that the process is working as planned. corroboration, consolidation and automa-
The overall goal is to use tion. Enablement is not
viewed pages, all submitted form val- the security team as appli- just putting the tools in
ues and all user keystrokes. cation security architects, the hands of the develop-
A single injection flaw targeted at
the database within an application
could lead to the remote access of all
the data stored within the central data-
rather than security
assessment resources.
In parallel with this
phased introduction to
• ment and quality assur-
ance teams to ensure that
testing can take place.
One of the key challenges
base. It is important to understand security testing, it is also Security is no of securing software with-
however, that injection flaws can be valuable to roll out an in an organization is that
targeted at the file system, LDAP and incremental vulnerability different than those responsible for the
XML data stores. The potential for test plan – especially in creation and testing of the
large scale data theft is significant. phases four and five. It is any other aspect software are not security
Understanding the vulnerabilities difficult to introduce the practitioners and are
and their potential business impact quality assurance and of software therefore not familiar
allows the organization to make sound development teams to with many of the threats
decisions around whether to accept, the full scope of security quality. It is and attacks that are con-
mitigate or transfer the business risk. issues that may occur. To tinually being developed
facilitate a more priori- only treated and communicated
Securing the SDLC tized and practical through the security and
Once an organization recognizes the approach, it is a worth- differently. underground communi-
vulnerabilities that exist, the causes while exercise to control ties.
the number and type of Education is a signifi-
tests that are transferred
to the quality assurance
team, slowly growing
both their responsibility
• cant and important com-
ponent. The quality
assurance team members
who will be responsible
and knowledge over time. for security testing must
Choosing an initial set of security be aware of the security issues and pos-
issues that can be mitigated through a sess the required skill set to run the
similar remediation strategy makes for automated scanning tools in their
a strong first step. An example of this toolkit. And because turnover is a real-
is the decision to automate the two ity in today’s software development
most common vulnerabilities, cross- world, an automated Web-based train-
site scripting and SQL injection. Table ing system can be considered the most
2 lists the key vulnerabilities that can cost-effective means of ensuring this
be integrated over time. ongoing education.
Over the longer term, the security Corroboration is an equally essential
team will continue to be responsible practice. Ensuring the security team
for the full breadth of security issues, has a close working relationship with
but by passing on the task of testing the quality assurance team is a key ele-
OCTOBER 2008
MONSTER MARRIAGE
organizational costs. ý
account the increasing value of the answer to this simple question is usually ware quality assurance results in lower
data that is stored in many software that the quality assurance team has put
TABLE 1: THE FATAL ELEMENTS from the other. Instead I was perform-
ing a fuzzy comparison of various ele-
Software Source Code Elements Description ments and mentally weighing them
Statements Cause actions to occur.They are sequence dependent. until at some point I declared them
Instructions Signify the actions to take place. similar or not similar – copied or not
Control words Control the program flow (e.g. “if”, “case”, “goto”, copied.
“loop”). This is in fact what most expert wit-
nesses in software IP litigation do.
Operators Manipulate data (e.g. +, -. *. /).
Often one expert will testify that “in
Identifiers Reference code or data.
my 25 years of experience I’ve never
Variables Identify data. seen code that matches as much as this
Constants Identify constants. unless it was copied, therefore this is
Functions Identify code. copied.” Whereas the opposing expert
Labels Specify locations in the program. might say “in my 30 years of experi-
Comments For documentation purposes only. Cause no actions ence, I’ve seen code that looks a lot
to occur. like this, and thus I believe this is not
copied.” Not only did this kind of
ences. When we meet a new person, we tual property disputes. I had been con- examination bother me ethically –
judge their intelligence based on how tacted by an attorney who needed me because of its pure subjectivity – but
many features they share with other to examine thousands of lines of com- I’ve always believed that everything is
intelligent people we’ve met. Because of puter source code to look for cross- quantifiable, even Roschian cate-
this, IQ tests measure only one aspect of contamination (code in one program gories.
intelligence and for each of us it may be getting into another). Being paid by I decided that I could quantify the
an important or unimportant aspect. In the hour, this might seem like the amount of similarity between two pro-
addition to logic reasoning ability, we ideal job, but I grew bored pretty grams by comparing specific ele-
may consider aspects of intelligence to quickly as I skimmed over hundreds ments, which I called “software source
comprise observational powers, memo- of lines of code and ran code correlation.” In sta-
•
ry, diction, artistic skills, musical ability, simple checks on the tistics, correlation is 0
fluency in multiple languages, common files. I decided to auto- for completely unrelated
sense, perseverance, or combinations of mate the process by writ- variables, 1 for perfectly
these. Upon entering the class I dis- ing a small utility pro- identical variables, and -1
agreed with the professor, but by the
end I had been convinced.
gram.
I realized there were
After discussions for completely opposite
variables2. For our pur-
The idea of Roschian categories and certain aspects of the poses, there is no such
prototypes stuck with me, but I never source code that I con-
with computer thing as source code that
thought I’d have a chance to apply it to sidered important. After is completely opposite (at
anything in my fields of study. That was discussions with other
scientists... least I can’t think
until I began working in software intel- computer scientists and of any such thing), so I
lectual property litigation and needed a expert witnesses, I deter- I determined only considered correla-
way to compare different programs to mined that source code tion values ranging from
find similarity or “correlation.” Slowly it could be divided into that source code 0 to 1.
became apparent that Roschian cate- basic elements. They’re I decided initially that
gories could be applied to software. shown in Figure 1 and could be divided there were three ele-
Even more importantly, I found that a described in Table 1. ments that I considered
mathematical framework could be cre- Once source code is into basic when comparing source
ated to measure similarity based on divided into these basic code – statements, com-
Roschian categories. I didn’t need to elements, I then thought elements. ments, and identifiers[8].
completely describe the program in about what I looked for I was mostly interested in
•
order to compare it with another pro- when comparing source statements that were
gram. And most importantly, this math- code from two different identical and comments
ematical framework could be applied, I programs to determine that were identical, but
believe, to more complex things in whether they were simi- identifiers that were iden-
other areas of computer science and lar. It occurred to me tical or nearly identical
also to areas of study outside the field. that comparing source code was a were also of interest because identifier
For example, I believe this can be used Roschian categorization process in names could easily be subject to a
to produce a better measure of intelli- that there was no single characteristic global search and replace to try to
gence than the IQ. that I was looking for, but rather a hide copying. Yet identifier names
number of characteristics. There was have a certain amount of useful infor-
Source Code Correlation also no particular element or a certain mation, so identifiers with similar
I first found the need to measure the number of matching elements that names could be a sign of copying.
similarity of computer programs in my would make me say the two programs With this in mind I defined three
work as an expert witness for intellec- were similar or that one was copied “dimensions” of correlation, each
pens, for example, when multiple pro- statement correlation and instruction piled and the resulting source code
grammers on a project need the same sequence correlation would be impor- can be compared to known source
function and each write it independ- tant whereas comment and identifier code. In that case, statement correla-
ently. It can also happen when pro- correlation would be less important. tion and instruction sequence correla-
grammers use third party code tion would be emphasized because
libraries or open source code in differ- Reverse Engineering identifiers and comments would most
ent places within a project. The ten- Reverse engineering is another field likely not exist in the decompiled
dency for duplication is higher in larg- where source code correlation may code.
er projects with many programmers, turn out to be important. The goal of
particularly if communication is poor. reverse engineering computer code is Software Testing
It also happens when code is modified to understand the functionality of the Source code correlation can be used
over a long period of time, especially code by analyzing it. Reverse engineer- to compare a new version of a pro-
when the modifications are imple- ing is typically performed on object gram’s source code to a previous ver-
sion to locate sections that have the
lowest correlation. These would be the
sections where the most significant
changes have been made, and test
engineers can focus on testing these
new sections. On large projects it may
not be possible for the programmers
to know where the most significant
changes occur without measuring cor-
relation.
Another testing area that could
benefit from the technology would be
comparisons of code output against so-
called golden files. With this kind of
testing, golden files are files that con-
tain program output from test cases
that have been manually verified for
correctness. When the software is
revised, the test cases are rerun and
the outputs are compared against the
golden files.
This technique may be the only
realistic way of testing certain kinds of
software, but it has major problems.
The most significant problem is that it
is rare for subsequent versions of soft-
ware to produce identical outputs.
Instead, the outputs vary from version
to version until the resulting output
mented by different programmers. code, where the source code may be file may be significantly different from
Clones make maintenance more “decompiled” into source code. Such the original golden file. Typically, test
difficult. Among the biggest problems decompiled code has few, if any, rele- engineers will look at differences and
with clones is when a modification vant identifier names and no com- sign off on the differences, but for
needs to be made to a particular algo- ments, since these are usually elimi- software of any complexity, this is an
rithm, especially if the modification is nated during compilation. Reverse error-prone process.
a correction to a bug. The modifica- engineering may involve understand- In the case of programs that output
tion needs to be made multiple times, ing source code that has been pur- program code, such as compilers, syn-
once for each clone. If the clones are posely obfuscated or for which the thesizers, and automatic code genera-
unknown, the bugs will continue to documentation no longer exists. tors, source code correlation can be
crop up during normal testing or use There are reverse engineering tools used to compare the output programs
in the field until all of the clones are to aid in this process3, and source code against the golden programs. In this
discovered and corrected. correlation can offer an important case, rather than looking for exact
Clone detection is the process of means. To reverse engineer source matches, correlation scores can be
automatically detecting clones in soft- code, it could be compared to code of used to determine where the output
ware source code1, 5. Source code cor- known functionality to determine sec- programs are most similar and where
relation is a great way of pinpointing tions that are similar. The source code they are most different. It is those
duplicate or near duplicate code could be compared to entire databases places where the code is most different
based on their high correlations. For of code where the code functionality is that a test engineer should examine
clone detection it would seem that known. Object code can be decom- more closely.
ven though many companies believe that automated software defects), and the usability of most-
almost exclusively, the trend is shift- zon.com is most user-friendly I acknowledge that this is a generaliza-
The goal here needs to be to tion, but while many developers con-
Elfriede Dustin is currently employed by improve our perceived quality. We can duct unit testing, and proponents of
Innovative Defense Technologies (IDT), a soft- accomplish this by focusing our testing test driven software development do a
ware testing consulting company specializing
on the most often used functionality really good job testing their software
in automated testing.
(which absolutely has to work without modules, there is still a lack of develop-
er integration or system testing. Some include: er, or Web system, log files may be writ-
might suggest shifting away from the • Building testability into the appli- ten on several machines, so it is impor-
testing focus and instead focusing on cation tant that the log includes enough
improving development processes. • GUI/Interface testing considera- information to determine the path of
While this is not a bad idea, even if we tions execution between machines.
implemented the best processes and • Adherence to open architecture It is important to place enough
had the most brilliant developers in standards information into the log that it will be
house: Software development is an art – • Adherence to documentation stan- useful for analysis and debugging, but
integration and system testing will dards to include standard ways of not so much information that it will
always be required, since most develop- documenting test cases and using not be helpful due to an overwhelm-
ers are concerned with their compo- the OMG Interface Description ing volume of information, which can
nents working only, without having the Language, for example; make it difficult to isolate important
big picture view of the system. • Following best practices, such as entries. A log “entry” is simply a for-
There are the human factors as to the Library Concept of Code Reuse matted message that contains key
why developers don’t system test: they Building testability into the applica- information that can be used during
don’t have time; they don’t specialize in tion and GUI/interface considerations analysis. A well-formed log entry will
testing and testing techniques; they are are discussed here, while the other best include the following pieces of infor-
busy churning out new code and func- practices listed will be discussed in the mation:
tionality, and it’s not their responsibility follow on article of this series. Class name and method name. This
to test the integration of the system can also simply be a function name if
code. As Josh Bloch, chief Java architect Building Testability Into the function is not a member of any
at Google one said: “Regardless of how tal- The Application class. This is important for determin-
ented and meticulous a developer is, bugs Software developers can support the ing a path of execution through sever-
and security vulnerabilities will be found in automated testing effort by building al components.
any body of code – open source or commer- testability into the application. Host name and process ID. This will
cial,” “Given this inevitability, it’s critical Building testability into the applica- allow log entries to be compared and
that all developers take the tion can be supported via tracked if they happen on different
time and measures to find various ways. One of the machines or in different processes on
and fix these errors.”
Developers
strapped cranking out
new features while trying
are
• most common ways to
increase the testability of
an application is to pro-
vide a logging or tracing
the same machine.
Timestamp of the entry (to the mil-
lisecond, at least). An accurate time-
stamp on all entries will allow the
to meet unreasonable It is important mechanism that provides events to be lined up if they occur in
deadlines. Again, getting information about what parallel or on different machines.
to market first is the key. to place enough components are doing, Messages. One of the most impor-
While the large corpo- including the data they tant pieces of the entry is the message.
rations focus only on information in are operating on, and any It is a description, written by the
R&D, there needs to be a information about appli- developer, of what is currently hap-
focus on R&D&T. the log that it cation state or errors that pening in the application. A message
Additionally, lack of are encountered while can also be an error encountered dur-
software development will be useful the application is run- ing execution, or a result code from
considerations for auto- ning. The test engineers an operation.
mated software testing, for debugging can use this information Gray box testing will greatly benefit
(i.e. building testability to determine where from the logging of persistent entity
into the application) is
another pitfall to avoid.
but not errors are occurring in
the system, or to track the
IDs or keys of major domain objects.
This will allow objects to be tracked
Automated software test- processing flow during through the system during execution
ing efforts can fail when
overwhelming. the execution of a test of a test procedure.
software development procedure. With these items written to the log
doesn’t consider the auto-
mated testing technolo-
gies or framework in
place. Software develop-
• As the application is
executing, all compo-
nents will write log
entries detailing what
file by every method, or function, of
every component in the system, the
following benefits can be realized:
• The execution of a test procedure
ers can contribute to the methods, also known as can be traced through the system
success of automated testing efforts, if functions, they are currently executing and lined up with the data in the
they consider the impacts to automated and the major objects they are dealing database that it is operating on.
testing efforts, when making code or with. The entries are written typically • In the case of a serious failure, the
technology changes. Additionally, if to a disk file or database, properly for- log records will indicate the
developers consider some of the select- matted for analysis or debugging that responsible component.
ed best practices described here, auto- will occur at some point in the future, • In the case of a computational
mated software testing efforts can reap after the execution of one or more test error, the log file will contain all
the benefits. The selected best practices procedures. In a complex, client/serv- of the components that partici-
A
pated in the execution of the test
procedure, and the IDs or keys of UTOMATION BEST PRACTICES
all entities used.
Along with the entity data from the A year-long IDT software automated testing survey was conducted; it was posted on com-
mercial QA user group sites; sent to tens of thousands of test engineers; and was posted on
database, this should be enough infor-
government tech sites, such as Government Computer News, Defense Systems, and
mation for the test team to pass on to announced during a webinarii we conducted, called “Automated Testing Selected Best
the development personnel to isolate Practices.” We received over 700 responses, world-wide. Here is a breakdown of the
the error in the source code. respondents’ demographics:
Following is an example of a log file
from an application that is retrieving a Over 73% of the respondents were from the US, while the rest was from other countries
customer object from a database: throughout the world, such as India, Pakistan, China, Europe, and others.
Nearly 70% claimed commercial as their organization type, while 10% claimed
Function: main (main.cpp, line 100) Government, and the rest claimed other, such as educational or independent.
Machine: testsrvr (PID=2201) For 40% the size of the organization was less than or equal to 300 employees, 60%
Timestamp: 8/6/2009 20:26:54.721 claimed an organization size of above 300 employees.
Message: connecting to database [dbserver1, cus-
tomer_db] The outcome of the survey showed that the value of automated software testing is gener-
Function: main (main.cpp, line 125)
ally understood, but often automation is not used or it fails. In the survey we asked respon-
Machine: testsrvr (PID=2201)
dents why in their experience automation is not used and the largest percentage respond-
Timestamp: 8/6/2009 20:26:56.153
Message: successfully connected to database ed with that Automated Software Testing does not get implemented due to lack of
[dbserver1, customer_db] resources, i.e. time, budget, skills.
Function: retrieveCustomer (customer.cpp line 20) Closely related to the above questions, we also received feedback as to why automation
Machine: testsrvr (PID=2201) fails. The highest percentage responded that many Automated Software Testing efforts fail
Timestamp: 8/6/2009 20:26:56.568
and tools end up as shelf-ware; and while 72% state that automation is useful and man-
Message: attempting to retrieve customer record
agement agrees, they either had not implemented it at all or had limited success.Their rea-
for customer ID [A1000723]
son for not implementing Automated Software Testing was:
Function: retrieveCustomer (customer.cpp line 25) • 37% lack of time
Machine: testsrvr (PID=2201) • 17% lack of budget
Timestamp: 8/6/2009 20:26:57.12 • 11% tool incompatibility
Message: ERROR: failed to retrieve customer • 20% lack of expertise
record, message [customer record for ID A1000723 • 25 % other (mix of above, etc.)
not found]
This log file excerpt demonstrates Here are various quotes providing reasons for limited automated software testing success
or failure:
a few of the major points of applica-
• “We have begun implementing, but aren’t allowed significant time to do so”
tion logging that can be used for • “Have implemented some efforts but lack of time, budget and resources prohibits us to
effective testing. fully perform this function”
In each entry, the function name is • “The company has previously implemented automated testing successfully, but this was
indicated, along with the filename and years ago and we currently don’t have the time or budget to re-implement”
line number in the code where the • “I’m the only one automating (so have some automation), but spend too much time on
entry was written. The host and New Feature release, need more people”
• “Accuracy of automated processes are the largest issues we have encountered”
process ID are also recorded, as well as
the time that the entry was written. Our survey results match what our experience has shown over the years: many agree that
Each message contains some useful automated software testing is the best way to approach testing in general, but there is often
information about the activity being a lack of budget, time, or experience available to execute successfully.
performed, for example, the database
server is “dbserver1”, the database is Additional reasons why automated software testing fails include:
• Research and Development does not generally focus on testing (manual or automated)
“customer_db” and the customer ID is
• Myths and Misperceptions about Automated Software Testing persist
“A1000723”. • Lack of Automated Software Testing Processes
From this log, it is evident that the • Lack of Software Development Considerations for Automated Software Testing, i.e.
application was not able to successful- building testability into the application
ly retrieve the specified customer • The Forest for the Trees – Not knowing which tool to pick
record. • Lack of Automated Software Testing Standards
In this situation, a tester could
examine the database on dbserver1,
using SQL tools, and query the cus- The tester is now not only report- scripting language and allow for script
tomer_db database for the customer ing a “symptom,” but along with the playback for baseline verification.
record with ID A1000723 to verify its symptom can document the internal When automated software testing tools
presence. application behavior that pinpoints interact with any type of display console
This information adds a substan- the cause of the problem. and Visual/GUI interface testing is
tial amount of defect diagnosis capa- required, the following recommendations
bility to the testing effort, since the GUI/Interface Testing should be considered, because cap-
tester can now pass this information Recommendations ture/playback tools are sensitive to any
along to the development staff as part Capture/playback tools record the test of the following changes when doing
of the defect information. engineer keystrokes in some type of bitmap recording, for example:
• Control “font smoothing” or other GUI testing tool scripts are often invented and implemented but can’t
text characteristics should not be based on object properties and other be tested or is difficult to test how do
changed. GUI controls, and therefore it is best if we know its level of quality?
• Don’t change the color depth of the developers understand how the Testing and Development can
the application under test. GUI testing tools functions and how drive a business, so much attention
• Display settings need to stay the the scripts have been implemented, so needs to be paid to speeding up not
same. the impact of any of his changes to the only development efforts, but also
• If possible, keep the testing efforts.
default settings in the OS Test automation is key.
•
related to visual settings – When developing software,
use standard visual settings. developers need to consider
Developers need to under- testing efforts and build testa-
stand the impact to the automat- bility into the application.
ed GUI testing scripts before When developing software, developers Additionally, developers
making any GUI changes. need to understand the
Sure, GUI changes are
inevitable and many developers
need to consider testing efforts and impact to the automated GUI
testing scripts before making
will scoff at the idea that their any GUI changes. Other ways
development should be limited
build testability into the application. to improve software develop-
by any automated testing tool. ment that can aid automated
But once a GUI has been base-
lined, the developer should
consider the impact some cos-
metic and possibly unnecessary
• testing efforts will be dis-
cussed in the next article of
this series, such as adhering
to open software develop-
standard documentation. ý
changes can have on the auto- ment standards and using
mated testing scripts. scripts can be lessened.
Items for developers to consider To avoid some of the major automa-
are to ideally minimize modification of tion pitfall failures it is important that REFERENCES
object properties once they are base- R&D also considers testing not just 1. Recommended also by Redstone Software, makers
of Eggplant
lined and/or maintaining the control development of the latest and greatest 2. http://video.google.com/videoplay?docid=8774618
flow. technologies. If the best technology is 466715423597&hl=en
SPTechCon
The SharePoint
Technology Conference
The Best
SC M 4 You
Software Configuration Management activities put in place to track the evolu-
tion of software items in the develop-
ment of a system.
Systems: What’s Not to Like? David E. Bellagio and Tom J. Milligan,
authors of Software Configuration Man-
agement Strategies and IBM Rational
By Vinny Vallarine ClearCase: A Practical Introduction, (IBM
Press, 2005) encapsulate a good defini-
onfiguration management is 1. Software Configuration Manage- tion of SCM by quoting the IEEE
zation, and management philosophy of the open source paradigm since occurs, an Atomic Commit ensures that
• Producing management and product it would hinder our ability to mod- all the changes are made or none at all.
information concerning the status of ify the source if needed. Non Atomic Commit systems run into
baselines, change control, tests, releases, 3. Seamless Integration with a defect trouble when, for example, during a
audits, etc. tracking tool commit of a bunch of files, a network
4. Eclipse and IDE support. connection is lost. The repository is then
Benefits of SCM 5. A quick learning curve for devel- put into an unknown and unstable con-
At a high level, a good SCM implemen- opers. dition. An Atomic Commit system would
tation should provide you with: 6. A tool that was widely supported in ensure that, since the network connec-
1. A secure repository to store your the community. There is nothing tion was lost before the entire commit
artifacts. worse than “Googling” a question finished none if the commit changes
2. A stable yet responsive workspace. about a problem you’re having make it into the master repository.
3. The ability to take a snapshot of with a particular piece of software • CVS was the only tool that did not
your baseline at incremental proj- and you get only 7 responses, and fully implement the atomic com-
ect milestones and “tag” it with a they’re in another language! mit model. This was a big minus in
known identifier. the CVS column.
4. The ability to revert to any of these The Tools Evaluated Intuitive Tags: Describes whether
“tagged” repository snapshots at We ended up evaluating a number of meaningful, human-readable tags can
anytime for reproducibility and different products to host our compa- be given to a particular revision. For
redeployment of a software build. nies SCM needs. There are plenty of example, “Latest Release 2.3 to DoD
5. Allow for simultaneous updates vendors offering very good SCM solu- Customer for contract num XYZ.123.”
/changes to artifacts in the reposi- tions but many of them can be eliminat- • All tools used Intuitive Tags. Sub-
tory. For example, it should provide ed immediately for not satisfying a par- version’s implementation of a “tag”
a controlled environment for hav- ticular need. Since we were mainly inter- has been criticized as being more of
ing multiple developers modify the ested in the open source community, a “cheap copy” but still appeared to
same file at the same time. There our options were narrowed down quite a satisfy our needs in this area.
are two main approaches that SCM bit. After researching the available prod- Web Based Interface: Some packages
products take towards solving this ucts, we identified the list of features come with a built in web interface. A
issue via its Concurrency Model: that we considered important and exam- web interface allows for the posting of
• Merge – Any number of users ined 4 tools with these features in mind. data to a web site for audits/analysis. A
may write a file at any time. The four chosen packages are: web interface could also allow for con-
The master repository may • CVS – Dick Grune trol over the SVN from any machine
spit out a “conflict error” if a • Subversion – CollabNet with a web browser.
user attempts to check a file • Mercurial – Matt Mackall • Subversion comes with an Apache 2
into the repository that is not • Bazaar – Canonical module built in. It also offers seam-
based off the latest revision. less integration with TRAC.
The user may then merge Features • Mercurial comes packaged with a
his/her local changes manu- Below you’ll find a list of the features web server.
ally or have the system that we considered, a brief explanation • CVS allows for the integration of
attempt the merge. of this feature and a description of how csvweb and ViewVC.
• Lock – One user at a time is each tool addresses this feature. Many of • Bazaar can be used with any simple
allowed to write a file. The the supporting applications referenced web server along with webserve, log-
master repository hands out below are not fully elaborated on as they gerhead or Trac.
locks one at a time. A user fall outside the scope of this writing. Development Status: Whether the appli-
must give up the lock for Ease of Use/Learning Curve: Is the cation is still being aggressively devel-
another use to edit the SCM package intuitive and overall easy oped, or is simply maintained with occa-
same file. to use. After all, the developers will be sional bug fixes incorporated. An appli-
6. The ability to audit and control interacting with it many times per day. Is cation that is being aggressively devel-
changes to our components. there a steep learning curve? oped is usually the most supported. The
More specifically, we needed: • Subversion, CVS and Bazaar seemed to developers of these applications are con-
1. A tool that adheres to the GNU be at the same level of “ease of use;” stantly adding new features and bug fixes.
General Public License (GPL) relatively easy. Many Website reviews Since the world of technology is con-
licensing model and was Open pointed to Mercurial as the more dif- stantly changing, these applications tend
Source ficult of the SCM tools. Subversion to address the latest needs of the devel-
2. Compatibility with Linux and had the largest “head start” here, opment community better than those
C/C++. We didn’t want to have to however, since some of us on the apps that are less actively developed.
install a proprietary compiler to team had prior experience with it. • Bazaar, Mercurial and Subversion are
build/setup the SCM software. Licensing: The license model the all actively developed.
Our developers know C/C++ and application corresponds to. These can • CVS is more or less in a mainte-
Linux well and having a SCM sys- be free or paid licenses. nance phase. New features are no
tem written in a proprietary lan- • All of the evaluated tools were free. longer being added.
guage would diminish the purpose Fully Atomic Commits: When a commit Programming Language: The lan-
end. These solutions seem to vary in the EOL characters for files so they the web for answers or do we want to
terms of ease of setup and use, how- match the EOL specific method for the spend the cash on a licensed product
ever. The details of these are outside OS in which it is used where the support will be formal?
our scope here, however. • Bazaar was the only package that Do we want a “tried and true” product
Speed: Describes the efficiency of its did not support EOL conversions. with known suitable features or do we put
branching, tagging and commit algo- Again, as we did with our defect our trust in an “up and coming” product
rithms. Also, a big factor affecting the tracking tool, we assigned a weight and with the potential of monumental
tools overall response time is whether a score value to the above features to enhancements over our older product.
it’s written in a compiled (C/C++) or come with the table below: Can we afford to pick a tool that may
interpreted language (Python). not be the best choice because it inte-
• Subversion and CVS seemed to have The Selection grates with another of our tools? Can
an advantage here seeing as As a result of our analysis, Subversion we afford to go back and re-evaluate
they’re written in C, whereas came out on top. Aside from the features our prior choice of tools to accommo-
Bazaar and Mercurial are written in listed above, one very important factor date this one?
Python. This analysis is simply remained. Our developers had experi- Balancing the benefits and liabilities
based on a compiled versus inter- ence with this application in prior jobs. of a tool is important in any choice. In my
preted language. The overall speed This was a big factor in determining “ease experience, there is nothing more valu-
of any SCM tool is dependent of use.” Picking the right tool for any job able that having a team that is aware of
upon its algorithms used. This is difficult and one needs to balance the the current options. A team of people,
aspect of performance wasn’t benefits versus the liability of any tool. In for example that read technical journals,
explored too deeply, however. this instance we had find the balance discuss new technologies with co-workers,
IDE Support: Describes whether the between the following questions and are generally interested in their field
tool has smooth integration with inte- Do we want a feature rich applica- will have a much better shot at picking
grated development environments.such tion even at the expense of ease of use? the right tool at the beginning of a proj-
as NetBeans, Eclipse and Visual Studio. Do we go with something that every- ect, where it’s the cheapest. Needing to
• All the SCM applications offer a vari- one knows and can start using immedi- change to another implementation of a
ety of IDE integration options. All of ately or we do look for something new, tool in the middle of a project can be
which integrate well with Eclipse, and possibly better. Will schedule allow detrimental. This could happen if
which was one of main needs. this “trial” period? needs/requirements change but should
By Prakash Sodhani
t has been well documented that devel- In many of the companies I have worked in,
can be as lengthy as the code under test otherwise it’s nothing more than a writing automated code as well.
itself. wild goose chase. Unless a tester has
I have been doing automated testing clear view of functionalities to be auto- 3) Testing
for years, and have been to a good many mated, creating the automation scripts The actual testing phase consists of
companies during that does not serve their two parts: Unit Testing and Integration
time. It’s interesting to intended purpose. It’s Testing. Let’s look at these two sepa-
•
learn people’s views of important to understand rately.
automation, which range what needs to be auto- 3a) Unit Testing. Unit testing involves
from manual testing with mated and use that infor- testing a single unit of functionality,
tools to simple record and mation to structure out the smallest possible component of
playback. I will explain It is imperative the automated tests. code. The purpose of this testing is to
some of key characteris- make sure the core components work
tics of automation and for people to 2) Code Generation fine in isolation. So, you pick a small
their relationship to Code generation refers to independent piece of code and make
development. Hopefully,
it will open the eyes of
understand that creating reusable automa-
tion scripts. Based on what
sure it works as per specifications. For
example: in development it may refer
people who have been business process needs to to a class, method, function, or a Web
pretending to know every-
automation is be automated, scripts are service call. You develop, test drive and
thing about automation created that can be run the associated unit with different
but in reality know close
not just record replayed again and again. parameters, both positive and nega-
to nothing. As is the case with devel- tive; to make sure it works as expected.
In a nutshell, the devel- and playback. opment, automated While creating test automation
opment process consists of scripts need to follow cod- scripts, you should structure your code
these distinct steps: You write code. ing guidelines and adhere in manageable units. I divide my code
1. Software Require- to standard practices. into small units called “Actions,” which
ment Analysis
2. Code Generation
3. Unit and Inte-
gration Testing
• It is imperative for peo-
ple to understand that
automation is just not
record and playback. You
are synonymous with classes in object
oriented programming. These classes
can be further subdivided into smaller
units, such as functions and methods,
4. Maintenance write code. The difference just like development methods and
Depending on the between writing code for functions. After the test script is ready,
company and project requirements, automated testing and coding for you test it with different parameters
there may be other steps added on or in development is that in testing you write and sets of data to ensure it works as
between, but automated testing has a not to develop something but to break expected.
one-to-one relationship with each and what is developed. All the considera- Let’s take an example to illustrate
every one of these steps. tions involved in development apply to unit testing correspondence. Let’s say
1) Requirement Analysis
Requirement analysis is always the first
step in a software development
process. The process is simple. Get the
requirements from the requirements
team and use them to design and
develop your code. Just as correct code
can’t be written without clear require-
ments, a correct quality approach can-
not be taken without correct require-
ments.
As much as requirement analysis
applies to almost every domain, it is
even more significant when it comes to
automation. I have been on more than
one occasion been in the evening and
told to start writing automation scripts
next day. Sometimes, it has even been
for projects for which I had no prior
information; I was just told that “every-
thing needs to be automated” within
few weeks.
As a tester, it’s your responsibility to
fully understand the expectations of
automation for a particular project,
you have an application with a login piece of code working fine in isolation Testing. You should have tested the
page. but failing when used in conjunction Login page with different sets of data.
Development. A developer will write with other units. Now you can include clicking on the
code to create the required compo- Let’s take an example to illustrate “Search” button as a part of your test
nents in the login page and ensure unit testing correspondence. Let’s say script. You can also add checks that
valid inputs are accepted and correct you have an application with a login ensure that clicking on the “Search”
results returned. He might put this page. When clicked on “Search” but- button brings up the home page (or
code in a function named “Login,” just ton in login page, it takes you to home search page, as appropriate). You run
as it might be called in many places. page of the application. this script with multiple sets of data.
Then he will plan to test the
“Login” unit with various sets 4) Final Check And
of data.
Testing. A tester will write
code with different verifica-
tion checks to make sure that
• Maintenance
There is always some mainte-
nance involved in anything you
do. It’s even more significant
the components in login The most successful projects I have when your code is used repeat-
page accepts valid inputs and edly and by many people.
rejects invalid ones. The worked on have involved testers from Development. Maintenance
tester might put this in a required in the development
function so that he can make the beginning and throughout. code is well known. Often in
it reusable across various projects, much of the total soft-
other automation scripts he
might write, and ease the
effort in maintaining the
code. Then, he might run
this script with various sets of
• ware life cycle involves some sort
of maintenance.
include making sure any
Activities
change. ý
er to bring about the right kind of
www.stpmag.com • 35
Best Practices
•
and decode text. The simplest tion and encryption. In other
cipher is a ROT (or rotation) words, you know you are talk-
cipher. In a ROT-1 cipher, ‘a’ ing to the correct party, and
changes to ‘b’, and so on. Such that no one else can overhear
encoded messages, if captured
by a third party, cannot be
Is there a performance hit with RSA the ‘conversation’ over the
world-wide-web. For example,
understood. Julius Caesar used
such a cipher to transmit
versus SSL? What vulnerabilities most websites that accept cred-
it cards use SSL or a scheme
coded messages during the like it.
Gallic wars. Germany’s Enigma
exist after the latest port scan?
Machine of World War II was a RSA
complex, multi-level ROT
cipher (There’s a free online
ROT cipher generator–at www
.unfiction.com/dev/tutorial
• A specific cipher, complex
enough for both business and
military applications. Wikipedia
refers to RSA as “one of the first
/ROT .html). great advances in public key
Web page is encrypted and transmitted cryptography.” Specifically, RSA pro-
FIREWALL using secure sockets. vides both encryption and authentica-
A software (or hardware) system tion – RSA messages cannot be created
designed to limit access from an out- NETWORK INTRUSION without a “private key.” As long as this
DETECTION SYSTEM (IDS)
encryption and digital “signing.” ý
side source. Firewalls are popular for key is protected, RSA can provide both
personal computers, but large organi- Software that monitors a network for
Index to Advertisers
Automated QA www.testcomplete.com/stp 10
Empirix www.empirix.com/freedom 4
Hewlett-Packard hp.com/go/quality 40
Ranorex www.ranorex.com/stp 34
Static Analysis,
For example, using static analysis as
an audit that occurs at later stages of the
SDLC only exacerbates its tendency to
drain development resources. Having
an inline process makes the analysis