Professional Documents
Culture Documents
Role
Capability
Documentation
Goal
Role
Test Runner
http://webtest.canoo.com A l l o w w e b a p p l i c a t i o n t e s t i n g t o b e a u t R u n s v i a H T T P
Role
Reporting
Role
Test Editor
X M L , H T M L , s o p h i s t i c a t e d a n y x m l e d i t o r o r i d e
Role
Repository
N o
Role
Role
How tests can load or query a database used N o by the SUT H t m l U n The means of interacting with the System i t SUT Interface Driver Under Tests Database
Role
Language
Database
Load data
Database
Clean data
J a v a , G r o o v y P o s s i b l e P o s s i b l e
Implementation
Native Language
Implementation
Java Accessible
Implementation
Ability to invoke .NET methods in target code ? base W e b API/WebUI/DesktopG U I UI Access to API and/or GUI or Web UI Other drivers/fixtures Other interfaces for driving SUTs .Net Accessible
Language
Data-driven
Language
Keyword-Driven
Y e s Y e s
Language Language
? N o
Language
Tabular
Language
Language
Multiple levels of Abstraction available Multiple levels of Abstraction - in same test variables as arguments of keywords variables can hold values returned by sut
Y e s Y e s
Language
Language
Language
Misc
Misc Misc
Misc
Misc
Y e s Y e s c o m m Free-form e commentary n interspersed with In line comments just like code comments are t s tests entered with // V e r y m a t u r e Maturity How mature is the tool? Y e s OpenSource Is there a free version with source code? N o Commercial Is there a paid version? Y e s Use tool to test itself Y e s One-button Install Does this also mean that abstractions for keywords can have parameters? (Rick) Mixed Auto/Manual Install N o
Misc
Misc
Mixed Auto/Manual
Misc Misc
N o P o s s i b l e ? P o s s i b l e N o t s o m u c h
Process
Process
Encourages communication
Reporting
Tagging/Categories Y e s Y e s Y e s Y e s
Reporting
Summary
Reporting
Clickthrough to failures
Reporting
Reporting
Reporting
Side-by-side.What does this mean? Seems very specific - is it a useful distinction? (Rick) ?
Repository
Builtin
Repository
SCM-Friendly
Repository Test Editor Test Editor Test Editor Test Editor Test Editor Test Editor
Storage format Record Tests Refactor Refactor Builtin keyword help Keyword Completion Argument Completion Updating tests automatically (from results? (Rick))
Indicate when keywords mispelled or wrong "Grammar-checker" args "Quick-Fix" Run test from editor Interactive execution
Test Runner
Command line
N o Y e s x m l N o N o N o N o N o N o x m l e d i t o r N o N o N o Y e s
Test Runner
GUI/Browser
N o Y e s
Test Runner
Test Runner
Run subsets of tests based on Tagging/Categories Run same tests via different interfaces Run same tests against different configurations
Test Runner
P o s s i b l e ? Y e s Y e s
Test Runner
-------------------
-------------------
-------------------
-------------------
Concordion
Cucumber
JBehave
http://www.concordion.org
http://jbehave.org/ http://wiki.github.com/aslakhellesoy/cucumber
Create a safety net for refactoring by separating the test intention (written in plain English) from the test implementation (written in Java), so the latter can change, as the system's design evolves, while the former catches regression bugs. Most other testing frameworks mix together the test intention with its implementation. Write user stories and acceptance criteria Write business scenarios (scenarios) in plain-text in structured English. files in any language.
JUnit (for the Java version of Concordion Command line, Rake, anyway) IDE
Any text editor Anywhere - stories are loaded via StoryLoader implementations - in classpath, from local or remote URL. Steps classes can use any Java framework. DbUnit is commonly used.
Anything
Instrumentation within HTML (for test case); and Java (for test implementation) Gherkin (plain text)
Within the Java fixture code you can do anything you like
Pluggable plain-text StoryParser. Default implementation is regexbased. Various @Before* and @After* annotated methods can be called. Inside those methods, any java framework for data loading can be invoked - DbUnit is commonly used.
Yes
Ruby, Java, Scala, Groovy, Javascript, Clojure, Ioke, .NET Yes http://wiki.github.com/asl akhellesoy/cuke4duke Yes
DbUnit - as above Java or any other language that you can access via Java API, e.g. groovy, jruby, jython, rhino.
Yes
Yes
No
Yes
No Plain language
Test is plain text, Execution framework is Java Yes Tabular input and output supported in different ways.
Yes
Yes
Singhal Variables are matched with text in story. Default implementation is regexbased
Scenario Outline
Yes
Yes
Mature
Mature
Yes No
Yes
Yes No
Yes
Yes
Yes
No
N/A
N/A
Yes
Yes
Yes
Yes
Indeed
Yes
Yes
Yes
Yes - customizable
No
No
As per JUnit
Yes
Yes - customizable
N/A
No
Yes
Yes
yes
HTML No No No No No No
Plain text + code (several languages) Plain text + code (Java) With Selenium, for No Selenium-based Steps No No --lang help In RubyMine In RubyMine No No ? No No
Abillity to do dry-run No Yes No Depends on editor When run in IDE via JUnit/TestNG
Yes
Yes Yes - whatever you want to add to Steps class Selenium/WebDriver classes exist
No
Yes
Yes
Yes
Possibly
No
Yes
Ant/Maven/IDE
No
Yes --profile
-------------------
-------------------
FitLibrary
Fitness-Slim
GreenPepper
http://sourceforge.net/projects/fitlibrary/
http://www.greenpeppersoftware.com
Detailed requirements in terms of business rules, objects and processes, test automation, table-based DSL
Use Automated Functional Testing as an "Executable Specification" mechanism that optimize requirement communication between customers/users/field experts and the developers team. All this in order to build the right software, the one needed.
Fitness
From a Wiki (XWiki, Confluence), from JIRA or from the IDE (Eclipse & Visual Studio plugin). Runner versions for Java, .Net and Ruby). Using continuous integration with Maven, Nant, MS Build.
Fitness
Fitness
Confluence, XWiki
Fitness
Yes
No
No
Yes
Multiple SUT configuration for a test suite. Configuration managed within the WIKI
Yes
Yes
No
Yes
Yes
Yes
Yes
Through API of language. Java fixtures/drivers for Swing, web/ajax (SpiderFixture) shell, email, PDFs, XML, web services, mocking web services, templating
API
Yes
Yes
Yes
Yes
No Yes
Usually, but also has plain-text form Yes. Procedural abstraction with defined actions with parameters, OO abstraction with classes (to organise defined actions and to make distinctions between different cases), business rule abstractions with multi-defined actions
Yes
Yes
Yes
Yes
Yes
No
Yes
No
Yes
mature
Yes No
Yes
All runners are opensource Pay for Server and technical support
Yes, extensively
Yes
No, not when used with FitNesse Yes, by running part of a storytest, but this is a little inconvenient
No
N/A
N/A
Yes Yes
Yes Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
No n/a Yes No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
-------------------
-------------------
GUIdancer
Jubula
PyWinauto
http://pywinau to.openqa.org /
http://www.guidancer.com
http://www.eclipse.org/jubula
Functional acceptance testing (blackbox) through the GUI to test programs as the end user will see them. Removal of programming effort from acceptance testing. Same goals as Jubula, but with added extras for professional use Use in agile processes (tests (code coverage, TestStyle, Mylyn created in parallel with integration, reporting) development).
See Jubula
Yes: Tests can be started from a command line, which can be called by e.g. Ant or CI tools
See Jubula. Additionally, long term BIRT trends can be generated from results.
In client itself (as summary table and full report tree). Also as HTML and XML reports.
See Jubula. Additionally, Mylyn tasks can be used for contextbased working. There is also a Test Style module for best practices for test structure etc.
In "Test Case Editor": tests can be added (referenced), removed and altered (e.g. order, names, components, data) Tests are stored in a database. Projects can be exported as XML files for version control
See Jubula.
Yes
Yes
Yes
Yes
See Jubula
Tests are written using pre-existing modules of test functionality. Test modules can be renamed so that tests can be read in natural language
See Jubula
See Jubula
Java
Java
Yes
Yes
No
No
See Jubula
Yes
Yes
Yes
Yes
No n/a
No n/a
No
No
See Jubula
Yes
Yes
Yes
Yes
Yes
Yes
See Jubula
Yes No
Yes No
Yes
Yes
Yes
Yes
See Jubula
Setup needs to be specified once as reusable module. Teardown can either be specified or the application can be restarted.
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
Yes
Yes
Yes
Yes
Yes
Yes
No
No
Yes
Yes
Yes
Yes
-------------------
-------------------
-------------------
Robot Framework
Sahi-Java
Robot Framework is a generic keyword-driven test automation framework for acceptance level testing and acceptance testProvide a Java driven development interface to Selenium tests web apps on any OS in any (ATDD) Sahi browser with any language.
Yes
Java
Yes
Yes
Yes
Java IDE
And database
Driver independent
Yes
Yes
Java
All of them
Yes
Yes
Python
Java + Javascript
All of them
Jython
Yes
Yes
IronPython
No
Yes
Pluggable
API, WebUI
API/WebUI
Yes
Yes
Yes
No
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Same as above
Yes
Yes
As above
mature
Yes No
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Needs to be coded
Yes
Yes
Multiple
No
1-page
Yes
Yes
Yes
No
No
Why?
Yes
Yes
Txt, html, tsv, rest No Yes with Robot IDE No Yes Yes
Java
In next release Yes (Selenium IDE) Java IDE No Yes Yes No Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse
Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse Yes, Selenium IDE. For RC, use an IDE like Eclipse
Yes
Yes
Yes
Yes
Yes
Yes
Multiple
No
Yes
No
Yes
Yes
-------------------
------------------- -------------------
Trace Analyst
Fitnesse
http://ulti-swat.wikispaces.com
Functional test of Swing and JavaFX UIs. Intended users: developers and testers
No
n/a
Yes
No
Yes
No
n/a
Yes
No
No
n/a
Yes
No
No
n/a
No
Yes
Yes
Yes
Yes
Yes
N/A
n/a
Yes
N/A
n/a
Possible
C#
Java
Java
Java
No
Yes
Yes
Java
Yes
No
In Future
Yes
WebUI
API
API
Doesn't work when Fitnesse run as a service Depends on test runner n/a
Fitnesse
Yes
Fitnesse
No
n/a
Yes
Fitnesse
Yes
n/a
Possible
Fitnesse
No
n/a
Yes
Fitnesse
Yes
n/a
Possible
Fitnesse
Yes
n/a
Possible
Fitnesse
Yes
n/a
No
Fitnesse
Yes
n/a
No
Fitnesse
n/a
Yes
Yes No
Yes No
No Yes
Yes
Yes
Yes
No
Yes
Yes
N/A
Yes
N/A
N/A
Yes
Fitnesse
Yes
Yes
Fitnesse
Fitnesse
Fitnesse
Fitnesse
Yes
Yes Yes, depending on test runner Yes, depending on test runner Yes, depending on test runner Yes, screenshots for test failures Yes, depending on test runner Yes, depending on test runner
Yes
Yes
n/a
Yes
Yes
Yes
No
Yes
No
N/A
Yes
Fitnesse
No
n/a
Wiki
Yes
n/a
Somewhat
n/a Yes Planned Planned Planned Yes Yes No Java IDE Java IDE Java IDE Java IDE Java IDE n/a n/a n/a n/a n/a n/a
No No Yes No
Fitnesse
Yes
n/a
Yes
Browser
Fitnesse
n/a
Browser
n/a
Yes
Fitnesse
n/a
Yes
-------------------
-------------------------------------
-------------------
Twist
http://www.thoughtworks-studios.com/twist
Twist is a collaboration platform in which business users can express testing intent (similar to requirements) in simple english and technical users can render these english statements executable using the inbuilt power of the IDE and supported variety of languages. Through this Twist promotes the notion of software specifications also becoming executable tests. Twist supports multiple industry standard drivers that can be used to implement test intent.
Yes any Java Driver. The 2.1 version comes bundled with Sahi, Selenium to test Web Applications, Frankenstien to drive Java Swing applications and SWTBot for Eclipse/SWT based applications
Through an inbuilt editor called "Scenario Editor" which support refactoring and reuse
Interact using any Java Driver. The 2.1 version comes bundled with Sahi, Selenium to test Web Applications, Frankenstien to drive Java Swing applications and SWTBot for Eclipse/SWT based applications
Java or Groovy
Yes
No
Pluggable Use any java driver, Version 2.1 onwards comes bundled with Sahi, Selenium, Frankenstein, SwtBot Currently, does not support any windows thick client applications. Currently support only Eclipse as an IDE
Yes
Supported
Yes, definitely
mature
No Yes
Yes
Yes Yes
Yes
Yes
Multiple
Yes
Yes
Yes
Yes
Yes
Text based wiki files Yes provided the driver supports recording Yes Yes Automatically evolves as you write scenarios and leverages on IDE capability to provide the support. Quick assist and auto complete for DSL Yes
Argument validation, and general syntax validation is possible Yes Yes Yes
Yes
Yes
Multiple
Possible
Yes
-------------------
http://watin.sourceforge.net/documentation.html
http://watir.com
.Net IDE
SciTE
No
No
DOM/JS
DOM/JS
.Net languages
Ruby
C#
Ruby
No
Yes
API
Roo
No
Roo
Yes Yes
Yes
Yes
.Net
.Net
Yes No
Yes
Yes
Yes
Yes
Yes
No
Depends on test runner .Net IDE .Net IDE Yes .Net IDE .Net IDE
.Net IDE .Net IDE Yes Yes (run tests in debug mode in .Net IDE) SciTE IRB
Yes
Yes
Yes
Yes
-------------------
-------------------
White
http://white.codeplex.com
Automate all windows applications except browser, if they support UIAutomation. Provide the driver to test and let the programming language of choice do the rest.
source control
n/a
yes
.net language
yes
C#
No/NA
Yes
Interpretation Tool does not support this functionality The tool doesn't support it directly but it can be done in some way. Not applicable to this kind of tool
Example
<Toolname>