You are on page 1of 60

Canoo WebTest

Role

Capability

Detailed description of capability or attribute

Documentation

Goal

Role

Test Runner

The means of choosing and invoking the tests to be run

http://webtest.canoo.com A l l o w w e b a p p l i c a t i o n t e s t i n g t o b e a u t R u n s v i a H T T P

Role

Reporting

How test results are reported

Role

Test Editor

How the tests are edited

X M L , H T M L , s o p h i s t i c a t e d a n y x m l e d i t o r o r i d e

Role

Repository

How the tests are stored

N o

Role

Role

How tests can load or query a database used N o by the SUT H t m l U n The means of interacting with the System i t SUT Interface Driver Under Tests Database

Role

Language

The language for expressing tests

Database

Load data

Test data can be loaded from a database

Database

Clean data

What does this mean? (Rick)

J a v a , G r o o v y P o s s i b l e P o s s i b l e

Implementation

Native Language

Implementation

Java Accessible

X M L y Ability to invoke Java methods in target code e s base

Implementation

Interface Interface Issues

Ability to invoke .NET methods in target code ? base W e b API/WebUI/DesktopG U I UI Access to API and/or GUI or Web UI Other drivers/fixtures Other interfaces for driving SUTs .Net Accessible

Language

Data-driven

Language

Keyword-Driven

Y e s Y e s

Language Language

Programming Language is used to write tests Fluent/Literate

? N o

Language

Tabular

Language

Language

Multiple levels of Abstraction available Multiple levels of Abstraction - in same test variables as arguments of keywords variables can hold values returned by sut

?Not sure what this means? (Rick)

Y e s Y e s

Language

Language

Language

Misc

Misc Misc

Misc

Misc

Y e s Y e s c o m m Free-form e commentary n interspersed with In line comments just like code comments are t s tests entered with // V e r y m a t u r e Maturity How mature is the tool? Y e s OpenSource Is there a free version with source code? N o Commercial Is there a paid version? Y e s Use tool to test itself Y e s One-button Install Does this also mean that abstractions for keywords can have parameters? (Rick) Mixed Auto/Manual Install N o

Misc

Misc

Mixed Auto/Manual

What does this mean? (Rick)

Misc Misc

Setup/Teardown of Context Use tool to test itself - exceptions

N o P o s s i b l e ? P o s s i b l e N o t s o m u c h

Process

Acceptance Test Driven development

Process

Encourages communication

Reporting

Tagging/Categories Y e s Y e s Y e s Y e s

Reporting

Summary

Reporting

Clickthrough to failures

Reporting

See screen captures

Reporting

Aggregate test results

Reporting

Side-by-side.What does this mean? Seems very specific - is it a useful distinction? (Rick) ?

Repository

Builtin

Repository

SCM-Friendly

Repository Test Editor Test Editor Test Editor Test Editor Test Editor Test Editor

Storage format Record Tests Refactor Refactor Builtin keyword help Keyword Completion Argument Completion Updating tests automatically (from results? (Rick))

Test Editor Test Editor Test Editor Test Editor

Indicate when keywords mispelled or wrong "Grammar-checker" args "Quick-Fix" Run test from editor Interactive execution

Test Runner

Command line

N o Y e s x m l N o N o N o N o N o N o x m l e d i t o r N o N o N o Y e s

Test Runner

GUI/Browser

Can run with a GUI or browser

N o Y e s

Test Runner

Run one test or many

Test Runner

Run subsets of tests based on Tagging/Categories Run same tests via different interfaces Run same tests against different configurations

Test Runner

e.g. GUI vs Browser

P o s s i b l e ? Y e s Y e s

Test Runner

e.g. Different browsers; with/without DB

-------------------

-------------------

-------------------

-------------------

Concordion

Cucumber

JBehave

http://www.concordion.org

http://jbehave.org/ http://wiki.github.com/aslakhellesoy/cucumber

Create a safety net for refactoring by separating the test intention (written in plain English) from the test implementation (written in Java), so the latter can change, as the system's design evolves, while the former catches regression bugs. Most other testing frameworks mix together the test intention with its implementation. Write user stories and acceptance criteria Write business scenarios (scenarios) in plain-text in structured English. files in any language.

JUnit (for the Java version of Concordion Command line, Rake, anyway) IDE

Ant, Maven, JUnit, custom

HTML written to a configurable location

Console, HTML, XML, PDF, custom

Console, HTML, XML, text, "stats", custom

HTML / text editor

Any text editor, RubyMine, NetBeans

With source code

File system/SCM Using ORM (ActiveRecord, Hibernate)

Within Java fixture code

Any text editor Anywhere - stories are loaded via StoryLoader implementations - in classpath, from local or remote URL. Steps classes can use any Java framework. DbUnit is commonly used.

Indirectly through a Java fixture

Anything

Anything you can invoke from a Java class.

Instrumentation within HTML (for test case); and Java (for test implementation) Gherkin (plain text)

Within the Java fixture code you can do anything you like

Several options. Before hooks. Background

Pluggable plain-text StoryParser. Default implementation is regexbased. Various @Before* and @After* annotated methods can be called. Inside those methods, any java framework for data loading can be invoked - DbUnit is commonly used.

Use DatabaseCleaner (Rails)

Java, .NET, Ruby, Python

Yes

Ruby, Java, Scala, Groovy, Javascript, Clojure, Ioke, .NET Yes http://wiki.github.com/asl akhellesoy/cuke4duke Yes

DbUnit - as above Java or any other language that you can access via Java API, e.g. groovy, jruby, jython, rhino.

Yes

Yes http://wiki.github.com/rich Requires Java/.NET ardlawrence/Cuke4Nuke interop

Yes, from fixture code

Yes, Selenium-based Steps already exist.

Yes

No

Yes

Yes - i18n'd keywords

No Plain language

Test is plain text, Execution framework is Java Yes Tabular input and output supported in different ways.

Supported, but not required

Yes

Yes

Singhal Variables are matched with text in story. Default implementation is regexbased

Variables as arguments of fixture methods

Scenario Outline

Yes

Yes, in Step Definitions

Yes, in Steps class

Yes

Yes - use !-- for Yes - use # for comments comments

Mature (since 2006)

Mature

Mature

Yes No

Yes

Yes No

Yes

Yes

Yes

No

gem install cucumber

jar file(s) / maven

N/A

N/A

Yes, within Java fixture code Yes

Before and After hooks ??

@Before* and @After* hooks (multiple levels) Yes

Yes

Yes

Yes

Yes

Indeed

Yes

Automatic generation of breadcrumbs based on naming conventions

Yes @foo @bar

No - planned for 3.1

Yes

Yes

Yes - customizable

No

No

When run in IDE via JUnit / TestNG

Through user-contributed plugin

Possible - see watir example

Possible - example exists

As per JUnit

Yes

Yes - customizable

N/A

No

Anything implementation of StoryLoader

Yes

Yes

yes

HTML No No No No No No

Plain text + code (several languages) Plain text + code (Java) With Selenium, for No Selenium-based Steps No No --lang help In RubyMine In RubyMine No No ? No No

No No Yes, from JUnit which is built into most Java IDEs No

Abillity to do dry-run No Yes No Depends on editor When run in IDE via JUnit/TestNG

Yes, as per JUnit

Yes

Yes Yes - whatever you want to add to Steps class Selenium/WebDriver classes exist

No

Yes - Watir, WebDriver, Celerity compatible

Yes

Yes

Yes

Possibly

Yes --tags @foo

No - planned for 3.1

No

Yes

Ant/Maven/IDE

No

Yes --profile

Createable - not part of framework

-------------------

-------------------

FitLibrary

Fitness-Slim

GreenPepper

http://sourceforge.net/projects/fitlibrary/

http://www.greenpeppersoftware.com

Detailed requirements in terms of business rules, objects and processes, test automation, table-based DSL

Use Automated Functional Testing as an "Executable Specification" mechanism that optimize requirement communication between customers/users/field experts and the developers team. All this in order to build the right software, the one needed.

Yes: FolderRunner + batch running + with FitNesse

Fitness

From a Wiki (XWiki, Confluence), from JIRA or from the IDE (Eclipse & Visual Studio plugin). Runner versions for Java, .Net and Ruby). Using continuous integration with Maven, Nant, MS Build.

As annotated tables, like Fit

Fitness

HTML in WIKI (Confluence, XWiki), JIRA or on disk (HTML)

FitNesse, Excel, html editor

Fitness

Confluence, XWiki

FitNesse pages or files

Fitness

Confluence, XWiki, local disk

Yes

No

No

Yes, optionally with an adapter

Yes

Multiple SUT configuration for a test suite. Configuration managed within the WIKI

table DSL, with some plain text

Yes

Plain text with fixture in Java, .Net and Ruby

Yes

No

Java JVM, .net, Python, Smalltalk, JavaScript. Reference implementation in Java.

Java, .NET, Ruby

Yes

Yes

Yes

Yes

Through API of language. Java fixtures/drivers for Swing, web/ajax (SpiderFixture) shell, email, PDFs, XML, web services, mocking web services, templating

API

Yes

Yes

Yes

Yes

Not in a programming language sense

No Yes

Usually, but also has plain-text form Yes. Procedural abstraction with defined actions with parameters, OO abstraction with classes (to organise defined actions and to make distinctions between different cases), business rule abstractions with multi-defined actions

Yes

Yes

Yes

Yes

Yes

No

Yes

No

Yes, any HTML

Yes

mature

mature. commercial offering since 2005 with support

Yes No

Yes

All runners are opensource Pay for Server and technical support

Yes, extensively

Yes

No, not when used with FitNesse Yes, by running part of a storytest, but this is a little inconvenient

No

N/A

?? don't understand the point of this row

N/A

Yes Yes

Yes Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes, with SpiderFixture with Firefox

No

Yes

Yes

? Don't understand this row

Various repositories are used

Yes

Yes

FitNesse pages, html, xls No No ? Don't understand this row No No No

Wiki page, html No No No No No No

No No Yes, with FitNesse No

No n/a Yes No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

-------------------

-------------------

GUIdancer

Jubula

PyWinauto
http://pywinau to.openqa.org /

http://www.guidancer.com

http://www.eclipse.org/jubula

Functional acceptance testing (blackbox) through the GUI to test programs as the end user will see them. Removal of programming effort from acceptance testing. Same goals as Jubula, but with added extras for professional use Use in agile processes (tests (code coverage, TestStyle, Mylyn created in parallel with integration, reporting) development).

See Jubula

Yes: Tests can be started from a command line, which can be called by e.g. Ant or CI tools

See Jubula. Additionally, long term BIRT trends can be generated from results.

In client itself (as summary table and full report tree). Also as HTML and XML reports.

See Jubula. Additionally, Mylyn tasks can be used for contextbased working. There is also a Test Style module for best practices for test structure etc.

In "Test Case Editor": tests can be added (referenced), removed and altered (e.g. order, names, components, data) Tests are stored in a database. Projects can be exported as XML files for version control

See Jubula.

Yes

Yes

Yes

Yes

See Jubula

Tests are written using pre-existing modules of test functionality. Test modules can be renamed so that tests can be read in natural language

See Jubula

Via an external script called from the test

See Jubula

Via an external script called from the test

Java

Java

Yes

Yes

No

No

See Jubula

Java Swing, SWT/RCP, WebUI, GEF

Yes

Yes

Yes

Yes

No n/a

No n/a

No

No

See Jubula

Yes - abstraction from data and actual components in the application.

Yes

Yes

Yes

Yes

Yes

Yes

See Jubula

Comments can be written for each test case

mature No, professional version of Jubula Yes

mature (Jubula consists of core features of GUIdancer)

Yes No

Yes No

Yes

Yes

Yes

Yes

See Jubula

Setup needs to be specified once as reusable module. Teardown can either be specified or the application can be restarted.

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

No

No

Yes

Yes

database/xml Yes Yes No Yes Yes Yes

database/xml Yes Yes No Yes Yes Yes

Yes Yes Yes Yes

Yes Yes Yes Yes

Yes

Yes

Yes

Yes

One test can be run per command line client at a time

One test can be run per command line client at a time

No

No

Yes

Yes

Yes

Yes

-------------------

-------------------

-------------------

Robot Framework

Sahi-Java

Selenium Web Testing Tool

http://robotframework.org http://sahi.co.in http://seleniumhq.org

Robot Framework is a generic keyword-driven test automation framework for acceptance level testing and acceptance testProvide a Java driven development interface to Selenium tests web apps on any OS in any (ATDD) Sahi browser with any language.

Yes

Java

Yes

Yes

JUnit reports for Java

Out of scope for an automation tool

Yes

Java IDE

Selenium IDE. For RC:Vim, Emacs, etc.

Files that can be stored to version control. No Yes http://code.google.com/p/ robotframeworkdblibrary/() No

Git, Svn, CVS, Perforce

And database

Driver independent

Yes

Yes

Java

All of them

Yes

Yes. Tests are in a programming language.

Yes

Yes. Tests are in a programming language.

Python

Java + Javascript

All of them

Jython

Yes

Yes

IronPython

No

Yes

Pluggable

API, WebUI

API/WebUI

Yes

Yes

Tests are code.

Yes

No

Yes (Selenium IDE)

Yes

Yes

Yes Tests are code.

Yes

No

Yes (Selenium IDE)

Yes

Yes

Tests are code in a programming language.

Yes

Yes

Tests are code in a programming language.

Yes

Yes

Tests are code in a programming language.

Yes

Yes

Same as above

Yes

Yes

As above

Stable releases since 2008

mature

Yes No

Yes

Yes

Yes

Yes

Not yet. :-)

Yes

No

Yes

Yes

Yes

Yes

Needs to be coded

Tests are code in a programming language. Yes

Yes

Tests are code in a programming language.

Yes

In what way? To whom?

Multiple

No

Tests are code in a programming language.

1-page

Tests are code in a programming language.

Yes

Tests are code in a programming language.

Depends on Driver ("Tool")

Yes

Yes

Yes. Tests are code in a programming language.

No

Yes. Tests are code in a programming language.

No

Why?

Yes

Yes

How could it not be?

Txt, html, tsv, rest No Yes with Robot IDE No Yes Yes

Java

Anything you want

In next release Yes (Selenium IDE) Java IDE No Yes Yes No Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse

Yes Yes Plug-in (private, possibly open-sourced) Yes No

Use an IDE like Eclipse Use an IDE like Eclipse Use an IDE like Eclipse Yes, Selenium IDE. For RC, use an IDE like Eclipse

Yes

Yes

Yes

Yes

Yes

Yes, Selenium IDE

Yes

Depends on implementaton Use an IDE like Eclipse

Multiple

No

Use an IDE like Eclipse

Yes

No

Use an IDE like Eclipse

Yes

Yes

Use an IDE like Eclipse

-------------------

------------------- -------------------

SWAT (Simple Web AutomationFEST Toolkit)

Trace Analyst

Fitnesse

http://ulti-swat.wikispaces.com

http://www.traceanal yst.com http://fest.easytesting.org/ http://fitness.org

Build on Fitnesse to improve it

Functional test of Swing and JavaFX UIs. Intended users: developers and testers

Requirements Traceability & Project Progress Monitoring

No

Any (JUnit, TestNG)

n/a

Yes

No

From test runner

Yes, HTML, XML, Excel

Yes

No

Java IDE, text editor

n/a

Yes

No

No

n/a

Yes

No

No

n/a

No

Yes

Yes

Yes

Yes

Java or any JVM-based language

Java, (.Net in future) Yes

Yes

N/A

n/a

Yes

N/A

n/a

Possible

C#

Java

Java

Java

No

Yes

Yes

Java

Yes

No

In Future

Yes

WebUI

API

Yes (custom connectors)

API

Doesn't work when Fitnesse run as a service Depends on test runner n/a

Fitnesse

Yes

Fitnesse

No

n/a

Yes

Fitnesse

Yes

n/a

Possible

Fitnesse

No

n/a

Yes

Fitnesse

Yes

n/a

Possible

Fitnesse

Yes

n/a

Possible

Fitnesse

Yes

n/a

No

Fitnesse

Yes

n/a

No

Fitnesse

Java or any JVM-based language

n/a

Yes

Swing: mature, JavaFX: development just started beta

Yes No

Yes No

No Yes

Yes

Yes

Yes

No

Yes

Yes

N/A

No Ant task & Maven plugin, Bamboo Plugin

Yes

N/A

Yes, depending on test runner

N/A

Yes, depending on test runner Not the editor Yes No

Yes

Fitnesse

Yes

Yes (create your own categories)

Yes

Fitnesse

Fitnesse

Fitnesse

Fitnesse

Yes

Yes Yes, depending on test runner Yes, depending on test runner Yes, depending on test runner Yes, screenshots for test failures Yes, depending on test runner Yes, depending on test runner

Yes

Yes

n/a

Yes

Yes

Yes

No

Yes

No

N/A

Yes

Fitnesse

No

n/a

Wiki

Yes

n/a

Somewhat

n/a Yes Planned Planned Planned Yes Yes No Java IDE Java IDE Java IDE Java IDE Java IDE n/a n/a n/a n/a n/a n/a

Wiki No Minimal (Rename/move) No No No No

Fitnesse Fitnesse Fitnesse Fitnesse

Java IDE Java IDE Java IDE No

n/a n/a n/a n/a

No No Yes No

Fitnesse

Yes

n/a

Yes

Browser

Fitnesse

No Yes, depending on test runner

n/a

Browser

n/a

Yes

Fitnesse

Yes, depending on test runner

n/a

Yes

No n/a Yes, support for different Operating Systems n/a

-------------------

-------------------------------------

-------------------

Twist

http://www.thoughtworks-studios.com/twist

Twist is a collaboration platform in which business users can express testing intent (similar to requirements) in simple english and technical users can render these english statements executable using the inbuilt power of the IDE and supported variety of languages. Through this Twist promotes the notion of software specifications also becoming executable tests. Twist supports multiple industry standard drivers that can be used to implement test intent.

Yes any Java Driver. The 2.1 version comes bundled with Sahi, Selenium to test Web Applications, Frankenstien to drive Java Swing applications and SWTBot for Eclipse/SWT based applications

Interactive HTML reports stored in a configurable location

Through an inbuilt editor called "Scenario Editor" which support refactoring and reuse

Tests can be stored in any SCM

Through a JDBC driver

Interact using any Java Driver. The 2.1 version comes bundled with Sahi, Selenium to test Web Applications, Frankenstien to drive Java Swing applications and SWTBot for Eclipse/SWT based applications

Java and Groovy

Possible using Java libraries

Possible using Java libraries

Java or Groovy

Yes

No

Pluggable Use any java driver, Version 2.1 onwards comes bundled with Sahi, Selenium, Frankenstein, SwtBot Currently, does not support any windows thick client applications. Currently support only Eclipse as an IDE

Yes

It supports DSL so use any keywords you like

Java and Groovy Yes

Supported

Yes, through Extract Concepts

Yes, through Extract Concepts

Twist Tests supports parameters

No (Can be done in underlying Java code and a context object)

Yes, definitely

mature

No Yes

Yes

Yes Installation process automated, can be also installed as an Eclipse Plugin

Twist supports Automated, Manual and Hybrid Test execution

Yes Yes

Yes

Yes

Multiple

Yes

Yes

Yes. Captures screenshots on failures

Yes

Yes

No, just add subclipse and you are done

Yes

Text based wiki files Yes provided the driver supports recording Yes Yes Automatically evolves as you write scenarios and leverages on IDE capability to provide the support. Quick assist and auto complete for DSL Yes

Argument validation, and general syntax validation is possible Yes Yes Yes

Yes

Yes, Eclipse IDE

Yes

Multiple

Possible

Yes

-------------------

Watin (Web Application Testing In .Net)

(Fire)Watir (Web Application Testing in Ruby)

http://watin.sourceforge.net/documentation.html

http://watir.com

Automate tests with Internet Explorer and FireFox using .Net

Automated testing that doesn't hurt

Use any test runner

Test runner and through Logger class

.Net IDE

SciTE

No

No

DOM/JS

DOM/JS

.Net languages

Ruby

C#

Ruby

No

Yes

API

Depends on test runner

Roo

No

Roo

Yes Yes

Yes

Yes (Page and Control model)

Yes

.Net

.Net

Yes No

Yes

Yes

Yes

Depends on the test runner

Depends on the test runner

Depends on the test runner Yes

Yes

Yes

Depends on test runner

Depends on test runner

Depends on test runner

Screencapture functionality available

Depends on test runner

Depends on test runner

No

Yes, plain text

Depends on test runner .Net IDE .Net IDE Yes .Net IDE .Net IDE

.Net IDE .Net IDE Yes Yes (run tests in debug mode in .Net IDE) SciTE IRB

Yes

Yes

Yes

Yes, but depends on how the tests are setup

Depends on test runner

Yes

Yes (IE, FireFox)

-------------------

-------------------

White

http://white.codeplex.com

Automate all windows applications except browser, if they support UIAutomation. Provide the driver to test and let the programming language of choice do the rest.

Any .net testing tool/any .NET language

Depending on the tool

.net language IDE

source control

n/a

yes

.net language

yes

C#

No/NA

Yes

Win32, WPF, WinForm, Silverlight (in IE) and SWT

Value Yes No Possible N/A

Interpretation Tool does not support this functionality The tool doesn't support it directly but it can be done in some way. Not applicable to this kind of tool

Example

<Toolname>

Refer to the column for <Toolname>

See Fitness for the editing environment for Fitness-Slim

You might also like