A Taxonomy of Software Assurance Tools and

the Security Bugs they Catch

Michael Kass
Information Technology Laboratory
National Institute of Standards and
OWAS Technology
http://samate.nist.gov
P michael.kass@nist.gov
AppSe
c
This is a work of the U.S. Government and is not subject to
copyright protection in the United States.

DC
October 2005
The OWASP
http://www.owasp.org/
Foundation
SAMATE Goals:

A taxonomy of software assurance tools

and their functions
A taxonomy of the software flaws that
those tools find
A reference dataset (RDS) of tests to
measure if SA tools correctly detect those
flaws

OWASP AppSec DC 2005 2

Why a Tool Taxonomy?

Provide a common reference/classification of

functions for evaluation of a tool’s effectiveness
A “first step” in defining a tool functional
specification
Tool specification becomes basis for creating
tool reference dataset/benchmarks
A common specification and reference dataset
for comparing SA tools of the same class

OWASP AppSec DC 2005 3

Tool Taxonomy and Follow-on
Products
Workshop tool function
SA tool
classes
taxonomy

select
tool class
and
functions

focus
group

functional
specification

test plan

test suite

OWASP AppSec DC 2005 4

Tool Survey and Taxonomy

 ”External” Tools

 Network Scanners
 Web Application Scanners
 Web Services Scanners
 Dynamic Analysis Tools

 “Internal” Tools

 Software Requirements Verifiers

 Software Design/Model Verifiers
 Compilers
 Static Source Code Scanners
 Static Byte Code Scanners
 Static Binary Code Scanners
 Database Scanners

Much of the information in the Software Security Tool Taxonomy is derived from the DISA Application Security
Assessment Tool Survey, July 2004, to be published as a DISA STIG.
OWASP AppSec DC 2005 5
Network Scanners
Remotely scan targeted machines, performing port scans, and
probing
for vulnerabilities known in operating systems and third party
network
software.

 Functions - scan for known vulnerabilities in:

 operating systems
 applications
 web servers
 network devices
 network protocols

 RDS requirements:
 networked testbed
 customize-able deployment scenarios for vulnerability
environment OWASP AppSec DC 2005 6

 test case documentation against a functional specification for

Web Application Scanners
A more “specialized” class of SA tool that focuses specifically
on web
applications only, and are not considered generalized network
scanners.

 Scan for vulnerabilities by:

 simulating a web browser client
 performing field manipulation/cookie poisoning
 exploiting relative URL paths
 injecting malicious data via form fields to exploit server

 RDS requirements:
 HTTP(S) server with applications having known security flaws
and vulnerabilities
 networked testbed
 customize-able deployment environment for vulnerability
scenarios OWASP AppSec DC 2005 7


Web Services Scanner Functions

A relatively new class of SA tool whose purpose is

the analysis of web service applications. Web service
scanners have functions of the following type:

 Scan for vulnerabilities by:

 functional test of SOAP server
 test XML encryption, signatures and signature verification
 test WS-Security
 perform XML validation of received messages

 RDS requirements:
 a networked testbed of web services with known security flaws
and vulnerabilities
 networked testbed
 customize-able deployment environment for vulnerability
scenarios
 test case documentation against a functional specification for
OWASP AppSec DC 2005 8
web services scanner SA tools
Dynamic Analysis Tool Functions

Dynamic analysis tools generate runtime vulnerability

scenarios through the following functions:

 Scan for vulnerabilities by:

 performing file corruption
 resource/network/system/interface fault injection
 design attacks
 implementation attacks

 RDS requirements:
 a testbed of applications with known security flaws and
vulnerabilities
 virtual environment (sandbox)
 customize-able operating environment to create vulnerability
scenarios
 test case documentation against a functional specification for
OWASP AppSec DC 2005 9
dynamic analysis SA tools
Software Requirements Verification Tool
Functions

Functions of some requirements verification tools that have

been developed include determining whether requirements
are: (from NASA’s Automated Requirements Measurement
Tool )

 Scan for vulnerabilities by verifying that application

requirements are:
 verifiable
 complete
 consistent
 unambiguous

 RDS requirements:
 a corpus of application requirement documents that introduce
security vulnerabilities into application design and/or
implementation
 test case documentation against a functional specification
OWASP AppSec DC 2005 for 10

requirements verification SA tools

Design/Model Verification Tool Functions

Design/model verification tools provide are a

vital “first line of defense“ in software
assurance.

 Identify potential flaws/vulnerabilities by performing:

 application simulation
 exhaustive verification
 proof of design

 RDS requirements:
 a corpus of design documents that introduce security
vulnerabilities into application design and/or implementation
 test case documentation against a functional specification for
design/model verification SA tools
OWASP AppSec DC 2005 11
Compilers
Choosing “type safe” compilers, or extending the
capability of some compilers can provide an additional
level of software security (although not guaranteed).
Some security functions of “extended compilers” include:

 Identify potential flaws/vulnerabilities by injecting run-

time code to identify:
 buffer overflows
 race conditions
 un-initialised variables

 RDS requirements:
 a testbed of applications with known security flaws and
vulnerabilities
 A virtual test environment
 customize-able operating environment to create vulnerability
scenarios OWASP AppSec DC 2005 12

 test case documentation against functional specification for

Source Code Scanners

A relatively “new” class of SA tool that can perform data-

flow, control-flow, inter-procedural analysis of code and
identify software security flaws and potential vulnerabilities.

 Some key functions include:

Identify security flaws and vulnerabilities
filter results based upon flaw type/priority
scale to large code sets
generate code metrics

 RDS requirements:
a corpus of source code examples with known security flaws
and vulnerabilities
test case documentation against functional specification for
source code analysis SA tools
OWASP AppSec DC 2005 13
Static Bytecode Analysis Tools

A step below “source code scanners’”, but above binary

analysis tools, byte code analysis tools have similar
functions to source code scanning tools.

 Some key functions include:

Identify security flaws and vulnerabilities
filter results based upon flaw type/priority
scale to large code sets

 RDS requirements:
a corpus of byte code examples with known security flaws and
vulnerabilities
test case documentation against functional specification for
static bytecode analysis tools

OWASP AppSec DC 2005 14

Static Binary Analysis Tools

The “last line of defense” in SA tools. Performs similar SA

functions as source code analysis tools, but does so at
“runtime”.

 Some key functions include:

perform boundary checks
identify memory leaks
identify unreachable code
identify race conditions
identify type mismatches

 RDS requirements:
a corpus of binary executable examples with known security
flaws and vulnerabilities
test case documentation against functional specification for
static binary code analysis tools OWASP AppSec DC 2005 15
Defining a Software Security Flaw Taxonomy

Because of the nature of software development (new

technologies, new languages and language features) a taxonomy
of software security flaws will be a living and changing entity.
Additional characteristics that must be considered include:

 Program vulnerabilities are usually a combination of security

flaws

 Mutual flaw exclusion will be difficult to deal with (examples :

authentication vs. logic flaw problem)

 Some of the flaws in the taxonomy cannot be identified by

tools today

 Some flaws have never been seen in real world code… yet

 Some flaws can be introduced at multiple points in the SDLC

OWASP AppSec DC 2005 16

A Taxonomy of Software Security Flaws and
their Point of Introduction in the SDLC
One of the first steps in comparing tools of the same class is to
determine what software flaws all tools of that class should be
able to identify. The following hierarchical classification of
software flaw root causes is taken from Secure Software’s CLASP
V1.1 Training Manual

• identify range and type errors

• Stack overflow – (requirements, design, implementation)
• Command injection (design, implementation)
• Double Free (implementation)
• identify environmental problems
• Resource exhaustion (design, implementation)
• identify synchronization and timing errors
• TOCTOU (time-of-check time-of-use) race condition (design,
implementation)
• identify protocol errors
• Use of a broken/risky cryptographic algorithm (design)
• identify general logic errors
 Performing a “chroot” without a “chdir” (implementation)
OWASP AppSec DC 2005 17
 Reference Dataset

The SAMATE team is creating a reference dataset of test cases to

measure SA tool performance against the functions defined for
that particular type of tool. Currently, we are focusing on source
code scanning tools.

 Existing source code material for a reference dataset include:

 OWASP WebGoat
 Foundstone HacmeBook and HacmeBank
 CVE listing of known vulnerabilities
 MIT Lincoln Lab Source Code Contribution (1000+ test cases)
 Fortify Software Source Code Contribution (80+ test cases)

 Both “In the Wild” and “Manufactured” code will be part of the reference
dataset

 Dataset will include representation of software flaws from the SA security

flaw taxonomy

 The SAMATE Reference Dataset will include testing artifacts from

requirements, modelling and design, implementation and deployment
phases of the SDLC
OWASP AppSec DC 2005 18
Ongoing Work

 NIST is developing prototype reference dataset for SA tools, for review by

SAMATE working group members

 November 7,8 -- Workshop on Software Security Assurance Tools,

Techniques, and Metrics , Long Beach, CA (http://samate.nist.gov)

 DAY #1 – Papers
 DAY #2 - “Target Practice” for source code scanner tool developers
against SAMATE SA Tool Reference Dataset

 Continue work through SAMATE focus groups (online and at follow-on

workshops) to refine:

 a taxonomy of common functions for source code scanners

 a common taxonomy of software security flaws
 content of a reference dataset for testing SA tools

OWASP AppSec DC 2005 19