A Comparison Study

of
Capability Maturity Model
and
ISO standards
by Alok Pareek

11/24/98
1
Object of the study

This Study introduces the CMM and the ISO

9000 standards individually by discussing the
background, the structure, and the future
development of each of the process
improvement models.

The industry’s experience with each of the

software process improvement models will also
be examined. Both process improvements
models are compared and contrasted.

11/24/98
2
Three Dominant Project Failures

Cost over runs

Schedule over runs
Failure in meeting deadlines

11/24/98
3
 CMM ISO 9001

Hierarchical Structure Flat Structure

Popular in USA Popular in Europe and

Asia
Mainly adopted by Good for small and
large organizations large organizations
Specifically for software Common standards for
all industries

11/24/98
4
Things to consider when reviewing a
software process improvement model
are:
-
Company objectives
Quality goals

Company size
Financial availability
Implementation time/schedule

Customers of the company

11/24/98
5
 process improvement models
• Successful, past processes can become base
models
- to assist in planning and scheduling
future projects
- to guide in
- meeting deadlines,
- saving on costs
- increasing efficiency
- developing better quality
products
- generating profits.
11/24/98
6
Brief history of the CMM

1986 Work begun on the maturity framework

1987 SEI releases a brief description of
software maturity framework.
1991 SEI releases v1.0 of the CMM.
1992 SEI releases v1.1 of the CMM

11/24/98
7
 What is CMM?
The CMM is a framework that describes the key
elements of an effective software process.

The CMM guides software organizations

striving to gain control of their processes for
developing and maintaining software, evolve
toward a software engineering culture, and
management excellence.

The CMM describes an evolutionary

improvement path for software
organizations from an ad-hoc, immature process to a
11/24/98 mature, disciplined one.
8
 CMM framework
The primary objective of the CMM is to
achieve a controlled and measured process as
the foundation for continuing
improvement by providing a framework that
can be used:

To identify strengths and weaknesses in an

organization.
To understand the activities necessary to
launch a process improvement program.
To define and improve their organization’s
process.
To categorize software companies according
to their capabilities, such that customers can
11/24/98 identify risk, when selecting contractors.
9
 Why the CMM?
The DoD has stressed that level one and
level two organizations are considered high
risk in technical evaluations and that "a
level three or higher CMM assessment is
required to bid on new government
software development".

Requirements like these have caused

companies, who primarily work on
US Defense Department contracts,
such as Hughes Aircraft, feel urged
to get a CMM assessment and to be at
least at a level three.
11/24/98
10
What is a process?

Process is a sequence of steps performed for

a given purpose.

Process is a system of operations in producing

something i.e. a series of actions, changes, or
functions that achieve an end result.

11/24/98
11
CMM structure

Maturity Levels contains Key Process Areas

Key Process Area organized by Common Features

Common Features contains Key Practices

11/24/98
12
CMM structure

Maturity levels indicates Process Capability

Key Process Areas achieve Goals

Common Features address Implementation &
Institutionalization
Key Practices describe Infrastructures or
Activities

11/24/98
13
 Maturity Level

A well-defined evolutionary plateau on

the path toward becoming a mature
software organization.

Each level is a layer in the foundation

for continuous process improvement.

There are five maturity level in the

CMM.

11/24/98
14
 What forms a Maturity Level?
Goals
-Use to determine if the organization or
project has effectively
implemented the key process area.
-Signify the scope, boundaries, and
intent of each KPA.

Common Features
- Commitment to Perform
- Ability to Perform
- Measurement and Analysis
- Verifying Implementation
- Activities Performed

Key Practices
11/24/98
15
 The Five Maturity Levels

Initial Unpredictable and poorly controlled

Repeatable Can repeat previously mastered tasks

Defined Process characterized, fairly well

understood
Managed Process measured and controlled

Optimizing Focus on process improvement

11/24/98
16
 Level 1 : Initial
Description

- The organization typically does not provide

a stable environment for
developing and maintaining software.

Characteristics

- 75% of organizations are at this level.

- Over commitment is very common at this
level.
- During crisis, projects typically abandon
planned
procedures and revert to coding and
testing.
- Success depends on having an exceptional
11/24/98
manager and
effective software team. 17
 Level 2: Repeatable
Description

-Disciplined level
-Planning and tracking of the software
project is stable and earlier success can
be repeated.

Characteristics

-15% of organizations are at this level.

-Software project standards are defined.
-Planning and managing new projects is
based on similar project
experience.

11/24/98
18
 Level 3: Defined
Description

-At this level the process is standard

and consistent.
-The software process is well defined,
because management and
software engineering activities
are stable and repeatable.

Characteristics

- 8% of organizations are at this

level.
-A standard process for developing
and maintaining software is
documented and used across the
organization.
11/24/98
-Management has good19 insight into
technical progress on the project.
 Level 4: Managed
Description

-Software process capability is

quantifiable and predictable.
-The process is measured and operates
within quantitative limits.

Characteristics

-1.5% of organizations are at this level.

-The organization sets quantitative
quality goals for both software product
and software processes.
-All project’s productivity and quality
are measured for important software
11/24/98
process activities.
20
 Level 5: Optimized
Description

-This level indicates continuous process

improvement.
-The organization is continuously
striving to improve their process capability
range.

Characteristics

-0.5% of organizations are at this level.

-The organization has the means to
identify weaknesses and strengthen the
process proactively.
-The goal is the prevention of defects.
11/24/98
21
 Key Process Areas
-Identify a cluster of related activities
that,
when performed collectively,
achieve a set
of goals considered important for
enhancing process capability.

-Defined to reside at a single

maturity level.

-Identify the issues that must be

addressed
to achieve a maturity level.

11/24/98
22
 Key Process Areas to Achieve
Level 2
Requirement Management
Software Project Planning
Software Project Tracking & Oversight
Software subcontract Management
Software Quality Assurance
Software Configuration Management

11/24/98
23
 Key Process Areas to Achieve
Level 3

Organization Process Focus

Organization Process Definition
Training Program
Integrated Software Management
Software Product Engineering
Inter-group Coordination
Peer Reviews

11/24/98
24
 Key Process Areas to Achieve
Level 4

Quantitative Process Management

Software Quality Management

11/24/98
25
 Key Process Areas to Achieve
Level 5

Defect Prevention

Technology Change Management

Process Change Management

11/24/98
26
 Responsibility for Implementation of
KPAs

The project is primarily responsible for

addressing many Key Process Areas.

The organization is primarily responsible for

addressing other Key Process Areas.

There are both project and organizational

responsibilities in all Key Process Areas.

11/24/98
27
 Common Features

Attributes that indicate whether the

implementation and institutionalization of a
key process area is effective, repeatable, and
lasting.

Used to organize the key practices in each

key process area.

11/24/98
28
 Common Features

Commitment to Perform
Ability to Perform
Activities Performed
Measurement and Analysis
Verifying Implementations

11/24/98
29
 Institutionalize and Implement

The organization outlives those who leave it.

The organizational culture must convey the

process.

Management must nurture the culture.

Culture is conveyed with role models and

rewards.

11/24/98
30
 Commitment to Perform
Describes the actions the organization
must take to ensure that the process is
established and will endure.

Typical sub-features include:

Policies
Senior Management Sponsorship
Responsibility

11/24/98
31
 Project Responsibilities

The project will have primary responsibility for

acting on

Requirements Management
Software Project Planning
Software Project Tracking and Oversight
Software Subcontract Management
Software Configuration Management

11/24/98
32
 Project Responsibilities
(Continued….)

Integrated Software Management

Software Product Engineering
Inter-group Coordination
Peer Reviews
Quantitative Process Management
Software Quality Management
Defect Prevention

11/24/98
33
 Organization Responsibilities
The organization will have primary
responsibility for acting on:

Software Quality Assurance

Organization Process Focus
Organization Process Definition
Training Program
Technology Change Management
Process Change Management
11/24/98
34
 Goals Criteria
All the CMM goals primarily address
process end-states.

Minimal use of subjective wording (i.e.,

“effective”, ”appropriate” ).

Each key practice maps to one or more

goals.

Each goal and its associated practices can

be considered as a ‘sub-process area’.

Goals and sub-process areas support rating

key process areas and maturity levels.
11/24/98
35
Goal Examples

Software estimates are documented for use in

planning and tracking the software project.

Software project activities and commitments

are planned and documented.

Affected groups and individuals agree to their

commitments related to the software project.

11/24/98
36
 Ability to Perform
Describes the preconditions that must exist in
the project or organization to implement the
software process competently. Typical sub-
features include:

Resources
Organization Structure
Delegation
Training
Orientation
11/24/98
37
 Activities Performed
Describes the roles and procedures necessary
to implement a key process area.

Typical sub-features include:

Establish plans and procedures

Performing the work
Tracking it
Taking corrective actions as necessary

11/24/98
38
 Measurement and Analysis
Describes the need to measure the process
and analyze the measurements.

It typically includes examples of the

measurements that could be taken to
determine the status and
effectiveness of the Activities Performed

11/24/98
39
 Verifying Implementation
Describes the steps to ensure that the
activities are performed in compliance with
the process that has been established.

Typical sub-features include reviews and

audits by

Senior Management
Project Management
Software quality assurance group

11/24/98
40
 Key Practices
The infrastructure and activities that
contribute most to the effective
implementation and institutionalization of a
key process area.

11/24/98
41
 Key Indicators

Key indicators offer insight in whether the

goals have been satisfied.

Key indicators form the basis for the SEI’s

maturity questionnaire, which is used by the
SEI to do internal process assessments and by
software buyers to evaluate a contractor’s
capability

11/24/98
42
 CMM critiques
In general, small businesses have criticized
the CMM. The three major problems
experienced were:

Cost Small businesses found that the minimum

cost to initiate the CMM are too high. Like
wise, cost of training is unjustifiable.
Resources The management structure in small
businesses is much flatter, so that the
software manager often assumes more than
one position. The CMM often identifies
positions that do not exist in small
organizations. Furthermore, small
businesses cannot afford full-time personnel
11/24/98
to support quality groups.
43
 CMM critiques ( Continued…)

KPAs It is also felt that many of the KPAs are

inapplicable to small businesses. Many
of the CMM issues address large
organizations, in which communication
is a problem. However, in small
organizations people often
communicate in person on an on-going
basis.

11/24/98
44
 CMM assessments
The CMM provides two types of assessments:

1) internal assessment

2) capability evaluation.

Software organizations use the Software

Process Assessment (SPA), which aid
organizations to evaluate their software
maturity and to identify key areas for
improvement, to conduct an internal
assessment.

On the other hand, SEI professionally trained

evaluators perform the capability evaluation
by using the Software Capability evaluation
11/24/98
(SCE) as an audit to identify
45
qualified
contractors and to monitor existing software
 CMM assessments (Continued…)

For both internal assessments and

capability evaluations, the SEI has
developed an assessment questionnaire
and methodology to identify a software
organization’s maturity status. The
assessment questionnaire is based on the
KPAs.

It contains 120 questions, where the

repeatable and defined levels contain
11/24/98
about 40 questions each and
46 the managed
and optimized levels contains about 20
 CMM assessments (Continued…)
Furthermore, a profile template is used which
lists each KPA, such that they can be checked
as not satisfied, partially satisfied, or fully
satisfied.

An organization’s maturity level is set at the

highest level at which it satisfies all KPAs on a
continuing basis.

11/24/98
47
 CMM benefits
Published results of software companies that
have adopted the CMM is very impressive.
Following are the general benefits of the
CMM:

Improved communication
Decreased work
Decreased integration problems
The average defect rate has decreased,
even though the product sizes have
increased.

11/24/98
48
 Improvements on the CMM
The CMM has not reached perfection. It is still
being tested and reviewed by the software
industry through use in software-process
testing, software-capability evaluations, and
process-improvement programs.

The SEI anticipates that CMM v1.1 will be

replaced by the CMM v2.0 very soon. More
changes will be implemented in the 2.0
version such as include new KPAs, especially
for level four and five.

11/24/98
49
 Improvements in the CMM
(Continued)

Other near-term changes will be oriented

towards tailored versions of the CMM, such as
a CMM for small projects and/or small
organizations.

Long-term activities may address technology

and human resource issues.

11/24/98
50
 Objective of the ISO 9000
The ISO 9000 series of standards is used to
ensure quality in every product component
and the related services before and after the
product development.

The ISO 9000 series dictates that each

development facility and any subcontractor
must be certified.

11/24/98
51
 There exists three different ISO
certifications: ISO9001, ISO 9002, and ISO
9003.

However, only the ISO 9001: "Quality

systems - model for quality assurance in
design/development, production,
installation, and servicing" is used in the
software industry.

Since the ISO 9001 was written to be used

by all kinds of industries, it is often difficult
to interpret it for the software
development. Hence, in 1991 ISO 9000-3:
"Guidelines for the application of ISO 9001
to the development, supply, and
maintenance of software" was published.
11/24/98
52
 The structure of the ISO 9000
The ISO 9000 is a set of generic standards for
quality management and assurance.

ISO 9001 The ISO 9001 contains 20 quality

elements.
ISO 9000-3 ISO 9000-3 is a subset of
guidelines. It is divided into three
groups:

1) general company and

management requirements
2) projects and maintenance
phase requirements
11/24/98
3) supporting activities
requirements. 53
 Objective of ISO 9001
The objective of ISO 9001 in specific is to
build a quality system which comprises:
- organizational structure
- responsibilities
- procedures
- processes
- resources for implementing quality
management.

While the objective of ISO 9000-3 is to provide

specifications on how to apply the ISO 9001 to
software development, supply and
maintenance.
11/24/98
54
 Key success factors of software
process improvement via ISO
9001:
Definition and documentation of the status quo
Identification of best practices
Identification of business processes
Simplification of routine procedures
Internal audits
Impetus and Incentive
Team spirit
Workshops and regular meetings
Definitions of common language
Customer perception surveys
11/24/98
55
 External benefits of ISO 9000
standards:
Higher perceived quality
Improved customer satisfaction
Competitive edge
Reduced customer quality audits
Increased market share
Quicker time to market

11/24/98
56
 Internal benefits of ISO 9000
standards:
Better documentation
-
Greater quality awareness
Positive Cultural Change
Increased operational
efficiency/productivity
Enhanced inter-company communications
Reduced scrap/rework expenses

11/24/98
57
ISO 9000 audits and certification
An audit determines that all phases of the
software development process are defined and
carried out. In
general, auditors make sure that

Procedures are in place for all quality elements.

Proofs, that the procedures followed exists.
The document control scheme, which makes sure
that obsolete documents are purged and current
documents are available where and when it is
needed, is valid.
Internal audits have been performed and corrective
actions have been performed.
11/24/98
58
 ISO 9000 audits (Continued)
The three different audits are:

1) First-party audit - conducted at least

once a year by or on the behalf of
company management to cover
different parts of a company.

2) Second-party audit - conducted by a

customer to audit a supplier’s
operation.

3) Third-party audit - conducted by

external auditors.
11/24/98
59
 ISO 9000 Certification (Continued)
A company can also be certified by a third-
party certification body which is accredited in
the country where it operates.

Once a software organization has been

certified, the certification body will regularly
make follow-up audits to check that the
certification is still valid. The certificate
expires after three years, and can
be reissued after successful reassessments.

In the USA, only the Registrar Accreditation

Board (RAB), an incorporated affiliate of the
American Society for Control (ANSQ), has the
11/24/98
authority to accredit.
60
 ISO 9000 critiques
There is an interesting problem with
certification. Each country has its own
registration body.
Hence the reciprocity of certificates between
countries is a big concern.

It has also been criticized that most software

companies do not take the whole ISO 9000
family into account. It is recommended that
for efficient software development ISO 9004
should also be considered.

11/24/98
61
 ISO 9000 critiques (Continued)

ISO 9004 provides quantitative guidelines to

gather information directly from ISO 9001.
They suggest documentation on control cost
of quality, continuous quality improvement
and metrics programs to analyze the
correlation of process and product quality.

11/24/98
62
 COMPARING CMM AND ISO 9000
1. Initiatives, Objectives and Scope

In general, the CMM and the ISO 9000 are

driven by similar issues and have the
common concern of quality and process
management.
ISO CMM

Its primary focus is Its focus is on the

the customer- supplier to improve the
supplier relationship internal software
to reduce a process.
customer’s risk in
11/24/98 choosing a supplier.
63
 Comparing CMM and ISO 9000
(Contd)
2. Objective
ISO CMM
It is written for a wide range Written specifically for
of industry other than software industry.
software.
Documents are more abstract Detailed document

ISO 9001 is only 5 pages long. CMM is over 500 pages

ISO 9000-3 is 11 pages. l long.
Identifies only the It describes the
minimal requirement for a software process in
quality system.
11/24/98 detail.
64
 Comparing CMM and ISO 9000
(Contd)
3. Product Development

Both the model support

1) definition and formalization of
processes; 2) standardized, objective
evaluations by third parties of supplier’s
capabilities
3) on-going self-assessment
ISO CMM

It has a broad scope It is specific to the

that encompasses software
hardware, software, development.
processed materials,
11/24/98
and services. 65
 Comparing CMM and ISO 9000
(Contd)

4. Concept
ISO CMM

The ISO 9000’s The CMM

concept is to follow a emphasizes on
set of standards to achieving "maturity"
make success and improving its
repeatable. process continuously

11/24/98
66
 Comparing CMM and ISO 9000
(Contd)
5. Structure

ISO CMM

It means that some It emphasizes on

basic practices are in continuous
place and the improvement, even at
challenge is only to the last level.
maintain certification

11/24/98
67
 ISO CMM

It provides a flat It provides a hierarchical

structure in which all framework in which
quality elements need to improvements are
be fulfilled for a company implemented
to be certified. incrementally.

11/24/98
68
 Comparing CMM and ISO 9000
(Contd)
6. Assessments, Capability
Evaluations, Audits, and Certification
In essence, CMM’s capability evaluation has
the same objective as ISO 9000’s third party
audits.

Both have been developed to check the

overall capability of a software organization to
produce software in a timely, repeatable
fashion. ISO CMM
In an ISO 9000 In a CMM capability
audit, a software evaluation, a
organization is software organization
checked that it is ranked according
11/24/98 follows a certain set to the five levels.
of standards. 69
 Comparing CMM and ISO 9000
(Contd)
6. Assessments, Capability
Evaluations, Audits, and Certification
(contd…..)
An organization did a CMM-style self
assessment after an ISO 9000 audit and found
that the auditors had mistakenly perceived
that certain practices were in place.

Internal Assessment:
ISO CMM
This model requires auditors, The CMM allows
such that the value of self-assessment.
certification depends on the
expertise and experience of
11/24/98
the auditors.
70
 Comparing CMM and ISO 9000
(Contd)
7. Software industry’s state
ISO View
From the ISO 9000 information,
no such information can be
extracted, since companies are
either certified or not certified.
A geographical conclusion can
be made in that Europe has the
highest number of certified
companies.
11/24/98
CMM View
From the CMM’s statistical
information, The software
industry still needs a lot of
improvement.
Many companies are still in level
one, and very few are in level
four and five.
71
 Comparing CMM and ISO 9000
(Contd)
8. Time needed
ISO CMM

It takes about one It takes an average

and a half years to of two years to move
obtain ISO 9000 between levels of
certification. the CMM.

It shows that the The CMM is aiming

ISO 9000 is aiming for a strong basis of
for a general software improvement.
11/24/98
improvement.
72
 Comparing CMM and ISO 9000
(Contd)
9. Benefits
These benefits are often accompanied with
great numbers. However, it should be noted
that companies feel more comfortable to
report successes rather than failures, and that
numbers are
sometimes bias because it depends how
those numbers have been calculated.

The common benefits are:

Positive cultural change
Increased productivity
Better communications
11/24/98
Improved customer satisfaction
73
 Comparing CMM and ISO 9000
(Contd)
10. Contrast

It shall be noted that the contrast and

comparison of those software process
models tends to be quite
subjective in some cases and slightly differ
The
in ISO 9000
among is left to freedom of interpretation,
authors.
some argue that if one reads the ISO 9000 with
insight, the ISO 9000 actually covers material of
level one to level three of the CMM.

On the other hand, software organizations, that

achieve certification for the sake of getting certified,
11/24/98
can only be compared to a level
74 one CMM.