Software Metrics: Roadmap

By Norman E. Fenton and Martin Neil

Presentation by Karim Dhambri

Authors (1/2)
n

Software Metrics: Roadmap

Norman Fenton is
Professor of
Computing at Queen
Mary (University of
London) and is also
Chief Executive Officer
of Agena, a company
that specialises in risk
management for
critical systems. He is
head of RADAR (Risk
Assessment and
Decision Analysis)
Group
2

Authors (2/2)
n

Martin Neil is a Reader in "Systems

Risk" at the Department of Computer
Science, Queen Mary, University of
London, where he teaches decision
and risk analysis and software
engineering. Martin is also a joint
founder and Chief Technology Officer
of Agena Ltd (UK)

Software Metrics: Roadmap

Plan
Introduction
n Brief history of software metrics
n Weaknesses of traditionnal
approaches
n Causal models
n Future works
n Comments on the article
n

Software Metrics: Roadmap

Introduction (1/9)
n

The car accidents example

n

Data on car accidents in both the US

and the UK reveal that January and
February are the months when the
fewest fatalities occur.

Software Metrics: Roadmap

Introduction (2/9)
n

The car accidents example

n

Thus, if you collect a database of

fatalities organised by months and use
this to build a regression model, your
model would predict that it is safest to
drive when weather is coldest and roads
are at their most treacherous.

Software Metrics: Roadmap

Introduction (3/9)
n

The car accidents example

n Such

a conclusion is perfectly sensible

given the data available, but intuitively
we know its wrong.
n The problem is that you do not have all
the relevant data to make a sensible
decision about the safest time to drive.

Software Metrics: Roadmap

Introduction (4/9)
n

The car accidents example

Software Metrics: Roadmap

Introduction (5/9)
n

So what has this got to do with

software metrics? Well, software
metrics has been dominated by
statistical models, such as regression
models, when what is really needed
are causal models.

Software Metrics: Roadmap

Introduction (6/9)
n

Software resource estimation

n Much

software metrics has been driven

by the need for resource prediction
models.
n Usually this work has involved models
of the form
effort=f(size)
Software Metrics: Roadmap

10

Introduction (7/9)
n

Problems with effort=f(size)

n Size

cannot cause effort.

n Such models cannot be used for risk
assessment because they lack
explanatory framework.
n Managers cant decide how to improve
things from the models outputs.

Software Metrics: Roadmap

11

Introduction (8/9)
n

Solution: causal modeling

n Provide

an explanatory structure to
explain events that can then be
quantified.
n Provide information to support
quantitative managerial decision-making
during the software lifecycle.
n Provide support for risk assessment and
reduction.
Software Metrics: Roadmap

12

Introduction (9/9)
n

Software resource estimation

Software Metrics: Roadmap

13

History of metrics (1/13)

n

Def.: Software metrics is a collective

term used to describe the very wide
range of activities concerned with
measurement in software
engineering.

Software Metrics: Roadmap

14

History of metrics (2/13)

n

These activities range from:

n Producing

numbers that characterize

properties of software code

n Models

that help predict software

resource requirements and software
quality

n Quality

control and assurance

Software Metrics: Roadmap

15

History of metrics (3/13)

n

Software metrics are used since the

mid-1960s

At that time, Lines of Code was used

as a measurement of productivity and
effort

Software Metrics: Roadmap

16

History of metrics (4/13)

n

Problems using metrics:

n Theory

and practice have been out of

step
n Metrics often misunderstood, misused,
and even reviled
n Industry is not convinced of metrics
benefits
n Metrics programs are used when things
go bad to satisfy some assessment body
(CMM)
Software Metrics: Roadmap

17

History of metrics (5/13)

n

The two components of software

metrics:
n The

component concerned with defining

the actual measures

n The

component concerned with how we

collect, manage and use the measures

Software Metrics: Roadmap

18

History of metrics (6/13)

Software Metrics: Roadmap

19

History of metrics (7/13)

n

Rationale for using metrics

n The

desire to assess or predict

effort/cost of development processes

n The

desire to asses or predict quality of

software products

Software Metrics: Roadmap

20

History of metrics (8/13)

n

The key in both cases has been the

assumption that product size should
drive any predictive models.

Software Metrics: Roadmap

21

History of metrics (9/13)

n

LOC/programmer month as
productivity measure

Regression-based resource prediction

by Putnam and Boehm:
Effort = f(LOC)

Program quality measurement

(usually defects/KLOC)
Software Metrics: Roadmap

22

History of metrics (10/13)

In the mid-1970s, we recognized the
drawbacks of using LOC as a measure
for different notions of program size.
n LOC cannot be compared between
high- and low-level programming
languages
n

Software Metrics: Roadmap

23

History of metrics (11/13)

From the mid-1970s interest in
measures of software complexity and
functional size (such as function
points)
n The rational for these metrics is still
to asses quality and effort/cost
n

Software Metrics: Roadmap

24

History of metrics (12/13)

n

Study of software metrics has been

dominated by defining specific
measures and models.

Much recent work has been

concerned with collecting, managing,
and using metrics in practice.

Software Metrics: Roadmap

25

History of metrics (13/13)

n

Most notable advances

n Work

on the mechanics of implementing

metrics programs
n Grady

and Caswell: first company-wide

software metrics program
n Basili, Rombach: GQM
n The

use of metrics in empirical software

engineering
n Benchmarking

and evaluating the

effectiveness of s.e. methods, tools and
technologies (Basili)
Software Metrics: Roadmap

26

Weaknesses of traditional
approaches (1/11)
n

The approaches to both quality

prediction and resource prediction
have remained fundamentally
unchanged since the early 1980s.

Software Metrics: Roadmap

27

Weaknesses of traditional
approaches (2/11)
n

These approaches have provided

some extremely valuable empirical
results, but cannot be used effectively
for quantitative management and risk
analysis, the primary objective of
metrics.

Software Metrics: Roadmap

28

Weaknesses of traditional
approaches (3/11)
n

Regression-based model for quality

prediction:
f(complexity metric) = defect density

Problems
n Incapable

of predicting defects

accurately
n No explanations of how defect
introduction and detection variable
affects defect counts
Software Metrics: Roadmap

29

Weaknesses of traditional
approaches (4/11)
n

A further empirical study (Fenton)

shown:
n Size

metrics (while correlated to gross

number of defects) are poor indicator of
defects
n Static complexity metrics are not
significantly better as predictors
n Counts of defects pre-release is a very
bad indicator of quality
n The

lunch story
Software Metrics: Roadmap

30

Weaknesses of traditional
approaches (5/11)

Software Metrics: Roadmap

31

Weaknesses of traditional
approaches (6/11)
n

These results invalidate models:

n using

pre-release faults as a measure for

operational quality

n using

complexity metrics to predict

modules fault-prone post release
n Complexity

metrics were judged valid if

correlated with pre-release fault density
Software Metrics: Roadmap

32

Weaknesses of traditional
approaches (7/11)
n

Empirical phenomenon observed by

Adam (1984):
n []

most operational system failures

are caused by a small proportion of the
latent faults.
n The fact that fault density (in terms of
pre-release faults) was used as a
measure of user perceived software
quality lead us to wrong conclusions.
Software Metrics: Roadmap

33

Weaknesses of traditional
approaches (8/11)
n

Explanations of the scatter plot

n Most

of the modules that had high

number of pre-release, low number of
post-release faults just happened to be
very well tested.
n A module that is never executed will
never reveal latent faults (no matter
how many), hence operational usage
must be taken into account.

Software Metrics: Roadmap

34

Weaknesses of traditional
approaches (9/11)
n

Other problems with regression-based

models for resource prediction:
n Lack

causal factors to explain variation

n Based on limited historical data
n Resource constraints not modeled
n Black box models
n Cannot handle uncertainty
n Little support for risk assessment and
reduction
Software Metrics: Roadmap

35

Weaknesses of traditional
approaches (10/11)
The classic problem : Is this system
sufficiently reliable to ship?
n Useful information:
n

n Measurement

data from testing (such as

defects found in various testing phases)
n Empirical data about the process and
resources used
n Subjective information about the
process/resources
n Very specific and important pieces of
evidence (proof of correctness)
Software Metrics: Roadmap

36

Weaknesses of traditional
approaches (11/11)
In practice, we only possess
fragments of such information.
n The question is how to combine such
diverse information and then how to
use it to help solve a decision
problem that involves risk.
n

Software Metrics: Roadmap

37

Causal models (1/7)

n

We need a model that take account

of missing concepts from regressionbased approaches:
n Diverse

process and product variables

n Empirical evidence and expert
judgement
n Genuine cause and effect relationship
n Uncertainty
n Incomplete information
Software Metrics: Roadmap

38

Causal models (2/7)

n

Def.: A BBN is a graphical network

together with an associated set of
probability tables. The nodes
represent uncertain variables and the
arcs represent the causal/relevance
relationship between the variables.

Software Metrics: Roadmap

39

Causal models (3/7)

Software Metrics: Roadmap

40

Causal models (4/7)

Building and executing realistic BBN
models is now possible because of
recent algorithms and software tools.
n Practical applications:
n

n Medical

diagnosis
n Mechanical failure diagnosis
n Help wizards in Microsoft Office

Software Metrics: Roadmap

41

Causal models (5/7)

Software Metrics: Roadmap

42

Causal models (6/7)

Software Metrics: Roadmap

43

Causal models (7/7)

n

Benefits of using BBNs:

n
n
n
n
n
n
n
n
n

Explicit modeling of ignorance and uncertainty

Combine diverse types of information
Makes assumption explicit
Intuitive graphical format
Ability to forecast with missing data
Use of what-if?
Use of subjectively or objectively derived
probability distributions
Rigorous math semantic
Availability of tools like Hugin
Software Metrics: Roadmap

44

Future works
Combining causal models such as
BBNs with preference models such as
those found in MCDA.
n Extending the emerging discipline of
empirical software engineering (cause
and effects hypotheses).
n Developing metric programs for
decision-support involving companyspecific data input.
n Technology
Softwaretransfer
Metrics: Roadmap (questionnaires)
45
n

Comments on the article

n

Positive
n Application

of simulation to software
engineering
n Causal models can constantly be tuned
n

Negative
n Would

have liked more details

concerning BBNs
n In practice, how can we determine the
probability for each node
Software Metrics: Roadmap

46