I

Creative

Approach

'

-\"\\'v

mVi

INTRODUCTION

TO ALGORITHMS

A Creative Approach

INTRODUCTION
TO ALGORITHMS
A

Creative

Approach

UDIMANBER
University

of Arizona

\342\226\274\342\226\274

ADDISON-WESLEY

PUBLISHING

Menlo
Reading, Massachusetts \342\200\242
Don Mills, Ontario \342\200\242
Wokingham,

Bonn

\342\200\242

Sydney

\342\200\242

Singapore

COMPANY
Park, California
England

\342\200\242

Tokyo

\342\200\242
New

York

\342\200\242
Amsterdam

\342\200\242
Madrid

\342\200\242
San Juan

Cataloging-in-PublicationData

of Congress

Library

Udi.

Manber,

to algorithms.

Introduction

Includes bibliographiesand index.

1. Data structures (Computer science)

2. Algorithms.

I. Title.

'

005.7 3

1989

QA76.9.D35M36

88-2186

0-201-12037-2

ISBN

by Addison-Wesley

Reproduced

from camera-ready

copysuppliedby

the

author.

programs and applications presented in this book have been included for their
value.
but are not guaranteed for any
care,
They have been tested with
The publisher does not offer
or
warranties
purpose.
any
representation, nor does it accept
liabilities
with
to
or
the
any
respect
programs
applications.
The

instructional

Reprinted

with

Copyright

All

rights

system,

corrections

\302\251
1989

No part

transmitted,

photocopying, recording,
Printed
in the United
States
EFGHIJ-DO-943210

Publishing Company

by Addison-Wesley

reserved.

or

1989

October,

in

of

this

any

publication

form

or otherwise,
of America.

or

without

Inc.

be reproduced, stored in a retrieval

means,
electronic,
mechanical,
by
any
written
of
the
publisher.
permission
prior
may

Published

simultaneously

in Canada.

To my

parents Eva and Meshulam

PREFACE

book

grew

Like many

other

This

out of my
teachers,

able to explain algorithms clearly.

hard for some students
to solve
for
them to
but it is also hard

not being

with

frustrations

that not

I discovered

is it

only

seemed to me) simple problems by themselves,

these
two parts \342\200\224
the
that are given to them. I believe that
understand the solutions
\342\200\224
and should not be separated.
are related
and the explanation
It is essential to
creation
in order to understand it fully.
to
It is not sufficient
follow the steps leading to a solution
look at the finished
product.
(what

book emphasizes the creative

how to design a new

This

show

of

sequence

Instead,

'

the

'problem

but

rather

Our goals in

provides

depth.
use

intellectual

looks more

problems

when

an algorithm is

not

they arise

as important

how

only
in

the

as teaching

as finished

to solve
future.

program,

principles.

algorithm's
than

presentan

into a

to translate
the

drawbacks,

The goal is to

so on.

particular

Teaching

the details

improvements,

turns),

wrong

in

so on.

X, the

\"problem
these

overcoming

through a creativeprocess,rather
are to show

like

and

P\\\"

of the

The

products.
problems,

the

thinking

solution.

involved
in creating algorithms, an
process
in this book. This
is
used
designing algorithms
for
and it also
known
covers
algorithms,
many
techniques
designing
in more
an elegant intuitive
for explaining
framework
the design of algorithms
we do
It does not, however,
of designing
and
cover all possible ways
algorithms,
it exclusively.
between the
The heart
of the methodology
lies in an analogy
of proving mathematical theorems by induction
and that of designing
process
further

To

methodology

program

is to

purpose

described

are not
P, program

(including possible

algorithm

algorithms

in designing

\"old-new\"

A\\

(although not always)

its drawbacks,
the difficulties

explained

teaching

how to solve new

involved

algorithm

better

thus

are

but also

A, algorithm

design. Its main

Algorithms

and
relation to other methods and algorithms,\"
it easier for a programmer
not in a way
that makes
in a way
makes
that
it easier to understand

algorithms

not

at

of algorithm

algorithm.

usually

algorithm,

first attempts
algorithm

X,

sequence

straightforward

analysis,

side

reader

the

help

methodology

combinatorial

the

thinking

for

two processes serve different

and
purposes
similar than they may appear to be. This
has been
observed by many people. The novelty
of this book is the
to
analogy
degree
We show that the analogy
which this analogy
is exploited.
encompasses
many known
in the process of algorithm
creation.
algorithm-design techniques,and helps considerably
1 and is introduced
in Chapter
The methodology
is discussed briefly
more
in
formally
achieve

algorithms.

Although

different types of results, they

Chapter 5.

these
are

more

vi

Preface

car, and

the following analogy.

Consider

to get

directions

want

to

your

Suppose that

you

at an unfamiliar

arrive

would be

You

hotel.

impatient

quite

rent

city,

were

if you

and so on. You

of the city, its general layout, the traffic patterns,
for
two
turn right, go
of
the
form
directions
blocks,
\"go straight
would
would
outlook
so
on.
and
three
However,
miles,\"
for
change if you
your
straight
around
for a while
You
could
for
a
time.
that
in
to live
probably get
city
long
planned
who
those
of the second form (if you find someone
directions),
directions
gives you
with
the city.
This book is not a source of
but eventually you will need to know more about
easy directions. It does contain explanations of how to solve many particular problems,
methods.
As a result, the
is
book
is on general principles and
the
but
emphasis
I
the
well
believe
that
extra
effort
is
and
involvement
demands
It
thinking.
challenging.
told

the history

about

rather

have

worthwhile.
nonnumeric
The design of efficient
diverse fields, including
mathematics,
This book can serve as an introduction

general.

is becoming important
molecular
biology, and

in

algorithms
statistics,

and to

to algorithms

Many professionals, and even scientists

not

nonnumeric

deeply

many

engineering.

computations

involved

with

in

computers,

more
than
nonintellectual work. It
grungy
programming is nothing
But such a belief may lead to straightforward,
solutions,
trivial, inefficient
of
to
convince
solutions
exist.
One
this
book
is
more
efficient
where
goal
elegant,
is an elegant discipline, as well as an important one.
readersthat algorithm
design
The bookis self-contained.
The
is mostly intuitive,
and technicalities
presentation
are either kept to a minimum
In particular,
or are separated
from the main discussion.
from
the algorithm-design
ideas as much
as
implementation details are separated
There
are
of
that
were
possible.
many
algorithms
designed especially to
examples
illustrate
the principles
in
the
book.
The
in
this
book
is not
material
emphasized
as
to
be
series
of
mastered
and
memorized.
It
is
as
a
presented
something
presented
and so on.
ideas, examples, counterexamples,
modifications,
improvements,
Pseudocodesfor most
the descriptions.
Numerous exercises and
algorithms are given following
a discussionof further reading, with a relevant bibliography,
follow
In
each chapter.
most chapters, the exercises
are divided into two
drill exercises
and creative
classes,
exercises. Drill exercises
are meant to test the reader's
of the specific
understanding

believe

that

is.

sometimes

examplesand
the reader's
particular
(those

ability

algorithms,
whose

presented

algorithms

numbers

to

use

to solve

new

problems.

are underlined)

also include a summary of the main

The book is organized as

material. Chapter 2 is

Creative exercisesare meant to test

developed in that chapter, in addition to the
Sketches
of solutions to selected exercises

in that chapter.

the techniques

an

ideas
follows.

introduction

are

at

given

the

end of

the

book.

The

chapters

introduced.
Chapters

to

through

mathematical

4 present

induction.

introductory

Mathematical

to algorithm
with
is, as we will
see,
very
important
design.
Experience
induction proofs is thereforevery helpful.
few computer-science students
Unfortunately,
2 may be quite
for some
difficult
get
proofs.
enough
exposure to induction
Chapter
the more difficult examples at first reading,
students. We suggestskipping
and returning
to them later. Chapter 3 is an introduction
to the analysis of algorithms.
It describes
the
and gives the basic tools one needs
to be able to perform
process of analyzing
algorithms,
induction

Preface
of the algorithms

analysis

simple

Readers

the book.
Chapter 4 is
with basic data structures

presented in

vii

a brief introduction

and
who have a
from
5
is
a good idea
background
directly
Chapter (it
always
5
the
basic
ideas
behind
the
though).
Chapter
presents
approach
the analogy to induction
It gives several
of designing
through
algorithms
proofs.
and describes their creation. If you read only one chapter
algorithms,
examples of simple

structures.

data

to

basic mathematical
to read the introduction

in

There are
book

the

can start

read Chapter

book,

this

5.

ways to organize a book on

to the subject of the algorithms,

basic

two

according

geometric algorithms.
Even

Another

Chapters

organization.

way

of

the emphasis

though

familiar

are

who

is to

divide the

this

book

is on

through

present

for

book

graph algorithms,

example,

to design

according

design techniques, I have

algorithms

techniques.

chosenthe

areas:

four

in

divide

way is to

One

algorithms.

former

for

algorithms

sequence comparisons, data compression),graph

trees, shortest paths,
matching),
geometric
algorithms
(e.g., spanning
(e.g.,
and
numerical
and
matrix
intersection
problems),
algebraic
algorithms
(e.g.,
Fourier
I believe that this organization
is clearer and
fast
transform).

sequences

sets

and

algorithms

convex hull,
multiplication,

sorting,

(e.g.,

easier to follow.
10

Chapter

earlier

chapters,

to reductions. Although
the subject is unique
and
important
is devoted

chapter also serves as an opening

of
This
of
NP-completeness.
aspect
subject

own. This

of algorithm

part

theory.

and

completeness

the

algorithms.

of parallel

computation.

The material
which leaves
include

parts

book is

this

of Chapters

for the

more

parts

course,

of these

and can

appear

of its

a chapter

theory has become

should know
algorithms

the

with

deals

in

essential

an

NP-

about

this property. Chapter 12is an introduction

different
models
interesting algorithms under

than

instructor.

3, 5, 6, 7, and

advanced

for a first

several

contains

choices

many

them. The more

are optional

in

It

enough to warrant
to Chapter 11, which

complexity

designs

for proving

techniques

to parallel

who

Anyone

act

of reductions

examples

8 in

can
A

be covered
first

some depth,

chapters, along

not

although

with

be used as a basisfor

a one-semester

in

in algorithm

course

Chapters
a more

course,

design

should

all of

necessarily
9, 10,

advanced

11, and
course.

12,

Acknowledgments
and

First

here

foremost

on which

methodology
\342\200\224

more

I thank
this

throughout

important

my wife Rachel for helping

She was instrumental

adventure.

the book is based. Shecontributed

else \342\200\224
sound
advice.

than anything

me

in more
in

the

suggestions,

I could

ways

than

corrections,

not have

list

I can

development

done it

of

the

and
without

her.

thanks are due to Jan van Leeuwen

for an excellent and thorough
of
review
of this book.
His detailed comments, numerous
and
suggestions,
many
the book enormously.
I also thank
corrections have improved
Eric
Bach, Darrah Chavey,
Kirk
read
and Sun Wu, who
and
made
Pruhs,
parts of the manuscript
many helpful
T.
Almes
H. Chan
comments, and the reviewers
(Rice
Guy
University),
Agnes
Dan
Gusfield
of
David
Harel
California,
(Northeastern University),
Davis),
(University
of
Israel), Daniel Hirschberg (University
California,
(Weizmann
Institute,
Irvine),
Special
a large portion

Preface

viii

of Iowa),
of Washington),
Victor Klee (University
H. Kingston (University
of
of New
Michael
J.
Martel
California,
Davis),
(University
Quinn
(University
Charles
M.
Diane
Madison
and
(
James
Spresser
University).
Hampshire),
who failed to supply
I thank the people at Addison-Wesley
pie with any examples
authors
are
so
fond
of
that
were
stories
of horror
very helpful and
telling.
They
In
I
and understanding.
thank
particular,
my
production
supervisor
patient
incredibly

Jefferey

Bette Aaronson, my editor

guided me
the

Tektronix,

Award,

a Linotronic

300

Jim

do

DeWolf,

and AT&T,

for
Digital

and
way

my copy editor
even when they

financial

Lyn

Hewlett

Corporation,

Equipment

who

not only

thank

I also

better.

a Presidential

through

support,

Dupr6,

knew

Young

Packard,

and

funds.

designed
at

my

things

Foundation

for matching

The book was

on

let me

Science

National

Investigator

also

but

the

typeset by me. It
of Computer
Department
and

was

formatted

Science,

University

in troff, and
of

printed

Arizona.

Ralph Griswold for his advice, and John Luiten, Allen Peckham, and Andrey
\342\200\22
the typesetting.
The figures were preparedwith gremlin
for technical help with
Yeatts
for Fig. 12.22, which
at the University of California,Berkeley\342\200\224
was
except
developed
was
drawn
Townsend.
The
index
and
with
the
of
a
by
Gregg
compiled
designed
help
I
thank
and
Brian
for
me
the
Kernighan [1988].
Kernighan
system by Bentley
supplying
within
minutes after I (indirectly) requestedit. The cover
code
was done by Marshall
Henrichs, basedon an idea by the author.
I must
was prepared by the typesetter.
stress, however, that the final manuscript
He was the one who decided to overlook many
comments
and suggestions
of the people
listed here. And he is the one who should bear the consequences.
thank

Tucson,

Udi Manber

Arizona
(Internet

address:

udi@arizona.edu.)

CONTENTS

Chapter 1

Introduction

Chapter2

Mathematical

2.1

Introduction

2.2

Three

2.3
2.4
2.5

Counting Regions in the Plane

A Simple Coloring Problem
A

More

Simple

2.7

Euler's

2.8

2.9

Gray Codes

2.10

Finding

2.11

Arithmetic

Summation

Complicated

Problem

16
17

Inequality

Formula
in Graph

Problem

Paths
Geometric

versus

18
20
23

Theory

Edge-Disjoint

2.12 Loop Invariants:

2.14

11
13
14
15

Examples

Simple

2.6

2.13

Induction

Converting

in a Graph
Mean Theorem
a Decimal

Number to

24

26

Binary

28

Errors

Common

29

Summary

Bibliographic Notes and Further

30

Reading

31

Exercises

Chapter

Analysis

3.1

Introduction

3.2

The

3.3
3.4
3.5

3.6
3.7

37

of Algorithms

0 Notation

Time and SpaceComplexity

Summations

Recurrence

Relations

3.5.1

Intelligent

3.5.2

Divide and

3.5.3

Recurrence

Guesses

Conquer Relations
Relations

with

Full History

Useful Facts
Summary

Bibliographic

Exercises

Notes

and Further Reading

37
39
42
43
46
47
50

51
53
55
55
56

IX

Contents

Chapter

61

Data Structures

4.1
4.2

4.3

61

Introduction

4.2.1

Elements

4.2.2

Arrays

4.2.3

Records

4.2.4

Linked

Lists

Trees

of Trees

4.3.1

Representation

4.3.2

Heaps

4.3.3

Binary Search Trees

4.3.4

AVL

71

Trees

4.4

Hashing

4.5

The Union-Find

4.6
4.7

62
62
63
63
64
66
67
68

Structures

Data

Elementary

75

78

Problem

80

83

Graphs

84

Summary

85

and Further Reading

Notes

Bibliographic

86

Exercises

Chapter 5 Designof Algorithms

5.1

91

Induction

by

91
92
95
96
98

Introduction

5.2

Evaluating

5.3

Maximal InducedSubgraph

5.4

Finding

5.5

The Celebrity

5.6

A Divide-and-Conquer

5.7
5.8

Computing BalanceFactors

Polynomials
One-to-One

the

Finding

Mappings

Problem

Strengthening

5.10
5.11

Dynamic Programming:
Common Errors

5.12

Summary

the

Induction

Skyline

102
104

Problem

Trees

in Binary

Consecutive

Maximum

5.9

The

Algorithm:

106

Subsequence

107

Hypothesis

The

Problem

Knapsack

108

111

112

Bibliographic

113

and Further Reading

Notes

114

Exercises

Chapter 6

6.1
6.2
6.3

6.4

and

Sequences

Involving

Algorithms

Sets

119

119

Introduction

120
125
127
127

Search and Variations

Binary
Search
Interpolation
Sorting

Sort and

6.4.1

Bucket

6.4.2

Insertion

6.4.3

Mergesort

Sort

and

Radix

Sort

Selection

Sort

130

130

xi

Contents

6.4.4

131

Quicksort

137

6.4.5 Heapsort

6.5

6.4.6

Order

Statistics

141

for Sorting

Bound

Lower

143

6.5.1

Maximum

6.5.2

Finding

Minimum

Elements

143

fcth-Smallest

Element

144

and

the

6.6

Data Compression

145

6.7

String Matching

148

6.8

SequenceComparisons

155

6.9

Probabilistic

158

Algorithms

6.9.1

Random

6.9.2

A Coloring Problem

6.9.3

6.11

6.12

161
Probabilistic

for Transforming

Technique

into

Algorithms

6.10

160

Numbers

Deterministic

161

Algorithms

164

a Majority

Finding

Three ProblemsExhibiting
6.11.1 LongestIncreasing
6.11.2

Finding

6.11.3

Computing the

Interesting

167

Techniques

167

Subsequence

Two

the

Proof

Largest

Elements

of a

Mode

in

169

a Set

171

Multiset

173

Summary

Notes and

Bibliographic

Further

173

Reading

175

Exercises

Chapter 7

Graph Algorithms

185

7.1

Introduction

185

7.2

Eulerian

187

7.3

Graph Traversal

Graphs

189

7.3.1 Depth-First
7.3.2

7.4

190

Search

198

Breadth-First Search

199

Sorting

Topological

7.6

Single-Source Shortest Paths

Minimum-Cost
Spanning Trees

7.7

All

7.8

Transitive Closure

7.5

7.9

Shortest

7.9.2
7.9.3

7.10

208
212

Paths

of

Decompositions

7.9.1

201

214
217

Graphs

217

Biconnected Components
Connected

Strongly

Examples of the

Use

Components
of Graph

226

Decomposition

7.10.1

Perfect

7.10.2

Bipartite

Matching
Matching

7.11

Network Flows

7.12

Hamiltonian

7.12.1

230
234

Matching

Very

Dense

Graphs

.234
235

238
243

Tours

Reversed

in

Induction

244

xii

Contents

7.12.2

7.13

Hamiltonian

Finding

Cycles

in Very

Dense Graphs

Summary

and Further Reading

Notes

Bibliographic

Exercises

Chapter 8 GeometricAlgorithms
8.1

Introduction

8.2

Determining

8.3

Constructing

8.4

Convex

8.5

265

a Point

Whether

265
266
270

Is Insidea Polygon

Polygons

Simple

273

Hulls

8.4.1

A Straightforward

8.4.2

Gift

8.4.3

Graham's

Approach

273

274

Wrapping

Scan

275

Closest Pair

278

8.6 Intersectionsof

8.7

and Vertical

Horizontal

281

Line Segments

285
286
287

Summary

and Further Reading

Notes

Bibliographic

Exercises

Chapter 9

Numeric

and

Algebraic

293

Algorithms

9.1

Introduction

293

9.2

Exponentiation

294

9.3
9.4

Euclid's Algorithm
Polynomial Multiplication

297

Matrix

Multiplication

301

9.5

9.6
9.7

298

Algorithm

301

Algorithm

301

9.5.1

Winograd's

9.5.2

Strassen's

9.5.3

Boolean Matrices

304

Transform

309

The

Fourier

Fast

316

Summary

Bibliographic Notesand

Further

316

Reading

317

Exercises

Chapter

244
246
247
248

321

10 Reductions

10.1

Introduction

10.2

Examples

10.2.1

10.3

321

323

of Reductions
A

Simple

String-Matching

Problem

10.2.2 Systems of Distinct

Representatives
A Reduction Involving
10.2.3
Sequence
Comparisons
10.2.4 Finding a Triangle in Undirected
Graphs
Reductions Involving
Linear
Programming
10.3.1 Introduction and Definitions
of Reductions to Linear Programming
10.3.2
Examples

323

323

324

325

327

327

329

Contents

10.4

10.4.1 A

for Finding

Bound

Lower

10.4.2

Reductions

Simple

331

Bounds

for Lower

Reductions

331

Simple Polygons

333

Matrices

Involving

10.5 Common Errors

334

10.6

336

Summary

Notes and

Bibliographic

Further

336

Reading

337

Exercises

Chapter 11

341

NP-Completeness

11.1

Introduction

11.2

Polynomial-Time

11.3

Nondeterminism

341

342

Reductions
Cook's

and

11.4 Examplesof

Theorem

344

Proofs

347

NP-Completeness

11.4.1 Vertex Cover

348

11.4.2 Dominating

348

11.4.3

Set

350

3SAT

351

11.4.4 Clique

11.4.5

352

3-Coloring

11.5

11.4.6

General

11.4.7

More

355

Observations

356

Problems

NP-Complete

Problems
Techniques For Dealingwith NP-Complete
11.5.1 Backtracking and Branch-and-Bound

11.5.2 Approximation

with

Algorithms

357

358

Guaranteed

363

Performance

11.6

368

Summary

Notes and

Bibliographic

Further

368

Reading

370

Exercises

Chapter 12

xiii

Parallel Algorithms
12.1

Introduction

12.2

Models

12.3

375

375
of Parallel
for

Algorithms

376

Computation

378
379
380
382

Machines

Shared-Memory

12.3.1 Parallel Addition

12.3.2

Algorithms

Maximum-Finding

12.3.3 The Parallel-PrefixProblem

12.4

Ranks

12.3.4

Finding

12.3.5

The Euler's

in

for Interconnection

Algorithms

Lists

Linked

385

Tour Technique

12.4.1

Sorting

on

12.4.2

Sorting

Networks

12.4.3

Finding the

12.4.4

Matrix

12.4.5

Routing

387

Networks

389

390

an Array

Multiplication
in a

393

fcth-SmallestElement
Hypercube

on the

Mesh

on

a Tree

396

398

401

xiv

Contents

12.5

SystolicComputation

404

Matrix-Vector

12.5.1

12.5.2 The Convolution

12.5.3

12.6

Sequence

404

Multiplication

405

Problem

407

Comparisons

409
409

Summary

Bibliographic Notesand

Further

Reading

411

Exercises

tions

to Selected

Exercises

417

References

445

Index

465

CHAPTER

INTRODUCTION

Great

of

has been

importance

necessary

and

For a

claim

condition

sufficient

sciences.

exact

of

no

Necessary,

be useful
to serve as a

mental effort, for it

things, it must first of all possess
to
more than
see
something

but

doubt,

to

construction

to this process
to see in it the
the progress of the

attached

rightly

some

and

\"construction,\"

and

not
mere

not

to higher

stepping-stone

a kind
the

sufficient!
waste of

enabling us
juxtaposition
of its
of unity

elements.
Henri

The Webster's

solving a

of steps

number

New

Ninth

mathematical

by-stepprocedure

starting

fires,

as \"a procedure for

dictionary defines an algorithm
of
the
common
divisor) in a finite
(as
finding
greatest
involves
or broadly, a stepa repetition of an operation;

frequently
for

building

a problem or accomplishingsome end.\" We

of algorithms is thus an old field of study.

solving

the broad definition.

The
always been interested in

1902

Collegiate
problem

that

Poincare,

design

pyramids,

methods

better

finding

or

the

sorting

to achieve

mail.

their

The study

goals,

stick

will
People

whether

to

have

those be

of computer algorithms

is

new. Some computer algorithms

use methods
developed before the invention
of computers,
but most problemsrequire
new
For one thing,
it is not enough
approaches.
to tell a computer to \"look over the hill and sound the alarm if an army is advancing.\"
A computer must know the exact meaning of \"look,\"
how
to identify an army, and how
of course

the alarm (for

receives its instructions
process to translate
regular

to sound

some reason,
via

instructions

necessary process, called programming,

millions

of people.

sounding

well-defined,

an alarm

limited

primitive

to a language that
is now performed

is

always

operations.
a computer

on one

easy).

A computer
It is a difficult

understands.

level or

another

This
by

Introduction

understood

instructions

to devise totally

to a

\"talk\"

what

possible.

with

dozens,

perform

of

used to

We are

counterintuitive.

of

millions

this order

learning

magnitude

is a tendency when
result,
and feel.
that works very
well
approach
straightforward
that work well for small problemsmay
algorithms
and inefficiency
of the complexity
easy to lose sight

in

designing

terrible

be

of

need

weird

is knowing

it

performed
was

than

by

ever

of items, and, at
or even trillions,
instructions per
primitive)
is something
new. It is in
terms of things
we can see
an algorithm
to use the

small

for

more

the

hundreds

with

(their

thinking

there

As

cases, we

to program;

much

maybe

deal

can

Computers

on

algorithms

Designing

respects

many

instructions.

and can

information,

of

bits

second.

of

thousands

past dealt

the

of

Algorithms

with

most,

not just

hard

well-

translating

just

In most

that were previously

operations

only

it

than

is

speed, computers can do

enormous

their

with

that makes

computer

to say. Computers execute not

humans;

a problem. It

for solving

methods

new

in which we

language

of

however,
requires more
computer,
a computer can understand.
to a language

Programming

an

billions,

problems. Unfortunately,
for large problems. It is
algorithm when applied to

large-scalecomputations.
too

not

to expend

efficient and

of

the problem

the

algorithm.

perfect

grocery
it,

we perform

algorithms

It is

too often.

performed

unpacking

is organized.

kitchen

The

problem.
not

ways of doing

efficient

more

the

way

to develop

effort

of

lot

consider

example,

the

aspect to this
and are
complicated

is another

There

life are

depending

in

our

daily

worthwhile

The payoff is too small. For

There are obviously less
bags.
on the contents of the bags and

Few people spend time

for

usually

not

even

thinking

about

this

other

hand, people who do

developing
algorithms
commercial
must
and
large-scale
develop good methods. Another
packing
unpacking
is
the
lawn.
We
can
the
the number of
example
mowing
improve
mowing
by minimizing
the
total
time
for
or
the
of
the
to
the
cans.
turns,
mowing,
trips
Again,
length
garbage
unless one really
hates
the lawn, one would
not spend
out how
an hour figuring
mowing
to save a minute
of
on the other
can deal with
hand,
mowing.
Computers,
very
have
to perform those tasks many
times.
It is
tasks, and they
complicated
may
worthwhile to spend a lot of time designing better methods, even if the resulting
to understand.
The potential
of a payoff is
algorithms are more complicatedand harder
much greater. (Of course, we should
not overoptimize,
of
hours
spending
programming
time to save overall a few seconds
of computer time.)

problem,

These
algorithms

learning
always

less

much

and

\342\200\224
the

issues

two

need

this subject.
the best. It is

for

complexities of
First, we must realize

the possible

important

of course, to learn new

methods for algorithm
design.
need

to continue
methods.

it.

the

On

approaches to large-scale

counterintuitive

these algorithms
that

\342\200\224

straightforward

the search for better

This

book

surveys

point

difficulties in
methods are not

to the

intuitive

methods. To do that,
and

illustrates

we

numerous

But it is not enough

of
to learn even a large number
to memorize many
of chess in order
to be a good
games
the principles
behind the methods.
One must know how
to
player. One must understand
when
to apply them.
apply them and, more important,
of an algorithm
A
and implementation
is analogous
to a design and
design
methods,

just as

it

is not

enough

Introduction

house.

of a

construction

the basic

with

start

We

concepts, based on

the

for

requirements

architect's job to present a plan that satisfies the requirements. It is

(so that the house will
job to make sure that the plan is feasible and correct
after a short
It is then the builder's
not collapse
based
while).
job to construct the house
on these plans. Of course,all along the way, the costs associatedwith each step must be
into account.
Each job is different,
but
analyzed and taken
they are all related and
A design of an algorithm
also starts with
the
basic
ideas and methods.
intertwined.
a
is
made.
We
must
the
correctness
of
the
and
make
sure that its
Then, plan
prove
plan
is
to implement the algorithm
for a particular computer.
cost is effective. The last step
we can divide the process
into four steps: design, proof of
oversimplification,
Risking
and implementation.
each
of these steps is different,
but
correctness,
analysis,
Again,
in
are
all
related.
None
of
them
can
be
made
a
without
a
to
the
vacuum,
they
regard
in
these
linear
in
all
order.
Difficulties
arise
One rarely goes through
others.
steps

the house.

It

is the

the engineer's

in
which
They usually require modifications to the design,
phases of the construction.
another
of
and
of
turn require
costs,
feasibility
proof, adjustment
change implementation.
This book concentrates on the first step, the design of algorithms.
our
Following
been
entitled
The Architecture of Algorithms.
However,
analogy, the book could have
has a different
so using this term would
be confusing.
architecture
meaning,
computer
A
all
the
other
discussion
of
The book does not, however,
correctness,
aspects.
ignore
\342\200\224
in detail
the description of most algorithms
for
follows
analysis, and implementation
for others.
The emphasis is on methods of design.
some algorithms, briefly
to be a good architect
and
to be able to
to learn many algorithms
It is not enough
behind
the design. We
One must understand the principles
design new algorithms.
of explaining
employ a different
way
algorithms in this book. First, we try to lead the
we strongly
believe that the best way to learn how
reader to find his or her own solution;
to create something
is to try to create it.
and more important, we follow a
Second,

for

methodology

The methodology,
process.
algorithms that helps this creative
the
intuitive
framework
for explaining
an
[1988], provides
elegant
the
to approach
in
more
It also provides a unified
depth.
way
and their
methods
that are encompassed by this methodology,

designing

in Manber

introduced

of algorithms

design

design. The different

numerous

variations,

are

among those many

possible

for a

however,

great

analogy between

the

possible

of the

majority

The methodology

and applying them

methods

not cover all

does

methodology

mathematical

process

of

ways

algorithms

is based on

intellectual

of choosing
same technique. The process
becomes more methodical.This

of the

instances

in

this

designing
book.

induction.

of proving

algorithms.
The

heart of

It is
it

lies

useful,

in an

mathematical theorems and that

in the principle of mathematical

combinatorial algorithms. The main

idea
statement need not be proven from scratch: It is sufficient to show
the correctness
of the statement follows from
the correctness
of the same statement
smaller
instances and the correctness
of the statement for a small base case. Translating
this
to algorithm design suggestsan approach
that concentrates
on extending
principle

designing
induction

The

two

(Houghton

is that a

wonderful
Mifflin,

books
1985),

by Tracy Kidder,
inspired this analogy.

The Soul of a New

Machine

(Little

Brown, 1981),

and

of

that

for

House

Introduction

Given a problem, if we can

of large problems.
of small problems to solutions
we
for smaller inputs,
then
of
the
same
a
solution
it by using
solve
to
how
problem
show
a
solution
on
on
rather
than
is
to
concentrate
idea
basic
The
building
extending
are done.
of doing
As we will show in the following chapters, there are many
ways
it from scratch.
design techniques.
this, leading to many algorithm
as a tool for explaining
and
induction
We use mathematical
designing
mainly
to formalize or axiomizethe approach.
make
little
We
attempt
algorithms.
high-level
solutions

Gries
done by several people, including Dijkstra [1976], Manna
[1980],
others.
This
and
Paull
book
Dershowitz
[1988],
[1983],
among
complements
[1981],
but of course whenever something
Our goal is mainly
books.
other
these
pedagogical,
we
better.
is
understood
it
better
can be explained
Among the proof techniques
usually

has been

This

the
induction
discuss are strengthening
sequence
hypothesis,
choosing the induction
is
The significance of our approach
and
reverse
induction.
wisely, double induction,
one
of algorithm design under
different
two-fold.
First, we collect seemingly
techniques
known
mathematical
second, we utilize
umbrella;
proof techniques for algorithm
design.
it opens the door to the
of powerful
since
use
The latter is especially important,
in
for
another
have
been
that
developed
discipline.
many
years
techniques
One notable weaknessof this approach is that it is not a universal approach. Not
the principle
be designed
with induction in mind.
can or should
all algorithms
However,
in
it is worthwhile
to
is so prevalent
the
of algorithms that
of induction
design
concentrate on it. The other principles are not ignored in this book. A common
criticism
of almost any new methodology
is that, although
an interesting way to
it may present
that
were
is
in
This is a valid
it
of
no
them.
created,
explain
things
help
already
creating
since only the future
will
is and how
tell how effective a certain methodology
criticism,
used
it
becomes.
I
is
another
tool for
believe
that
induction
not
widely
strongly
only just
is
in
but
it
order
to
understand
them.
explaining algorithms,
necessary
Personally, even
I

though

methodology,
algorithms

a good

had

I found
more

experience
it

quickly

and,

helpful,

(Manber

and

in

algorithms

developing

at least
McVoy

in

two

[1988],

cases,

Manber

without

following

this

me to develop new
and Myers [1989]).

it led

Notation for Describing Algorithms

In

addition

development,
including

to
we

the
describing
also include

programs

is to

algorithms

pseudocodes for

enhance

optimize the programs, and we do

we made a consciousdecisionnot
it introduces

the

through
many

creative
algorithms.

process
The

of

purpose

their

of

made a great effort to

descriptions.
not recommend simply copying them. In some cases,
to include
the most optimized version of the program,

the

We

have not

additional
which distracts from the main
ideas of the
complexity,
in detail how we translate
the algorithmic
ideas
algorithm. We sometimesdo not explain
into a program. Such translations
The
sometimes
are obvious and sometimes are not.
in this book, as we mentioned,
is on the principles of algorithm
emphasis
design.
For the most part,
we use a Pascal-like
In
language (sometimes even pure Pascal).
into a table,\" or \"check
as \"insert
many cases, we include high-level descriptions(such
the set is empty\") inside a Pascal code to make
whether
it more readable. One notable
exception we make to the rules of Pascal is the use of begin and end to encompass

because

Exercises 5
at the beginning
and end of the programs,
and
This
convention
saves
without
separate
space
causing
We usually do not include
declarations
of variables and data types
precise
ambiguities.
in cases where such declarations are clear (e.g.,we may say that G is a graph, or that T is

let the

these statements

include

We

blocks.

only

blocks.

the

indentation

a tree).

Exercises
whose
Exercises

are underlined have

numbers

by a star

marked

are judged

by the

solutions at

author to be substantially

the back
more

of the

difficult

book. Exercisesthat are

than other exercises.

The exercisesin

this chapter do not require

any previous
knowledge of algorithms.
They address
for specific
relatively simple problems
inputs. The reader is asked to find the answers by hand.
is to illustrate
in dealing with
exercises
the difficulty
a very large
The main purpose of these
In other words, one of the goals
of these exercises is to cause frustration
number of possibilities.
here will be discussed in the following
with straightforward methods. The problems
given

chapters.

1.1

Write

1.2

Write
about

the numbers

down

them in

100 each

to

on a

separate

card.

Shuffle

the cards

and

rearrange

order again.

down the following 100 numbers

each on a separate card
1.1.
between this exercise and Exercise

and

sort

the cards.

Think

the differences

11 72 971 8234 2223849283

3295
32918 21192 11923423388231
8312
29347 3102 32883 20938293016823 9234 9236 29372 2218 9222 21202
2831
83721
9238 8221 30234 93920 81102 101118152
29133
9229 10039
9235 48395283237927 73492 8402 48201 38024 2800 321552273 82930
2221 3841 311 30223809929920
74212 7011 1823 903 29919335
28349
29123
28910 29281 3772 20012 7045830572
38013
72032 28001 83835
3017 9262673825 29263 2017 262 8362 77302 85933826 9374 2001
444
83261 48402 4845 79794 27271 3999222836
2937 37201 37322

49472 113292253
1.3

the following list of numbers.

of those numbers as
Your job is to erase as few
that the remaining numbers appear
in increasing
order. For example,erasing
everything
except the first two numbers leaves an increasing
sequence; erasing everything
for first, third, sixth, and eighth
does the same (but fewer numbers are
numbers,
except

Consider

possible

such

erased).

9 44 32

12 7 42 34 92 35 37 41 8 20 27 83 6461 28 39 93 29 17 13 1455
72 23 73 99 1 2 88 77 3 65 83 84 62 5 11 74 68 76 78 67 75 69 70 22
25 26

21 66

7124
1 -4

Solve Exercise

1-5

Suppose

that

29, 41, and

and

8 cents

1.3, such

in a
67

strange

the remaining

that

Find

(1808 cents).You

a combination

have

decreasing

order.

are five types of coins with denominations

of these coins to pay the sum
enough coins of each type in your pocket.

there

country

(all cents).

numbers are in

of 15, 23,
of 18 dollars

Introduction

The

list of

is a

input

waiting

it

particular,
k will

answer

cannot

problem is to
(any

of integers

pairs

an answer from

for

y.

any questions

a sequence

find

do). If such a

since everyone

is waiting

You can use a

pencil

of pairs

(You may draw

figure.
general method of solving

of

a piece

such

proceed,

any kind of computation,

involving
you cannot draw
any kind of a
a
particular
input, to help you design

make

and

paper,

creating

however,

tables);

unrelated to

figures,

some k >

can

else.

for someone
and

x |), for

No one

is a deadlock.

there

for

waiting

x is

and, in
it. The

else,

anything

be

\342\200\242
\342\200\242
\342\200\242,
(**_, xk\\ (xk

x^),

(x | jc2), (x2

is that

of a pair (jt, y)

do

that may

others

from

sequenceexists, then

(e.g., comparisons,

numbers

given below. The meaning

it cannot
x is waiting,

When

this

a problem.)

24 1, 25 10,35

45, 36 37, 3842,

22, 29 2, 3048,
31 15,32
4,48
17,6 45,6 1,5 35,5 20,5
10,49
32,7 31,7 4,
5 33, 6 29,6 12, 6 11, 6 3, 6 17,45 27, 47 34, 48 20, 7 40, 7 34, 8 11, 9 19,
II 30, 11 4, 11 22, 11 25,20 24, 21 23, 21 46, 22 47, 23 49, 3 39, 3 34, 4
14,4 37, 5 42, 5 8, 15 2, 15 50, 154, 1537, 16 13, 17 38, 18 28, 198, 26
15, 26 42, 27 18,28 35, 13 36, 13 50, 1334, 13 22, 29 34, 29 38,29 30, 29
16, 44 33, 44 36, 44 7, 44 3, 44 32,44 21, 33 9, 33 21, 33 35, 33 19, 33 41,
16, 2

39 41,

21, 2 25, 2 22,23

12 37, 1223, 123,

50,

23 47,

12 20,

9, 42 3,

14 25,41

7, 36

43 5,43

28,511,48

26 10,2644,2616,26
39,2617
two-dimensional 15 by 15 table given in Fig. 1.1. The /th row and the /th
correspond to the same place. Each entry in the table indicates the direct
between the places in the corresponding
distance
row and column. The \"-\" symbol
indicates that there is no direct link between the two places. The direct
distance
may not be
the shortest
distance.
There may be a shorter path between
two places going through
a third
1 and 6 is through 5 and
the shortest route between
place (or several places). For example,
12. Find the shortest route
1 and
between
15, between 4 and 3, and between 15 and 8.
input is the

The

column

Consider

(for any /)

the table

in

Fig.

1.1. Find the

shortest route

between

5 and

all

other

places.

Consider the graph shown in Fig. 1.2. Find a closed route along the edges of the graph
which includes every vertex exactly
of a
once.
(This graph corresponds to the edges
Sir William
R.
dodecahedron; this puzzle was first described by the Irish mathematician
Hamilton, and we discuss it further in Section 7.12.)

The

is a regular maze problem, with

that the maze is given
in
the exception
in a rectangle with
11
representation (rather than a picture). The maze is contained
rows and columns, numbered
the rows and
from
0 to 10. The maze is traversed
along
columns \342\200\224
The starting point is 0,0 and the target is 10,10. The
up, down, right, or left.
are obstacles you cannot traverse through:
following
points
following

numeric

(3,2) (6,6)(7,0) (2,8)(5,9) (8,4) (2,4) (0,8) (1,3) (6,3) (9,3)(1,9)(3,0) (3,7)
(4,2) (7,8) (2,2) (4,5) (5,6)(10,5)(6,2)
(4,0) (7,5) (7,9) (8,1) (5,7)
(6,10)

(4,4)(8,7)
a.
b.

Find

a path from

Find

a shortest

obstacles.

(9,2)

(10,9) (2,6)

the starting point to the

target

path from the starting point

to

that does not include

the

target

that does

any

not

of the
include

obstacles.
any

of the

Exercises

3
3

10

12

13

14

15

9
-

0
-

4
?

3
2

10

12

13

14

7
2

15

11

3
2

8
2

9
8

0
2

Figure 1.1 The table

1.11

Find

the

of two

1.12

greatest

for Exercises

1.7 and 1.8.

common divisor of 225277and 178794.

(The greatest common
of
that
divides
both
number
them.)
largest

integers is the

Compute the

value

of 2M. Try

Figure

to find

1.2

a way

to minimize the number of multiplications.

Hamilton's puzzle.

divisor

Introduction

1.13

the number of electoral votes for each state in the 1988

candidate
(the
Presidential
receiving the majority of the votes in a state collects all
There are altogether 538 electoralvotes. Determine
for
that
votes
state).
electoral
the
for the election to end up in a tie. (This problem
is
it is (mathematically)
whether
possible
it is a special case of the
known as the partition problem, and
problem
knapsack
The

list represents

following

election

discussed

in

Section

5.10.)

Alabama

Alaska

Arizona

Arkansas

California

47

Colorado

Connecticut

6
8

Delaware

Florida

21

Georgia

12

Hawaii

Idaho

Illinois

24

Indiana

4
8

Kansas

Maine

Michigan

20

Minnesota

Missouri

11

Montana

Nevada

South Carolina

4
5
3
7
8

Texas

29

Utah

Virginia

12 Washington
6 Wisconsin

New Mexico
Dakota

North

Oregon

West

Virginia

12

7
8

Iowa

9
10
10
4
4

New York

36

North

Ohio

23

Oklahoma

13
8

Kentucky

Maryland

New

Hampshire

Pennsylvania

South

Dakota

Louisiana

10

Massachusetts

13

Mississippi

Nebraska
New Jersey

16

Carolina

25

Rhode Island

Tennessee

11

Vermont

10

Washington,

11

Wyoming

D.C.

CHAPTER

INDUCTION

MATHEMATICAL

one believes an hypothesis except its originator,

but
everyone believes an experiment
except the experimenter.

No

Anon

is always the

Obviousness

enemy

of correctness.

Bertrand Russell (1872-1970)

Introduction

2.1

in the

see

will

We

In this

design.

The

examples.

proofs may

this

find

T be

value

whose

a theorem

can be

of proving

Instead

natural

any

directly

that

Readers

difficult.

algorithm
through
not

We claim

are

algorithms

constructing

role in

induction
who have
similar,

seen
the

that

and

thus

helpful.
powerful

technique.

proof

It usually works

as

to prove. Suppose that T includes

a parameter
is
number
natural
number
a
(a
positive integer).
holds for all values of ny we prove the following two
want

we

that

plays a major
to mathematical

to be relatively hard.

chapter

of constructing proofs and

with induction proofs is very
experience
Mathematical
induction
is a very
processes

follows. Let

induction

chapter, we presenta brief introduction

examples range from easy to quite

induction

many

that

chapters

following

conditions:

1. T
2.

holds for

For every

reason these
that T holds for n

The

and

so on.

two

= 2.

= 1
> 1,

if T

conditions

If T

holds for n

- 1,

are sufficient

holds for

The induction principle

= 2,

then T holdsfor
is clear.

Conditions1 and

then condition

itself is so basic that

2 implies that
it is

usually

2 imply
T holds

directly
for n = 3,

not proved; rather,

it

Induction

Mathematical

10

definition of the natural numbers.

2 is easier in many
condition
cases
Condition
simple to prove. Proving
- 1.
n
we
can
use
the
T
for
since
that
holds
theorem
the
assumption
directly,
than proving
some
In
we
induction
the
induction
the
called
sense,
hypothesis.
get
This assumption is
to one with
to reduce the theorem
value of ny
smaller
free.
It is enough
for
hypothesis
We concentrate on this reduction.
start right
scratch.
Let's
than proving it from
rather

is stated as an

is usually

an example.

with

away

in the

axiom

Theorem
\342\226\241

2.1

For all

numbers

natural

x and

n, xn

by x

is divisible

- 1.

is trivially
induction on n. The theorem
true for n = 1. We
assume that the theorem is true for n 1; namely, we assume that xn~x - 1 is divisible by
x. We now have to prove that xn -\\ is divisible by x- 1.
numbers
x- 1 for all natural
The idea is to try to write the expression xn -\\ using xn~{ -1, which,
by the induction
- 1:
x
divisible
is
by
hypothesis,

Proof: The

is by

proof

+ (x-\\).

xn-\\=x(xn-l-\\)
But

just

jc

is divisible

left term

the

by

hypothesis, and

induction

the

by

the

term

right

\342\200\224
1.

The

If a

is

\342\226\2

induction

statement

P,

the truth of

is

principle
with a
n - 1

P for

as follows:

defined

thus

parameter
implies

n,

for n =

is true

its truth

for

n,

then

\\,

and

P is

every

if for

true for

> 1,

all natural

numbers.

of using

Instead

-1

and

P,

with a

we
\302\253,

use

sometimes

n and

/i

which

+ l,

is completely

equivalent:

If a statement
the

of P

truth

for

parameter

implies

n,

its truth

for n =

is true

for

+ \\, then

\\,

and

P is

every n>\\,

if for
true

for

all natural

numbers.

The proof of

Theorem
2.1 illustrates
a simple application of induction.
have
been developed.
For example, the
many variations of induction
called strong induction, is very common.

If a statement

P,

the truth of
true

for

P for all

all natural

The differenceis that

< n
useful.

in

proving

Another

with a

parameter

natural

n,

numbers

is true

< n

for n =

\\,

implies

its

and
truth

if for
for

Over
following

the years,
variation,

every n > 1,
n, then P is

numbers.

the assumption that the statement

is true for all numbers
for n. In many cases, this stronger assumption can be very
simple variation is the following:
we

can use

the statement

Three

2.2

If a

statement

a parameter

with

P,

the truth
every n>2,
all natural numbers.

for
for

This variation
step imply
for all even

is true

for n =

implies

for all

numbers.

statement P, with a parameter

n, is true for
that n is an integer power of 2, the truth
n, then P is true for all natural numbers that

If a
such

for

This variation

n=2, and if,

n, then P is true

and

truth

its

for

for

in two parallel tracks. The base casefor

odd numbers; the base case for n =2 and the
Another
common
variation is the following:

''works\"
P

for

ofP

n,

n-2

the induction

induction

step

imply P

n>

\\,

1, and

of P

for n/2 implies its truth

integer powers of 2.

if,

every

for

n as 2k, and carrying

the parameter
from k = 0).
different
can also be used in many
Induction
ways to prove propertiesof structures
n that measures
the
other than numbers. In most cases, the induction is on some number
which
of
the
measure
to
the
induction
of
the
instance
the
size
right
problem. Finding
should be applied is not straightforward.
(For example, we could have appliedinduction
much
than
to n\\ this would
have made the proof
to x in the previous example, rather
is not natural, and it has to be invented
more complicated.) Sometimes, this measure
just
The common thread to all these proofs is the extension
for the purpose of the induction.
for larger structures.
of claims for smaller structures
to claims

Three

2.2
The

+ 2

the

to

S(l)=l = l-(1+

the

of

the

first

n natural

first

is by

proof

We

l)/2.

numbers is n{n + \\)l2.

induction on n. If n-

now assume

that

the

sum

1, then
of the

the
first

is true

claim

(n +

because

numbers

n natural

l)/2, and prove that this assumption

implies that the sum of the
numbers
is S(/i + 1)= (/i + 1)(/i+2)/2.
We know from the
definition
+ l) = S(\302\253)
+ \302\253
+1.
S(\302\253
But,
+1)/2,
by the assumption, 5(\302\253)= \302\253(\302\253
+ l)/2,
which is exactly what
S(/i + 1)= /i(/i+ 1)/2 + /i + 1 = (\302\253+2)(\302\253
n

numbers

natural

theorem.

following

2.2

The sum of the

Proof: The

for the sum

expression

We prove

\342\226\241
Theorem

is

the

find

writing

Examples

\342\200\242
\342\200\242
\342\200\242

+n.

by

k (starting

parameter

Simple
is

problem

S(n)=

from

follows

induction for

out the

the first one

n = 1 and

n =

are

11

Simple Examples

n +

first

S(n)
natural

and

S(n) that
therefore

we

wanted

of

prove.

to

\342\226\

We

compute

continue
the

sum

with a slightly
more
sum. Suppose
complicated
= $ + 13+ 18+ 23+ \342\200\242\342\200\242\342\200\242
sum
+(3 + 5\302\253). The

T(n)

that

we

in the

want

to

previous

of the elements
to n2l2 + nl2. Each
in the current
is
example
the corresponding element in the previous
Hence,
example.
it is reasonable
to guess that T(n)
is also a quadratic expression. Let's try the implicit
That
the parameters c,, c2, and c3, and
guess G(n) = cln2 +
is, we introduce
it is convenient
we can determine the
to do so. For example,
determine their values when

example,S(\302\253), is
slightly

more

than

equal
f\\\\e

times

c2n+c3.

Induction

Mathematical

12

the first few terms. If n =

parameters by checking
two equations:
n = 1 and n - 2, we get the following

(2) 4-c, +
If we

We

expression.

verified
+ l)

We have

now

+ l)2

77^

series

5\302\253w
fl/f/je

18 + 23+

end this

Ifn is a

+3

5.5\302\253+5.5

1).

We have to

example.

simple

+jc>

0, then

(2.1)

+ (n

prove that

+ 1)jc +

assumption

+\302\253jc
for

(1 +

(by induction)

nx2

> 1 + (n

multiply

that

the

l+*>0.

= 1,

all x

jc)/,+i >

>

+JC)\302\253

were able to

clearly nonnegative.

induction on n. If

that (1 +jc)\"> 1

(1 +xy?+l = (i +JC)(i

that we

and

is by

proof

+ 1.

another

with

1+fljt.

assume

the

right

already
to prove that

\342\226\2

We

because of

the

is

have

theorem.

number

natural

Proof: The

Notice

try

that

2.4

(\\+x)n>

case of n

we

and

+ 5(/i+l)
+

+ 5\302\253
+ 2.5
2.5\302\2532

We

implies

\342\200\242\342\200\242\342\200\242
+ 5\302\253)
+(3

section

\342\226\241
Theorem

+*.

For

+ 5.5\302\253.

is2.5\302\2532

2c, =5, which

= 2.5n2+5.5n

induction.

by

C(/i)

induction)

+5.5(/i + 1)= C(/i+

following

2.3

8+ 13+

to

l) + 3 = (by

= 2.5(/i

Theorem
\342\226\241

= T(n)

G(n) = T(n),

that

+ 5.5\302\253+5\302\253
+ 8

the

guess

assume

We

(2), we get
that G(n)

from

that G(n)

prove

= 2.5a?2

proved

We

therefore

to

5(/? +

7\302\273 +

it

subtract

We

try

a base case.
= 7,(/i + l):
+ l)

7,(/?

0.

2 and

by

and c2 = 5.5.

Cj=2.5,

C(/?

= 13 +

2-c2
(1)

multiply

be

lc2 = 8

l-c, +

(1)

so c3 must

sum is 0,

0, the

then both

such

+ (n

that
1 )a\\

sides of (2.1)are
and

+jc>0,

for all

x such

that

equal

consider
1+

the

x> 0:

(1 +jc)(1 +\302\253jc)
1 )jc.

inequality

(implied

The last step was

by

the

induction)

possible

because

by (1 +jc)
nx2 is

\342\226\2

2.3

2.3

Counting

Regions in the

Counting

Regions

in the

Plane

13

Plane

if no two lines are parallel and

is said to be in general
set of lines in the plane
position
is to compute the number
no three lines intersect at a common point. The next problem
for the right
of regions in the plane formed by n lines in general position. Good hints
lines
cases.
When
n = 1, there are 2. Two intersecting
from
small
can
be
obtained
guess
a
7
at
form
It
do
not
intersect
form 4 regions; three lines that
regions. seems, at
point
<
least for / 3, that the /th line adds / regions. If this is true for all /, then the number of
section.
from S(n), which was computed in the previous
computed
regions can be easily
line is
one more
of the number of regionswhen
Therefore, we concentrate on the growth
added. The claim we are trying to prove is the following:
A

Guess:

Adding

increasesthe

one more line to

number

As we have already seen, the

induction hypothesis, and try

n-\\ lines in

is true

guess

general

position

We

can now

in the

plane

by n.

of regions

for

<3.

use

the

guess

as our

one line to n lines in general position

does not deal
Notice that the hypothesis
of
of regions
the
number
with
the
but
rather
of
with
the
number
growth
regions,
directly
when one line is added. Even if the hypothesis is true, we will still need to compute the
increases

total

the

number

number

of

of regions, but
can a new line

to prove

regions

this

that adding

by n +1.

will be

part

straightforward.

Consider Fig. 2.1. Sinceall

a region at the border; it can either
touch
are in general position, a line cannot just
lines
from
is formed), or be disjoint
cut a region into two parts (in which case one more
region
+ l
it. Consequently,
we need only to prove that the (rt + l)th line intersects exactly \302\253
we
want
to
this
but
at
the
theorem
is
to
It
point,
directly
prove
possible
existing regions.
\302\253th
for
moment
the
the
Let's
remove
of
induction
illustrate
another
proofs.
technique
n new
the Arth line,
the (n + l)th line is adding
line.
By the induction hypothesis, without
line
causes
the
that
of
the
nth
to
the
(n + l)th
prove
presence
regions. Thus, we need only
lines
are in
Since all
Let's
line
to add one additional
region.
put the Arth line back.
a
be inside
the nth and (n + l)th lines intersect at a point /?, which must
general
position,
How

increase the

Figure 2.1

number

+ 1 lines

of regions?

in general

position.

Mathematical Induction

14

R. Each line separately cuts R into two pieces, but

intersect
Both lines thus
of the (n + l)th
when the nth
into
four
R
cut
line,
pieces! So, the addition
together they
+
of
the
the
addition
when
two
R
into
cuts
But,
line,
(n
l)th
regions.
line is not present,
more
is
cut
from
two
to
four
R
two
affects
regions (R
by adding
the nth line is present,
R.

region

lines

two

the

one.
of just adding
meet at only one point.

instead

regions)

of the

presence

nth

but

line,

R is

Furthermore,
the

Hence,

it adds

n+

only
line

1th

the

with

+ 1 regions

the

so affected,

region
adds n
A?th

regions
and

line,

without

since
the

the proof is

complete.
2.5

Theorem
\342\226\241

The number of regions in

/?(/i + l)/2+l.

Proof:We

have

2 +2+
1+2

3 +

+ 3+

4 + 5+

+ l)/2;

+n=n(n

therefore,

the

regions. The first

(for a?>1) is
in
the
previous section that
number
of regions is n(n + l)/2+ 1.\342\226\2
adds

line

total

n more

of

number

seen

We
\342\200\242\342\200\242\342\200\242+/!.
have

\342\200\242
\342\200\242

nth

total

the

position is

in general

n lines

by

that the

hence,

regions;

formed

plane

proved

already

two

introduces

line

the

regions

in this proof. First,

the hypothesis
dealt
with
rather
than
the
function.
As
after,
directly
growth
the induction
of the growth of the function.
a result,
proof concentrated on the growth
is no need to define the hypothesis
There
such that it proves
the theorem directly. We
can achieve
the proof
in two or more steps. As long as we are learning
more
about the
There is no need to hurry,
or to attempt too much too
situation, we are making
progress.
Patience
quickly.
usually
pays. Second, the same induction
hypothesis was used twice
in
two
different
+ l)th
once
for
the
A?th
line
and
once
for the (\302\253
line
configurations:
as
A?th
an
line.
This
use
is
not
double
the
lesson
it
is
and
teaches
uncommon,
4'acting\"
that we should
utilize
our assumptions
to their fullest.
There are

Comments

of the

the

with

2.4

A Simple

We are

again

n distinct
in

neighboring

we

lines

assigning

were

Problem
a plane,

in

colors

have different

regions

points

interesting

Coloring

interested

Consider

two

function

to

colors

this time
the
(two

regions
regions

not

necessarily

in general

position.

by these lines such

are considered
neighbors if

formed

that

and

edge
say
possible to color\" the
follow this rule, and
of
colors
a valid coloring.
assignment
In general,
it is possible to color any planar
colors
of this fact
with
four
(the
proof
map
has occupied mathematicians
for about
a hundred years, and was found only recently).
The regionsformed
as is shown
lines, however, have specialcharacteristics,
by (infinite)

if they have
if we can
regions

only

in

the

next

in common).

an

We will
we call the

that

\"it is

theorem.

\342\226\241
Theorem

2.6

It is possible to color
with only two colors.

the regions

formed

by

any

number

of lines

in

the

plane

2.5
Proof: We

Induction hypothesis:It is
lines in the
It is

clear

plane

15

Problem

to color

possible

the regions

formed by

<

two colors.

only

are necessary

colors

two

that

with

Summation

Complicated

induction hypothesis.

the natural

use

More

n=

for

sufficient

and

1.

the

Assume

induction

to modify
and consider
the coloring
the nth line is added.
when
side
according to which
the same
as before, and
of the nth line they lie. Leave all regions on one sidecolored
To prove that this is a valid coloring,
the colors of all regions on the other
side.
reverse
R
If
both are on the same side of the A?th
and
we considertwo neighboring
\\
R2.
regions
line was added (by the induction
then
were
colored
before
the
line,
they
differently
the
reverse
but
have
If the edge
colors,
They may
they are still different.
hypothesis).
between them is part of the nth line, then they belonged to the same region before the line
colored
Since the color of one region
was reversed, they are now
was added.
differently.

the only question is how

Again,
Dividethe regions into two groups

lines.

hypothesis,

\342\226\2

The

Comments

flexibility,

as much

as

given a
was

general

or for more
coloring,

the

to handle

used

degrees

of

of new

formation

next

example

idea is

The

by the

regions

complicated. Considerthe

is more

following
=

The
of

is to

the

64

21+23+25+27+29

125

the

rows

seem

This idea

11

for the sum

to follow

of the

regular

ith

row,

pattern;

and prove its correctness.

They look like a sequence

cubes.

Induction
The

27

expression

for

an

search

the hypothesis
idea was that,

triangle.

13+15+17+19

find

sums of

the

stretch

=8

7 + 9

problem

usually

to

added line.

The

is

example

Summation Problem

2.5 A More Complicated

The

this

in

illustrated

freedom.

in order to get the most out of it. In this case, the key
we can reverse all colors and still have a valid coloring.

possible

valid

method

problem
problem

to be

vaguely

hypothesis:

The sum of row

i in

the triangle

is i3.

the hypothesis are definedin terms of a picture. It is not easy to define

let alone to solve it. In practice,
for problems
it is not uncommon
precisely,
A major part of any
defined.
is to extract the
solution
right
problem.
and

that are consistent with the picture, and solve

assumptions
other
The zth row
(It is possible to make
assumptions.)
i numbers. The numbers
contains
are the odd numbers in order.
let's concentrate
Again,
on the difference
two consecutive rows. To prove
between
that
the sum of row
i is
to show
indeed z3, we need only
that the difference between row
z'+l
and row i is
is true for i <4).
(z + l)3 - z3 (we have already seen that the hypothesis

Therefore,
the

problem

we

will

make

accordingly.

some

16

Induction

Mathematical

is the difference betweenthe first number in row i + 1 and the first number in
the numbers are the odd numbers in order and there are i of them in row i,
Since
/?
row
the second number in row i + 1
between
the difference is 2/. This is also the difference
third
the
fourth
in
row
the
number
number,
i,
number, and so on. Overall,
and the second
is
also
the
of
size
2i.
There
last
element at the end of row
each
are / differences,
there
What

i+l,

to

matched

not

is

which

previous row. Hence,

in the

number

any

of the last number

the two rows is 2/2 plus the value
= 3/2 + 3/+ 1, we need only
to
that the value of
z'3
+
prove
l)3
(/
- 2/2 = i2 + 3i + 1. This
the guess that
is where
i + 1 is 3/2 + 3/ + 1

play.

We prove

element.

of

the problem

reduced

have

We

Nested induction hypothesis: The

That is, we
the last number

differences.
i+

and

[ i2 + 3/
But

we

and

1 ]

at

problem,

this

illustrates

difference

starting

final

the

with

Thisis a very

the

sum

is i3

comes to
an

finding

+ 3/

is i2

+ 1.

by induction, to check
between the last number

1 ]

= 2/

the

only

in row

+ 2.
in row i +

numbers

corresponding

established.
that we should
policy to advance

again

from

l. Since
in row

equal to

a good

is

It

proof also

the

illustrates

a simpler

problem

try to

always

in

achieve

long as

as

stages,

the

we are

of \"going backward\" to arrive

method

problem
and simplify

method

common

not

and

our

working
it by

(not only

in

reducing

way toward

the

to simpler

and

is clearly

true

it

mathematics).

Inequality

we prove

section,

sufficient,

the

- 1)+

been

of
we start

A Simple

it is

difference betweenany

thus

step.

Instead

simpler problems.

In

+ 3(i

1f

This

progress.

a proof.

2.6

This proof
in one

proof

final

[ (i

Now,

i is

row

in

The guess has

Comments

making

1.

in row i

number

last

have to prove that

know that the

already

is 2i.

whole

for i =

is true

claim

The

finding

last

again by induction.

statement

last

the

number

i+

problem of

to a

sum

the

the

difference

the

row

in

between

the

following

inequality.

2.7

\342\226\241
Theorem

foralln>\\}

Proof:
for

we

information

'This

assume
get

to prove
that

(2.2)

from the

is usually written
of series; this formulation

inequality

knowledge

want

We

We

n-\\.

the theorem by
is true for

induction

as a

fact

hypothesis
about

is completely

convergence
finite.

The

induction.
and
\302\253,

is that

of

we

theorem

consider

the sum of

infinite

series,

but

/i
the

+ l.
first

The

only
is

n terms

we do not assume

any

less than

extend

can we

How

1.

to

it

the n +

include

1th

term?

side may potentially increase the sum to more than 1.

in a different order. Given the sum
induction
1

the last

by

^fl

+
+ \342\200\224

the

left

apply

the

+l

the induction

is to

terms:

1
1
\342\200\224
\342\200\224

here

*y\\

we look at

l/2\"+l to

Adding

The trick

hand

17

Formula

Eulers

2.7

hypothesis.

now

But

2+4+I+
we can

2\"

add 1/2 to both

and get

sides

the expression

\342\226\

(2.2)for\302\253+l.

It is

Comments
the

induction

instances where
some

The

planar
outside

next

it are

If

is easier
the

run

you

to consider

l)th element in
There are other

be a special element
be flexible, and consider

problems,

examples extend this

following

(n +

element.

element

+ l)th
(\302\253

into

as the

first

the

notion

satisfying

as many

further.

Formula
is for

proof

with

map

it

to let

better

The

can.

Eulers

The

faces.)
map

by traversing

connected.

are

said

E edges,

as one
map
to be

edges of the
Intuitively,

as

Euler's

and F

faces.

known

theorem

vertices,

is counted

region

edges and two

vertices of a
other

is

it

the last element

to consider

necessary

properties.

special

options as you
2.7

not

Sometimes

proof.

in

face, so, for

Fig.

map.

A map

face

11 vertices,
it

is possible

is called

a map is connectedif

is an

Consider a connected
enclosed region. The
has

a square

example,

2.2 has

connected if

Formula.
(A

to go from

connectedif

it consists

vertices

four

19edges,and

10 faces.

one vertex to

every

two

of one part.

2.8

\342\226\241
Theorem

The number of vertices(V), edges (E), and faces (F) in an arbitrary

V + F = E + 2.
connected planar map are related by the formula

Figure

2.2

A planar

map

with

11 vertices,

19 edges,

and

four

Two

10 faces.

vertices

the
in

Induction

Mathematical

18

Proof: We

will

The

induction.

proceeds

of induction

a variation

by

on the

first

as

known

double

and then on

vertices

of

number

the

of faces.

number

Consider first
because, otherwise,

a map
the

cycle

A connected

face.

another

all trees, V+

l=\302\243+

with only one face. Such

would
form at least one

map

base case is

is

a cycle

without

does not

a map

face and the

called a tree. We

a cycle

contain

outside
first

form
that, for

would
prove

2.

induction hypothesis:

First
The

this theorem

prove

induction

with

n vertices

have n

edges.

edges,

and consider

one edge.
if
the tree
and
we
traverse
Otherwise,
edges
then
we
are
to
vertex
from
to
return
a
vertex,
any
guaranteed
along the edge, starting
is a
stuck.
But this means that there
is a cycle, which
getting
already visited without
v along with
the edge
to it. The
connected
We can remove the vertex
contradiction.
But it has one less vertex and one
thus, it is still a tree.
map is still connected;
resulting
trees

with

There

n + 1 vertices.

less edge,which

Considera map
not

with

and

one

cycle.

of

less edge

has E edgesand

one

neighbor of the outside

edge of this cycle will

separates /

from

the

outside.

follows.

parameters. The proof

\342\226\2

used

on

induction

on
(the
required another induction
another
parameter (the number of vertices). The proof shows that we have to be careful
about
the right sequence of induction.
the induction switches from
Sometimes,
choosing
one parameter to another;
it is based on a combined value
of
several
sometimes,
and sometimes, it is applied to two different
at the same time.
parameters;
parameters
Choosingthe right sequence can make a big differencein the difficulty of the proof. As
we will see in the
the right sequence of induction
can also
following
chapters,
choosing
make a big difference
in efficiency of algorithms.

2.8

number

of faces),

but

is a

which

Removing

the edges that

the theorem
and

This theorem included three

Comments

to only

of faces.

number

with n faces

have a face/,

by a

We removeone

have one lessface

one parameter

connected

+ 2.

It must

faces.

on the
map

planar

Any

= E

+ n

is surrounded

the map.

disconnect

We now

n +

a face, it

Since /is

face.

that

such

induction

an

induction hypothesis:

vertices

two

least

at

vertex

one

the claim.

implies

This serves as a basecasefor

Main

least

at

are connected to

vertices

all

if

must be

vertices has

with n

tree

trees

that

Assume

trivial.

the

base

case

in Graph Theory

A Problem

We
first
need to introduce some basic concepts of graph
(these concepts are
theory
discussedin detail in Chapter 7). A graph G = (V, E) consistsof a set V of vertices and a
set E of edges.
Each edge corresponds to a pair of distinct
A graph can be
vertices.
directed or undirected. The edges in a directed graph are ordered pairs: The order

between
as an

the two

arrow

pointing

vertices the
from

edge

one

is important.
(the tail) to another

connects

vertex

In

this
(the

case,
head).

we draw an
The edges

edge
in

an

2.8

A Problem

in

19

Theory

Graph

in this section. The

pairs. We deal with directed
graphs
of
v.
A
the
number
incident
to
path is a sequenceof
edges
degree
that
the
are
connected
vertices
\342\200\242\342\200\242\342\200\242,
v*
(Vj, v2), (v2, v3),..., (v*_j,
by
edges
vk)
Vj, v2,
considered
to be part of the path).
u is said to be
Vertex
(these
edges are also usually
reachablefrom vertex v if there is a path from v to u. Let G = (V, E) be a graph, and U a
H = (\302\243/,
induced
set of vertices U c:V. The subgraph
F) such that F
by U is a subgraph
to U. An independent set 5
consists of all the edges in E both of whosevertices
belong
in 5 are adjacent.
such that no two vertices
G =(V, E) is a set of vertices
in a graph

undirected
of

are

graph

unordered

v is

a vertex

2.9

Theorem
\342\226\241

Let G = (V,

E)

be a

directed graph.

that every vertex

at most 2.
path of length
such

Proof: The proof is by

Induction

in

can

on the

induction

The theorem

hypothesis:

Thereexistsan
be reached

is

from a

number of
true

for

set S(G)

independent
in

vertex

in

S(G)

by a

with

< n

vertices.

all directed

graphs

vertices.

The theorem is

trivial

for

n <3.

Let

be

vertex

an arbitrary

in

V.

= {v|

Let N(v)

of v. The graph H induced by

is the neighborhood
vertices V-N(v) has fewer vertices
than
does
G; thus, we can use the
H.
Let
set of H implied by the
for
be
the
S(H)
independent
hypothesis
[w

w)e E). N(v)

V | (v,

hypothesis.

There are two

1.

u {v| is independent.

S(H)

because every vertex

in

2.

induction

induction

cases.

in N(v)

are reachable

N(v)

induction

^j

set of

the

In

this

case,

we can

is reachable from

from a

vertex

in

S(H)

with

set 5(G)
distance

with distance

be

to

S(H)

1. The
at

most

{v|,

vertices not
2 by

the

hypothesis.

be a vertex w e S(H)
that
In this case, there must
u {v| is not independent.
which implies that
is adjacent to v. Now, we S(H) implies that we V-/V(v),
w is adjacent
to v, (w, v)
of G. But, since we assumedthat
(v, w) is not an edge
in N(v)
can be reached
must be an edge of G. In that case, however, every vertex
w (through
from
v) with distance at most 2. We can set 5(G) to be S(H)
{j {vv|,

S(H)

which

completes

the proof.

of

\342\226\2

fixed.
in this proof was not
That is, we
\"reduction\"
on
instance
the
from
to
number
n
a
smaller
depending
problem
of
the problem. Furthermore, the smaller
of smaller
not
was
an
arbitrary
problem
problem
size. It depended
on the particular
We removed just enough
heavily
problem.
larger
vertices
to make the proof feasible.
There is a very
fine balance
in such proofs between
too
which
the
in
case
is
too
too
weak, and removing
removing
hypothesis
many vertices,
few vertices, in which
case the hypothesis is too strong. Finding
in
this balance
is,
many
Notice also that we used the strong induction
the heart of the induction
cases,
proof.
for all instances of smaller size.
it was required to assume the theorem
because
principle,

The amount

Comments

reduced the

size

of the

20

Mathematical

2.9

Gray
are

We

Induction

Codes
of n

a set

given

objects

a unique
scheme.

We deal

circular

in a

names

with

only
such

list

we want to

name them.

is represented by
for
a
may
many
objectives
\"good\" naming
We would like to arrange the
one objective in this example.
from
that each name can be obtained
the previous name by
and

string of bits. There

Each

name

different

be

several
are
one bit. Such a scheme is called a Gray code.2There
sensor
scan
some
to
For
a
is
better
It
codes.
of
may
example,
objects.
Gray
applications
from
one
to
the
of
next.
The
be able to change representations
quickly
object
purpose
it is possible to construct a Gray
code
for any number
is to find out whether
section
this
of objects. The objects themselves
play no part in the problem; we care only about their
exactly

changing

number.
the relationship between the names
is by using graphs.
two
vertices
of
the
and
names
are
connectedif they
graph,
correspond
A Gray code corresponds to a cycle containing
all the vertices.
differ by only one bit.
What
We start by trying small values of n. The casesof n = 1 and n = 2 are trivial.
=
3. If
3? It is not hard to see that it is impossible to find a Gray code of length
about n
we either get the same
we start with any string and change one bit twice,
string or another
with
a two-bit difference; we cannot get the same
string after three changes. In
string
is
that
it
to
construct
a Gray code of any odd
this observation
fact,
impossible
implies
is
a?
=4?
The
a
code
of
4: 00, 01, 11, 10. The
What
about
following
Gray
length
length.
is
course
a
We
now
for
our
first
of
are
square.
ready
corresponding
graph
attempt.

A good

visualize

to

way

The names

to the

2.10

\342\226\241
Theorem

There exists a Gray

Proof:

code

The proof is

by

of length
induction

a Gray code of size 2k and

a Gray code of size2k. Clearly,
if we
result
is still a Gray code. The following
there exists

Ojj, ljj,
the

Although

of

length

is

it

size

each

possible
n with

fewer

in

than

n/2

any

on k.

The case
2(k +

consider

objects

bits?

of

1). Let

k.

= 1 is trivial.

Assume

s \\ ,j2\302\273\342\200\224\302\273^2it
correspond

that
to

add a leading 0 or a leading

1 to all the strings, the
is thus a Gray code of size 2k + 2 (see Fig. 2.3):

\\s2,

0s2,

0s3,

0j4,

..., 0s2k-

with

|~log2rtl bits.

is not
number

I-,
very
of

satisfactory. The
objects. In general,

Can we construct Gray codes of

of bits, we need to add
number

To achieve a logarithmic
of objects is doubled. Let's assume

we know how to
that
construct Gray codes for all even numbers 2k, such that k < n. Given 2n objects, we try
to construct
the code from two smaller
codes each of size n.
We
run into a problem. Although
is a Gray
there
2n is even, and thus
immediately
code of that size, n may be odd, and there is no odd-sizeGray code.
we
Consequently,
one

bit

whenever

the number

integer

positive

is complete, the construction

code is at least one-half of the

proof

the
string
to represent

2k for

Gray codes usually refer

to the

case where

the number of objects

is a power of 2. We use it for all values of n.

2.9

Codes

Gray

21

05,

IS,

o_

052

lj2

2.3

Figure
able

be

not

may

to values

ourselves

code of

a Gray

Constructing

to use the induction

hypothesis
of n that are powers of 2. We

whenever
that

assume

size Ik.
n is

odd. Let's

we know

restrict

to construct

how

short) Gray codes for all powers of 2 less than n, and

to a Gray code of size nil. We can again add
j,j2,...,j\342\200\236/2 correspond
the
two
0s
or
such
that
also
Is,
sequences 0j j,0j2,...,0j\342\200\236/2, and l51,l52,..,l^/2
leading
in
We
can
then
these
two
into
one
the
to
codes.
correspond
Gray
merge
sequences
way (see Fig. 2.4):
following
short

(we

will

see

later

how

Let j

n.

consider

!,l5,, lj\342\200\236/2,
l^\342\200\236/2-l ls2l52,052,053,...,0^/2,05
For example,

extendthe

we can

The

two

with

only one more

sequences

are

000,

Gray

code

001, 011,

for n =4 to

010, and

a Gray code for

100,

110,111.We
sequenceis 101,001,011,010,000,100,
string

will

be

bit

than

we used

for

the

Gray

code

101,

Hence,

the

log2w.

Figure

2.4

Constructing

a Gray

code from

two

smaller

ones.

as follows.

The combined

a Gray

constructed

for nil.

n =8

111, 110.

code for

length

of each

How do
Looking
sufficient

(namely, one with

exactly

two

be

for

the general

it may

to

construction

this

extend

we

even

any

value

Recall

of nl

that

the

with constructing odd Gray codes was that it was impossible to close the cycle.
it
back at Fig. 2.4, we can seethat it is not necessary to have two closed
cycles;
we
construct
If
can
an
code
two
have
to
open
Gray
open sequences.

problem

is

Induction

Mathematical

22

sufficient

names that differ

than

more

by

We now

construction.

have

one bit) of
two cases.

odd length,

then

2.11

Theorem
\342\226\241

There exist Gray codes of length \\\\og2k] for any positive integer k. The
Gray codesfor the even values of k are closed,and the Gray codes for odd
values ofk are open.

Proof: We

both

prove

cases

one

with

Induction hypothesis: There

values k < n. Ifk is even, then

induction

stronger

exist

Gray

the code

hypothesis.

codes of length
\\\\og2k\\
is closed: ifk is odd, then

all

for

code

the

is open.

The base of

the

We

is trivial.

induction

a Gray

construct

now

code of

size n.

are

There

two cases:

1.

a?

is even:

The reduction

a power

was

of 2.

By

is similar to

case

this

in

induction

the

the

reduction

there exists

hypothesis,

for the

a Gray

case where

code

of length
one with

nil (either open or closed).We can construct

two copies of this code,
Os
and
one
with
connect
them into a cycle (as in Fig. 2.4).
and
Is,
leading
leading
Also by the induction
the
number
of bits in the smaller codes is
hypothesis,
We
add
one
bit
and
double
the
number
of objects; thus, the number
of
[log2(w/2)].
bits for the new code is |\"log2(rt/2)] + 1 = [\"log2\302\253)l.
n is odd: Let n =2k + 1. Construct two Gray
them in
codes of size k, and
connect
the same way as before.
If 2k is not a power
of 2, then there are some strings of
which
have not been used as names.
One
of these strings is
length
|~log2(2/:)l,
connected to one of the strings that has been used. We can now break the cycle of
2k by adding this new
in an open path of length 2k + 1 (see
length
string,
resulting
The
number
of
bits
satisfies
the
condition.
If 2k is a power of 2, there
Fig. 2.5).
are no unused
bit to the code. The total
left, and we need to add one more
strings
number
of bits
is thus
+1.
But since 2k is a power of
2,
|~log2(2A:)1

2.

riog2(2*)l

= log2(2*),

and

+ 1)].

1 =[log2(2\302\243

log2(2*)+

\342\226\2

In this example, we had a theorem

The natural
cases.
with two distinct
do is to considereach case separately.
the best thing
However, this is not always
to do. Even though
the two cases were different,
it was easier to consider them
together
Comments

thing to
and

to include

both

case benefited from

climbing
benefits
induction

two

with

from

the

the steps

problem

in one

induction

induction

not

by

the

such that

it

covers

included

the

plan

taken

merely

hypothesis.

hypothesis

We do

feet.

hypothesis

generalized

them

of

other

concerning
steps of

foot.

a more
two

cases.

It

is

This way, the

the other case.

solution
It

of

is much

each foot separately. Each

sometimes

better

general problem.
In the next
section,
In

this

to define
example,
we present

one

like
foot
the

the
an

2.10

Figure 2.5

example where
more

with

the

that

tradeoff.

for n

statement

But,

the proof.
the

section

next

values

less than

2.10

the

The advantage to
hypothesis is stronger and can
We need to prove the induction

Paths in

a Graph

23

Gray code.

open

an extended

by solving

problem

structures.

general
induction

obvious

an

Constructing

to solve

is easier

it

Finding Edge-Disjoint

on

working

a more

problem dealing

general problem is

more effectively. There is an

for n + 1 assuming that the
for n is stronger, then
it is easier to use it in
is correct. If the statement
on the other hand, there is more to prove. We discuss
this issue further in
in the hypothesis
all
Notice also that we included
and in Section
5.10.
rather
than just 2n-2.
2\302\253,
statement

Paths in a

Edge-Disjoint

Finding

be used

Graph

connected undirected graph. Two paths in G are said to be edge

contain the same edge. Let O be the set of vertices in V with odd
they
we
We
first
claim
that the number of verticesin O is even. To prove this claim,
degrees.
number
notice that, by summing
of
all
we
twice
the
the
vertices, get exactly
degrees
up
of edges (sinceeach edge is counted twice). But, since all vertices
of even degree
of vertices of odd
number
contribute
an even number to this
there must be an even
sum,
degree.We now prove the following theorem.
Let

be a

E)

G=(V,

disjoint, if

do not

2.12

\342\226\241
Theorem

Let G =(V,
vertices
find

E) be a
odd

with

paths

edge-disjoint

The proof is

Proof:

true for m =

undirected

connected

degrees.

by

We can

vertices

connecting
induction

graph,

divide

on the

the
in

and let

vertices
each

number of

O be

in O

the

set

into pairs

pair.

edges. The

theorem

1.

Induction

graphs

hypothesis:
with

< m

edges.

The theorem

is

true

for

of

and

all connected

undirected

is clearly

Mathematical Induction

24

m edges,
and let O be the set of odd
undirected
G with
connected
graph
is
true.
then
the
theorem
take any two
is
O
If
Otherwise,
trivially
empty,
vertices.
there
is
a
them.
is
Remove
the whole
G
Since
O.
in
connected,
path connecting
vertices
We
would
induction
has
fewer
like
to
use
the
The
edges.
graph
remaining
path from G.
odd
the
rest
of
the
and
to
the
for
to find the paths
vertices,
proof.
complete
hypothesis,
the path, we may
the
have
disconnected
however, is that, by removing
The problem,
to
connected
to be very
We
have
induction
The
only
hypothesis
applied
graphs.
graph.

Considera

careful

about

this case

in

now

We

way

ingenious

The problem
it.

induction

the

using
an

graphs having

< m

with

induction

following

induction

Revised

was

encountered

we

have the

hypothesis correctly. We
change the hypothesis

\342\200\224
we will

the

can
and

avoid
adapt

requirement.

connectivity

this

in

difficulty

needs!
Let's remove

it to our

hypothesis:

hypothesis: The

theorem

is

true

for

all undirected

edges.

is simpler.
Consider
This is obviously a stronger theorem.Its proof, on the other hand,
not be
m
and
as
before.
The
with
O
undirected
an
may
edges,
graph
graph
again
connected components.
We
into several
connected. In this case, the graph is partitioned
is
Since
each
a
will
two odd vertices from the
same
take
component.
component
of
odd
vertices.
if
it
must
have
an
even
number
there
connected
Hence,
by itself,
graph
we can find two of them in the same component.
So now we are
are any odd vertices,
we can
vertices are in the same component,
basically done. Since the two chosen
connect them by a path. We then remove the path. The graph has now less than m
because
it does not require
edges, and we can use the induction
connectivity.
hypothesis
in
we
can
the
odd
vertices
We can
Thus, in the remaining
pair
paths.
graph,
edge disjoint
then add the path we removed and complete the proof.
We actually proved a stronger theorem
than
the one we sought! We proved that
the connectivity
is unnecessary. And the proof was easier.
\342\226\2
requirement

powerful technique we call strengthening

sense to the method used in the previous
section. The main trick
is to change
the hypothesis
Even though
the
to fit our needs.
theorem
becomes stronger, the proof
this
be
to
obtain.
calls
easier
may
Polya
principle
the inventor
paradox (Polya [1954]). The reason we can achieve this apparent
paradox
is that, although
we attempt
to prove more, we have more on which
to base the proof,
because the induction
is also stronger. We will see additional
hypothesis
examples of
this
method
of strengthening
the book. This method
the induction
throughout
hypothesis
This is

Comments

the

induction

is very

Arithmetic
next

geometric

an

of a very

example

It is

similar in

some

important.

2.11
The

hypothesis.

versus

Geometric

is a beautiful proof, attributed

an elegant
mean theorem. It employs

example

will use later.

Mean Theorem

to Cauchy,
of the arithmetic
nonstandard use of induction,

versus
which

we

2.11

2.13

Theorem
\342\226\241

are all positive

Ifx i, *2,..., jc\342\200\236

*-

(jr,jr2

values of n to one
principle:
If a statement
truth

values of

for larger

is true

for an

n implies

for

n-\\,

for

fact

an assumption
for smaller
the following reversed induction

extending

use

subset

infinite

truth

its

n,

the

then

we

is identical to
hypothesis
that the induction
proceeds

induction

from

case and

a base

proving

(2-3)

\302\253.
The

proof comes

of the

part

of

Instead

backward.

on

induction

by

then

-\342\226\240

The proof is

(2.3). The interesting

its

numbers,

\342\200\242\342\200\242\342\200\242*\342\200\236)\"

Proof:

25

Theorem

Mean

Geometric

versus

Arithmetic

natural numbers, and if

of the

then P

is

true

natural

all

for

numbers.
holds because

This principle

that for

guarantees

We
to

will

x\\

steps.

step, we

The claim

of n that
becomes

statement

holds for

a greaternumber
from

In the

of n that are

values

second

the

In

=2.

Considers

for

only

two

in

first all values

Consider

n.

all
\\.

induction step to go backward

need.

set we

infinite

k, there is

number

the theorem

prove

the theorem

prove

for
n=

every

the reversed

use

then

the

that

fact

the

natural

an

set

infinite

the set; we

can

k.

m to

first

m in

step,

we use

powers of 2.

The

regular
powers

use reversedinduction
to prove
are powers of 2. The theorem

induction

of 2

is

the

the theorem
is trivial

for

+*2

^[x^x~2~<

now
that (2.3) is
we can verify
by squaring both sides. Assume
easily
=2*+l.
We
rewrite
the
left-hand
side
of
and consider 2n
(2.3) as follows:

which
n =2*,

/I

JL
\"

(x\\x2\"

We
^2

can

'x2n)

x2n)1'\"-

for

theorem

the

use

'''

The

n=2

with

y{

(2.4) becomes

expression

+:v2

y\\

\"-x2n)2n

(x{x2

But, by the

induction

hypothesis

*1
y\\

and the

'''
+*2 +

2
for
+xn

a?,

we

\"

claim follows immediately.

have

xn + \\ +xn+2

+^2

=^y~2<

' ' '

+ x2n

for

(2.4)

+ \\xn+2\"'x2n)n-

=y(x\\x2'''xn)n(xn

now

(x,,+\\xn+2

JL

true

= (x{x2

'''xn)xin,

and

Induction

Mathematical

26

We are now
that (2.3) is
Assume
Z =

That

the theorem

prove

- 1.

consider

for

all

n.

Define

be

for

true

\"

in

it is

particular,

\"

especially to \"collapse\" the

chosen

so,

numbers,

n positive

any

true

is,

Ul*2-\"*i.-l*)\"
was

to

induction

and

a?,

n-\\

forjC!,Jc2,..;X\342\200\236-\\*z-

we

an arbitrary

for

The theorem is assumedto

(z

use reversed

to

ready
true

of this

side

right-hand

Z-

expression.) Hence,

have

<z,

(xxx2'-xn_xz)n

which implies that

XXX2

-'X\342\200\236-XZ<Zn,

and
l

(xXX2'\"Xn.x)n

the same

is exactly

which

2.12

n-\\

as (2.3) for

Invariants:

Loop

<Z =

- 1.

\342\226\

Number

a Decimal

Converting

to Binary
a loop

result

of

the

relationships

algorithm

in

Fig.

represented by

the

Algorithm

variables

the

array

ignoring

is

b (which

Convert_to_Binary

statement increments k, which

t mod
2, which is the reminder

otherwise). The

the

The

during

third

is an
of

statement

initially

induction
the

hypothesis

Such

the

use of

decimal number

loop
into

reflect

When
ofn

invariants

a binary

the

array

b.

with

the

number

The first
consists of one loop with three statements.
index to the array b. The second statement computes
1 if t is odd, and 0
the division of t by 2 (namely,
division (namely,
divides t by 2, using
an
integer

Binary

ConvertJo
in

the

zero).

2.14

Algorithm

is stored

the

an induction

fractions).
\342\226\241
Theorem

that the
on

induction

should

execution.

loop

illustrate

use

that

program

to prove

want

We

We can

result.

intended

a loop invariant. We
converts a
2.6, which

called

is

hypothesis

indeed

is executed.

loop

between

a certain value.

to compute

the loop is

executing

Consider a

of algorithms.

correctness

proving

is supposed

that

of times

number

for

useful

is very

Induction

contains

terminates,

the

binary

representation

Convert

Algorithm

Input:
Output:

Jo _Binary

(a positive

integer).

b (an

of bits

Converting a Decimal

Loop Invariants:

2.12

array

27

to Binary

Number

(n);
to

corresponding

the

binary

of n).

representation

begin
t

{ we use a new

n ;

:=

t to

variable

preserve

0 ;

k :=

0 do

t>

while

+ 1;

k:=k

:=tmod2

b[k]

t :=tdiv2

end
Figure 2.6 Algorithm

Proof: The

proof is by

Convert

Jo _Binary.

the loop is executed.

theorem statement. It can
In this case, the main part is the loop, and we use
of the algorithm.
apply to only a part
of the loop.
The hypothesis, in
to verify
the execution pattern
the induction hypothesis
about the variables
that is
of as an invariant.
It is a statement
this case, can be thought
The
most
difficult
the
of
the
number
of
times
we
execute
correct independent
part
loop.
of the proof is finding the right induction
Consider the following
hypothesis.
hypothesis.

The induction

Induction

be

not have to

If m is

hypothesis:

on k,

induction

does

hypothesis

the

the

number

the

same

integer

of

times

as the

represented

by the

binary array

b[\\..k],then n=t-2k+m.
The

algorithm.

number
k of
t,

The

shifted

the

the

loop

binary

by k,

To prove the

heart

states

hypothesis

of times

the loop,

when

is the

t-2k+m

expression

is executed.
array

represents

corresponds to the

the

of

that

the

loop invariant, and is also the heart

of this expression is independent

of

the

of

the

value

It captures

the

idea

the k least significant

rest of the bits.

behind
bits

the algorithm.
of

a?,

and

At

step

the value

of

(1)
algorithm, we have to prove three conditions:
at step k
of the loop, (2) the truth of the hypothesis
implies its truth for step k + 1, and (3) when the loop terminates, the hypothesis
implies
the correctness of the algorithm.
At the beginning of the loop, k = 0, m = 0 (by definition,
since the array is empty), and n=t.
2k +m at the start
of the \302\243th
Assume
that n =t \342\200\242
loop,
and consider the corresponding values at the end of the kth loop.
There
are two cases.
assume
that t is even at the start of the kih loop.
In this case, t mod 2 is 0. Thus,
First,
there is no contribution
to the array (namely, m is unchanged), / is divided by 2, and k is
is still true. Second, assume that
incremented. Hence, the hypothesis
m is odd.
In this
contributes
2k
which
b
+
to
/
is
set
to
to
is
w,
case, [k 1]
1,
(/- l)/2, and k is
changed
the
incremented.
So, at the end of the &th loop,
corresponding
expression is
=
=
+
/w+2*
is
which
+w+2*
t-2k+m=n,
(r-l)-2*
(f-l)/2-2A+I
exactly what we

the hypothesis is

true

correctness
at the

of this

beginning

Mathematical

28

need to prove. Finally,

Induction

the

loop

terminates

when

= 0,

which implies,

by

the

hypothesis,

\342\226\

thatA?=0-2*+w=w.

fall

If

and examples of common

one can easily
traps
come
from
convictions.
Many
proofs
hastily.
wrong
strong
one tends to take as evident certain seemingly
the theorem,
takes the
often
\"facts\"
by it. In induction proofs, this phenomenon
implied
one
sometimes
Since the theorem is \"evident,\"
form.
implicitly adds to the
evident \"facts.\" The proof of the step from n to n +1 uses these
several

finish

We

Errors

Common

2.13

this chapter

with

a few

warnings

into
by using induction
in
one believes strongly

trivial
following

hypothesis

is implicitly strengthened, but the stronger

Thus, the induction
hypothesis
one
never
For
are
example,
may overlook the fact that the graphs in
proven.
assumptions
and
to check the reduced graphs
for
were assumed to be connected,
the theorem
forget
Such an omission could be very subtle,
and, of course, could lead to a very
connectivity.
to state the induction
hypothesis
precisely.
wrong proof. It is important
The main step in an induction
Another common error is the following.
proof is
+
We
of
the
n
its
truth
for
n
l.
can
either
the
truth
theorem
for
that
start
implies
showing
that it follows from the n instance,
or start with the n
with
the n +1 instance and
show
are valid.
instance
and show that
it implies
the n +1 instance. Both
approaches
The proof will be wrong if
instance!
However, the n + l instance must be an arbitrary
we start with
an n instance
and extend it to an n +1 instance that has some special
properties. For example,considerthe following
proof of Theorem 2.8. We start
wrong
with an arbitrary map with
n faces,
and assume, by induction,
that V + n=E + 2. We take
an arbitrary face and add a new
new
with
two
vertices
that
cuts
the face in two.
edge
two
new
vertices
\"cuts\"
new
two
old
each
one
into
two
Adding
edges,
edges. Overall,
we added one more
three
more
two
more
vertices.
face,
But,
edges, and
V + 2 + a? + 1=\302\243,
+ 3 + 2, and
The reason this is not a
the
claim
is true for n +1 faces.
valid proof is that the addition of the edge was done in a special way. An edge can also
be added between
or between one existing
vertex
and one new vertex.
vertices,
existing
In fact,
the graphs we get by adding
new
between
vertices
have vertices only
edges only
of degree 3 or less, so they are very special indeed. In general,
with
it is safer to start
an
instance
and
to
rather
than
the
other
it
the
induction
arbitrary
try
prove
hypothesis,
using

assumptions.

way

around.

Another

dangerous trap
of the form

involves exceptionsto

the

theorem.

It is

common to have

The induction
30.\"
or \"a? is not a prime less than
exceptions
for a? =2 from the hypothesis for
principle depends on the ability to imply the hypothesis
n = l, the
hypothesis for a? =3 from the hypothesis for a? =2, and so on. If even one of
these
of this trap. The first
two examples
steps fails, the whole proof fails. We present
the second example is a more serious one.
anecdote;
example is a simple amusing
minor

Consider

the

following

n > 3,

claim.

Ridiculous claim: Given n

to each other, all linesmust

lines

in the

have

one point

plane, no
in

two

common.

of

which

are parallel

2.14

29

Summary

is obviously true
This claim is clearlywrong,
but let's look at a ''proof\" of it. The claim
more careful and consider two
the claim is still
for one line. Let's even be a little
lines;
Assume
that the claim is true
n lines, and consider n +1
for
true.
lines.
By the
n
lines
the
first
in
common.
have
a
also
the
the last
But,
by
hypothesis,
point
hypothesis,
+
in
the
have
a
common.
The
common
n lines
(n l)th line)
point
(including
point of the
n
the
n
lines
lines and
last
must be common to all n + 1 lines, because lines having
first
in common are equal. But,
in that case, the (n + l)th
line passes
two
points
through the
same point, and the claim is proven.
is wrong
with this proof? Actually
What
The only wrong
little.
is that
very
step
the
fact
n
the proof unintentionally
that
must
(or in this case very intentionally)
ignores
to work.
That is, the claim is true for n - 1, n = 2, and also,
be at least 3 for the argument
if

true for

it is

from

step

a?

a?

to

=2

claim

in this

miss.

Let's look

/? =

(The

\\

+ (/?

the

that

another

example

following

claim:

- 1) \\

goes to

expression

show

at

the

Consider

expression

this part (it is

correct).

expression

n+

1. If we squareboth
=

+ n

Rearranging

that is

Vl +

(a?

+2)

proof of

is a

converges
If n = 1, then (2.5)

a?,

that

so

becomes

converges). Assume that

of (2.5) we get
+(/i +

+Wl

+(/i-l)\\l

\342\200\242
\342\200\242
\342\200\242.
The

5,

is

problem

only

the whole proof, and

example is too obvious

the

the
to

so obvious

not

for all

sides

+ 1 = 4,

+ 1)VT+

(a?

Here

infinity.)

in parenthesis

the

n2

wrong.

very

true for

it is

small exception is enough

to make
The reader may think that this

This

=3.

a?

case,

\342\200\242
\342\200\242
then
\342\200\242,

4,

=3,

\342\200\242
\342\200\242
\342\200\242.

(2.5) by

induction.

the claim is
1 = \\1

+0(

for

omit

We

meaningful.

\342\226\240
\342\200\242
which
\342\200\242),

is correct

(2.5)

we have to

First,

is true
a?,

(since

consider

and

+ (/?+2)---.

l)Vl

terms, we get

-5-^i = n
A7-1

+ 1 =

\\1 + n

Vl + (a?

+ 1 Wl

+2)

(a?

^,

is exactly
Or is it? The only
(2.5) for n + 1. The proof is now complete.
wrong
was
n
1
this
that
is
not
0. But, n - 1 =0
when
without
value
step
dividing by
verifying
n = 1, which
is the first step in the induction!
works
one
for
except
Again, everything
which

\342\200\224
the one

implication

whole

In this

proof.

2.14

case, by

the

way,

= 1 to a? =2

\342\200\224
and

the claim is

this is

correct, but

the

enough to invalidate
proof is not that

the
easy.

Summary

Mathematical induction
and explored
hypothesis.

that goes from

is a

some of

the

We have to

cases, there is

only

one

rich technique. We
for using it.

have seen many

methods

decide to

parameter,

which

and the

parameter

The

variations

step is to define
we apply the induction.

first

choice is clear.

In

other

cases,

of induction,
the

induction
In

many

however,

we

30

MathematicalInduction

have

a fair

introduced

always

amount of flexibility.
especially for the proof.
the

from

directly

follow

steps, each leading

defined
one,
may be even a newly
not
does
seen, the induction
hypothesis
in
theorem statement. Sometimes,we apply
induction
The

parameter
As we have

to the

proof. At other times, we strengthen the

theorem.
implies
hypothesis
There are two steps in every induction proof: the base case and the reduction
step.
to
but not always, easy. Because it is easy,
there is a tendency
The base case is usually,
to
is the heart of the induction
There
are
reduction
The
it.
proof.
ways
step
many
ignore
n to the
common
achieve the reduction. The most
way is to reduce a claim
involving
It is also common to \"go\"
from
n + \\ to n. A strong
n-\\.
same claim involving
smaller
values
a claim involving n to one or several claims involving
reduces
induction
variations
include
Other
2n
from
to
not
a?, and
than n (but
n-\\).
necessarily
going
n
is
+
the
for
from
a
claim
n
1
claim
for
and a base
induction, in which
reversed
implied
set is proved. The key
to any reduction
is that it must
case consisting of an infinite
of the claim.
No additional
can be made about
assumptions
preserve the exact statement
included in the induction
the reduced
claim, unless they are specifically
hypothesis.
as
an
extension
We
extend the claim
can
also
be
The reduction
step.
step
regarded
to a larger value. We
have
to ensure that the
from a smaller value of the parameter
claim is a
values of the parameter, and that the extended
extension ''covers\" all possible
or constraints. In
assumptions
general claim of the theorem without any additional
see that there is a direct analogy
between
the variations
of induction
5, we will
Chapter
several

such that

introduced

in this

chapter

and

Notes
of

discovery

closer

a stronger

it

Bibliographic
The

us

the

several

algorithm

design techniques.

and Further

mathematical

induction

Reading

principle

is

attributed

to

the

Italian

induction
to note that a

Maurolycus (b. 1494).

history
(see also Vacca [1909]). It is interesting
was used in the 12th century in
induction
principle
very similar to mathematical
to the Talmud (this
observation
is due to J. Gillis). The
was to
interpretation
problem
At the time of the
interpret a rule that specifies a date as \"3 days beforea holiday.\"
of the Talmud, it was not uncommon,
when one said \"x days before a holiday,\"
writing
to include the holiday
itself as part of the x days. The question was whether
or not the
be included
as part of the 3 days specified in the rule. The interpretation
holiday should
was that the 3 days do not include
lead
to
because doing so would
the holiday
An
inductive
The
base
case
that
conclusion.
was
used
arrive
at
to
ambiguities.
argument
was 1 day. It makes no sense to say \"1 day before a holiday\" when we mean the
itself.
does
not include the holiday. Now,
Therefore, \"1 day before a holiday\"
holiday
\"2 days before a holiday\"
must
also exclude the holiday,
it will have
otherwise
because
the same meaning as \"1 day
a holiday.\"
before
Therefore, \"3 days before a holiday\"
does
not include the holiday. This is clearlyan inductive
argument.
in Section
The summation problem given
2.5 is from Polya [1957]. A brilliant
discussion
on the generalization of Euler's formula
to three-dimensional
objects is given
It is warmly
Lakatos
recommended.
The example
2.8 is from
in
Section
[1976].
by
Franciscus

mathematician

is described

in

Bussey

The

[1917]

of mathematical

31

Exercises

Lovasz

[1979].

found

be

introduced

(see,

Cauchy

More on coding theory

[1953].

Gray

by

proof of the arithmetic

example, Polya and Szego

The

[1986].

Hamming

theorem is due to

for

versus
[1972]

invariants

can

geometric
or Beckenbach

A bibliography for graph

is given in Chapter
[1961]).
theory
can be found in Gries [1981]. The example
of the proof of

Bellman

us by

codes were

Gray

in

7.

More on

mean
and

loop

(2.5) was shown

to

Darrah Chavey.
induction can

mathematical

on

material

Further

books [1954; 1957; 1981]. Additional

Golovina and Yaglom [1963],and, of

course,

be

found

be found

can

examples

in

Polya's

in

Sominskii

wonderful
[1963],

this book.

throughout

Exercises

2.1
2.2

\342\200\242
**

a, +tf2+
2.3

Find

all natural

to

general

prove

numbers jc, y

(x *y),

sums.

arithmetic

and n.

That is,

sum

the

find

are constants.

andc,,c2

c1/i+c,2,

sum and

Find the following

2.2

for

=
#\342\200\236

where

+an,

1-2 + 2-3+
2.4

in Section

solution

the

Extend

is divisible byx-y

x\" -y\"

that

Prove

your claim:

\342\200\242\342\200\242\342\200\242
+

l).

+/i(rt

sum and prove your

the following

claim:

111

1
\342\200\224
+ \342\200\242\342\200\242\342\200\242
+ \342\200\224
+ \342\200\224
+ \342\200\224

2
2.5

Find the

2.6

Prove

sum of the

of the

squares

+ 32 -42

a set of n

Given

are two

2.8

2\"

first

numbers and prove your

n natural

claim.

that

l2 -22

2.7

\342\200\242
\342\200\242
\342\200\242
+

(-I)*\"1*2

numbers

numbers in

Let a, b, and

n be

the set,

= (-l)*-'*(*

out of the
one of which

first

In

+ l)/2.
numbers l,2,...,2/i, prove

natural

that

there

the other.

divides

positive integers. Prove that

V-\\an+bn)>{a+b)n.

2.9

2.10

Prove

by

induction

only

if the

Find

an

sum of

representation,

is divisible

by 3 if

and

sum of the ith row of the following triangle, which is called the
The sides of the triangle
are Is,
prove the correctness of your claim.
is the sum of the two entries directly above it.

for the

expression

Pascal triangle,
and each other

that a number, given in its decimal

is divisible by 3.
digits

its

and
entry

1
1
1

13
14

32

2.11

Induction

Mathematical

an

for the

correctness

expression
of your

above it

(a nonexisting

Find

claim.

sum of

Each

/th

the

in the

entry

row of the following triangle, and prove the

is the sum of the three entries directly
triangle

entry is considered 0).

1
1

12

2.12

for all

that,

Prove

'

_L_

2.13 Prove that,

1

+...+_L

for all n

i +

can

k_

2.15 Considerthe

first

as a sum

be written

10

16

19 16

10 4

'

is an

even number.

1, 2, 3, 4, 5, 10,20, 40, ..., which

becomes a geometric series. Prove
from this series.
numbers

series,

following

but after the

series,

13

odd number and

2.14 Considerthe

In

3
k is an

where

> 1,

i+...+i..

> 1,

n+2

n+\\

5 terms

of distinct

starts as
that

an

arithmetic

any positive

integer

starts as an arithmetic
series, 1, 2, 3, 6, 12,24, 54, 84, 114,..., which
terms it becomes a geometric series, and then, after 3 more terms, it
If it
becomes an arithmetic series again.
Does your proof of Exercise 2.14 fit this problem?
be written
as a sum of distinct
does, find the error in it since, for example, 81 cannot
numbers
this series. What
is the subtle point in the proof of Exercise 2.14?

series, after

2.16 Considern

following

first 3

the

lines in

>3

general

position

plane. Prove

in the

that

one of

at least

the

regions

they form is a triangle.

2.17

Considern
n

2.18

lines

>3

- 2
triangles.
a set of n

Given

cycle, prove that

2.19

that

2.20

any

Prove
can

2.21

that

Prove

position

points in the plane such

all n points are contained

the regions

neighboring

in the

that

plane.

Prove

these

three of them are

unit-size cycle.

any

in a

that

lines

contained

formed by n circles in the plane can be coloredwith

regions are coloreddifferently.

two

form

at least

in a

unit-size

colors

such

that the regions formed

one chord (see Fig. 2.7),
by n circles in the plane, each with
with three colorssuch that any neighboring regions are coloreddifferently.

be colored

Prove that the

colored

2.22

in general

regions formed

with two colors such

by a
that

planar map

no two

all

of whose

neighboring regions

vertices have even degreecan

have the same color.

be

that a planar map can be coloredwith

three colors, such that every
two neighboring
regions are coloredwith different colors, if and only if each region has an even number of
are considered
neighboring regions. Two regions
neighbors if they have an edge in

Prove

common.

33

Exercises

Figure 2.7
*

Circleswith

one

chord.

in the plane are the points

with integer coordinates. Let P be a polygon
2.23 The lattice points
a polygon is called simple) such that all of its vertices are
that does not cross itself
(such
lattice
of lattice points that are on the boundary of
(see Fig. 2.8). Let p be the number
points
the polygon
its vertices), and let q be the number of lattice
that are inside
points
(including
the polygon.
is p/2 + q - 1.
Prove that the area of the polygon

2.24

can define anti-Gray codes in

between two consecutive strings,

We

encoding for

any

even

value

Figure

2.8

the
we

following
can try

of objects

such

A simple polygon

way. Instead of

to

that

maximize

each

on the

it.

minimizing

Is it

possible to design
strings differ

two consecutive

lattice points.

the difference

an
by

34

Mathematical Induction

etc.)?If
2.25

is k is

(where

bits

it is

possible,

there is at

prove that

common,

n >2,
2.26 Let dx, d2,...,d\342\200\236,

2.27

there exists a

segment. Assume
regions formed by

2.28 Let

T = {V,

tree with

undirected

an

the following

2.29 The pigeonhole

principle

are

n boxes,

inside

principle

meet

inside the

circle,

tree. Let /be a

condition: If (v, w) is

in

simplest variation)
one box will

that

function

dn.

vertices to vertices,

maps

either (f (v),/(w)) is
either
maps an edge to an edge, or
exists either a vertex v in V such that

an edge

in \302\243,
then

(in

other words,

there is

the following: If n+\\ balls (n > 1)

more than one ball.
Prove
this

states

at least

d2,...,

dx,

contain

A complete binary
tree of height
h
+
1 consists
tree
of
of
complete
binary
height
two complete binary
trees of height h whose
roots are connected to a new root. Let T be a
tree of height h. The height
of a node in T is h minus the node's distance
complete
binary
from the root (e.g., the root has height /?, whereas
a leaf has height
0). Prove that the sum of
the heights
of all the nodes in T is 2/,+l -h-2.

F(/i) = F(/i-1)

and

m be

n2 -m(n

(Hint: Use a
either

forward

number,

the induction

Strengthen

integers such

\\)

+ 2n

\"two

that

+ m2 <

n2

or

in the

\\<m<n.

defined

is

which
for

n>2.

Prove

hypothesis by
section on Gray
Prove

by

inductively as follows:
that F(n)2 + F(/i + l)2

proving

induction

two seemingly

separate

codes.)

that

+ n.

on m. Prove two
backward from m = n.)

induction

sided\"

from m =

+F(/i-2),

same time, as is done

at the

theorems

Let

(Hint:

as follows.

inductively

is the root.

Fibonacci

nth

the

= F(2)=1.

+ 1).

is defined

which

node

be

Let F(n)
F(2n

2.33

(in its
then

tree

binary

of

consists

F(1)

2.32

3,

induction.

by

A complete

2.31

k-

each point to all the others by a line

at a point. Calculate the number
of
and prove your claim.

connect

(v) =/ (w). In other words, the function

or/
edge in \302\243
it contracts an edge to a single vertex. Prove that there
= v, or an edge (v, w) in E such that /(v) = w and f(w)
/(v)
either
a vertex or a edge that the function maps to itself).

2.30

(or k -2,

bits

if

whose degrees are exactly

n vertices

an

put

each pair of subtrees has at least one vertex

in common
to all the subtrees.

that

integers. Prove that,

line segments

these

E) be

satisfies

which

one vertex

on the boundary of a circle, and

that no three line
segments

n points

Put

How about

string)?

construction.

T such

least

be positive

each

'\342\200\242\342\200\242
+d\342\200\236=2n-2,

di+d2+
then

in

an efficient

find

T and k subtrees of

a tree

Given

of bits

number

the

base

cases,

m =

and

m =/i, and

go

Let
graph is an edge whose removal disconnects the graph.
be a connected
undirected
G has the
graph without a bridge. Prove that
\"ear decomposition\" (see Fig. 2.9). The edges of G can be partitioned
into
such that \302\243,
sets
is a cycle, and, for each
is a path
...,\302\243*\302\273
z, 1 <i<k,
disjoint
\302\243|,\302\2432,
\302\243,
whose endpoints are vertices that already
appear in a previous \302\243,,
j < /, and its other
vertices (if any) have not appeared in previous
path may be a closed one, in
\302\243,s.(The
which
case it includes only one previous vertex.)
A

bridge

G=(V,E)
following

in an undirected

Exercises

denote
2.34 Let K\342\200\236

the complete

connected), and let n be an

trees.
exactly nil spanning
vertices
and no cycles.)

2.35 Given

common.

a graph

In

with

decomposition.

undirected graph

n vertices (namely, every two vertices are

the edges of Kn can be partitioned into
is a connected
that contains all
subgraph

with

number.

Prove

(A spanning

tree

even

graph G = (V, F), a

A perfect matching

an undirected

vertex in

An ear

2.9

Figure

that

set of edges no two of which have a

Construct
all vertices are matched.
G has exactly one unique
perfect

is a

matching

is one in
and n2 edges such

vertices

35

which
that

matching.

2.36

Let a,,
the

a2,..., an

real numbers such

inequality,

geometric

Try a reduction by introducing

from the sequence.)

(Hint:
numbers

that

- \342\200\242
= \\. Prove,
a\342\200\236

a2

without

using

that

\342\200\242\342\200\242\342\200\242
(l+a\342\200\236)>2\\

(l+fli)(l+fl2)

2.37

be positive

versus

arithmetic

another

variable

that

two specially

replaces

chosen

numbers F (/i) = F (/i - 1)+ F (/i - 2). Without

recurrence
the
to G(n) defined
F(n)
by
solving
compare
G (n) = G (n - 1) + G (n - 2) + 1. It seems obvious that G(n)>F(n) (becauseof the extra 1).
We
Yet the following is a seemingly
valid
proof (by induction) that G{n) = F(n)-\\.
=
and we consider
that G(k)
assume, by induction,
F(k)-\\ for all k such that 1 <k<ny

Considerthe

recurrence

this

for Fibonacci

relation

recurrence,

G(/i + l):
G(/i
What

= G(/i) + G(/i-1)+1

+ l)

is wrong

= F(/i)-1+F(/i-1)-1

+1

= F(/i

+ l)-l.

with this proof?

2.38 The following

is another
versus
The
mean inequality.
proof of the arithmetic
geometric
which
makes it incomplete in general.
Describe this weakness
proof has a major weakness,
and
then define the restrictions on the theorem that are needed to make this proof correct.

Let S =xt
numbers

\342\200\242
\342\200\242 To find
+x\342\200\236.
+x2+ \342\200\242

whose sum is

5 and

whose

a contradiction
geometric

to

mean is

the
larger

theorem,
than

we need to exhibit
n
Sin. It makes sense to

36

Induction

Mathematical

is maximum over all such sets.

numbers whose sum is S and whose product
fix the sum (5) and try to maximize the product. Let {jt|v*2,...,*\342\200\236
I be a
the product, and whose sum is S. If jc, *jc2, then we can replace both jc,
set that maximizes
the same, but the product grows,
and x2 with their average (jc, +jt2)/2. The sum remains

look for

a set of

In other

words, we

because
I

Xi +X>>

X\\X2<

with equality holding only if jc, =x2. If all the numbers are
this is a contradiction to the maximality
assumption

Otherwise,

2.39

2.40

equal,

then

of the

the theorem

to convert an binary number to a decimal number.

an algorithm
The
Design
of algorithm Convert to_Binary
should be the opposite
(see Fig. 2.6). The input
n. Prove the correctness of your
of bits b of length
k, and the output is a number
invariant.
a
loop
by using
Modify algorithm
base 6 to a binary
of bits.

Prove the

ConvertJo _Binary
number.
correctness

holds.

set.
algorithm
is an array
algorithm

a number given in
(see Fig. 2.6) such that it converts
The input is an array of base-6 digits, and the output is an array
of your algorithm by using a loop invariant.

CHAPTER

ANALYSIS

OF

It

ALGORITHMS

not depend

does

on size,or a cow
Proverb

German

Pennsylvania

rabbit.

catch a

would

He is a foolwho looks at the fruit of lofty trees,

but does not measure
their height.

Introduction

3.1

The purpose
of algorithm
of an algorithm
without
It
so
are
clear.
doing

it is
are too

characteristics of the
most

the

implementation

On

perfect.
algorithms

the

the

behavior,

computer

underlying

factors.

influencing

attach

predict the
Instead, we

every
a

Furthermore,

algorithms.

for each

of
the

It would

part of the program.

exact behavior of
try

to

extract

be
an

the main

certain parameters and certain measuresthat

exact
details
concerning the
analysis.
Many

We define

algorithm.

Most
algorithm.
to determine
the

the efficiency

system changes.

alternatives

to

test

and

''small\"

different

impossible

usually

many

important

and

time,

the running

especially

the algorithm

implement

for

the

are ignored. The analysis is thus

other
hand, even a rough
approximation

mileage claims,
running

the

work

Unfortunately,
There
algorithm.

about

to

than
in

program usually includes many

to test thoroughly all different

complicated

the

The
it on a specific computer.
advantages
implementing
more convenient to have simple measures for
is much

of an algorithm
time a certain parameter

too much

is to predict

analysis

efficiency

are

Rufus

Curtius

Quintus

important,

best one for

a disclaimer

using
our

only

an

approximation',

can yield

analysis, we
We can
purposes.
this

significant

it

not

is

information

can compare different

use an analogy
to

saying \"Use for comparison only

car

\342\200\224

your

times may vary.\"

ti

38

of

Analysis

chapter, we describe one methodology

and for comparing different
of algorithms

In this
times

running
this

algorithm as the

goes to

algorithm consists of

steps,

predicting
The

algorithms.

on

concentrate

For

infinity.

\\00n

for

and

factors

constant

of the input

size

if the

and
\302\253,

we ignore

is that

approach

size

Algorithms

example,

the approximate
main feature of

we ignore the
will introduce precise
then

of the

behavior

the

is

if the input

an

array

of

100 and

constant

time is approximately n (we

notation
shortly).
say
2 and 50 and say that the
then we ignore the constants
If the number of steps is 2n2 +50,
n2 is larger than
we say that the second
n2.
Since
\302\253,
running time is approximately
for
the
first
even
for
is
\302\253=5,
slower,
example,
though
algorithm requires 500
algorithm
100
This
second
one
the
whereas
steps.
requires only
approximation is valid,
steps,
the

that

however,
for all n

running

n is

if

>50. On the

100\302\2531,8.

however,

than

2\302\2532+50.

their

will

times.

running

most

well

works

The result

even

Thus,

is sufficient

behavior

time

slower

than the

though

in practice.

as a first

approximation

In

the

approach

asymptotic

in

the

cases,

be smaller

to

expression

can be

looking at only
and indication of efficiency.

most

one

algorithm was
than n2. In this

have small constants

algorithms

first

of the first

other
hand, suppose
running
first algorithm seems better, since nx% is smaller
have
to be approximately 300,000,000 for 100\302\25318

Fortunately,

sometimes,it

is indeed

that the

the

Again,

case,

The secondalgorithm

large enough.

the

of

misleading
asymptotic

should indicate how long the algorithm in question is

input. However, we cannot list the precise running times
for all inputs,
the algorithm
unless
is very simple.
The number of different
possibilities
of inputs is enormous, and
behave
for
different
most
algorithms
differently
inputs.
we attach a measureto the input, called the size of the input, and present analysis
Instead,
relative to that size. The algorithm will not behave exactly the same for all inputs
of
but
we
defined
equal size,
hope that the variation will be reasonable. The size is usually
as a measure of the amount
of space required to store the
We will not try to
input.
introduce
one general definition
for all algorithms,
of size of the input
because we will be
interested in comparing
for
the
In most cases,
different
same
mainly
problem.
algorithms

expected to run

the

for

of

definition

our

of

analysis

a particular

size

will

Unless specifiedotherwise,
Given
the running

time\"
for all

a problem
time of the
be

will

given

be
the

We
straightforward.
size will be denoted by

will

see

some

examples

shortly.

n.

of size, we want
to find an expression that
gives
'
of
relative
to the size. (The precisedefinition
algorithm
'running
in Section 3.3.) As we said earlier, there is usually
not just one value
and

a definition

of the same
all inputs
Consequently, we must choose, among
to use as our indicator.
The most common choice is the worstcase input. This may seem peculiar.
or the average input?
not use the best input,
Why
The best input is usually ruled out because, in most cases, it is not representative;
there is usually an input
for which the problem is trivial.
The
average-case
input may be
a goodchoice,but it is sometimes very hard to measure effectively. First, it is generally
not clear what
an \"average\"
over
input is. We can average
many different parameters in
not
If
we
are
different
the
cases
contain many
that
can
careful,
many
ways.
average
never occur in practice,
thus making this measure irrelevant.
serious
Another
problem
case is the mathematical
with
the average
in analyzing average-case
taking
difficulty
performance. We are still very far from having
easy-to-use
relatively
comprehensive,

size, the

inputs

input

of equal

size.

we want

3.2 The

O Notation

39

We
will
discuss
average-case
analysis for few
resort
worst-case
to
the
worst input as
mainly
analysis. Choosing
problems,
indicator
turns out to be very
useful.
In some cases, the worst
is
an
input
very close to
and
to
observations.
In
other
even
the
cases,
average
input
experimental
though the
is
from
the
different
the
that
worst
achieves
input
substantially
average input,
algorithm
the best performancefor the worst input also performs very well for all cases. Unless
we will use worst-case analysis throughout
this book.
specifiedotherwise,
In summary, both
and worst-case
asymptotic
analysis
analysis are only

for average-case

techniques

do not

definitely

They

running time

of the

approximations

analysis.

will

we

but

give

the

of a

whole

story.

particular

a particular

under

algorithm

They are, however,

good

very

input.

indicators

in

most cases.

The O Notation

3.2

to
said, our approach will be to ignore constant factors when
trying
time
of a particular algorithm.
To do that effectively
we need
We
is 0(f(n))
for another
function
g(n)
say that a function
f(n)
special notation.
if
or
sometimes
there
exist
constants
c and /V,
\"Oh,\"
Oh,\"
\"Big
of/(\302\253)),
(pronounced
In other
for large enough
such
words,
a?, the
that, for all n >N, we have g(n)<cf(n).
is
more
than
a
constant
times
the
The
function
no
function
g (n) may
g (n)
function/ (n).
it only from above. For
be less than cf (\302\253),
even
bounds
less; the O notation
substantially
= 0{n2),
At
since 5n2 + \\5<6n2 for
n>4.
the same time,
+15
example, 5\302\2532
5n2 + 15= 0(\302\2533), since 5n2 + 15<n3 for all n >6.
we
can
The O notation allows us to ignore constants conveniently. Although
We always
write
include constants within
the O notation,
there is no reason to do that.
we write
without specifying the
0(n) instead of, say, 0(5n+4). Similarly,
0(\\ogn)
As

have

we

already

the

evaluate

running

the logarithm only by a constant.

bases
changes
logarithm, because changing
We write
if we want
to
to denote a constant. We can also use the O notation
0{\\)
write
the constants only
in parts
of the expression. For example, we may
specify
+5n
+0 (\\).
T(n) = 3n2 + 0 (n),orS (n)= 2n\\og2n
not
be
In general, determining
whether
a certain function g(n) is 0(f(n)) may
in
are
in
of
this
book
Most
of
the
the
functions
involved
easy.
analysis
algorithms
base

of the

With

simple.

relatively

The

cases.

most

monotonically growing

functions

rule
if

the
simple rules, we can cover
We say
is the following:

n j >n2

that/(a?

implies

of (but

majority
that

function

not

f(n)

all)

is

,)>/(\302\2532)-

3.1

\342\226\241
Theorem

For all

some

useful

constants c > 0

and

>

1, and

for all

growing

monotonically

f(n),

(f(n)Y=0(a^n)).
In

other

polynomial

words,
function.

an

exponential

function

grows

faster

than

does a

\342\226\2

40

of Algorithms

Analysis

to compare many
functions.
for
all
constants
that,
get

be used

This rule can

/ (n) in Theorem

3.1, we

For

example,
a > 1,

we

if

n for

substitute

c > 0 and

(3.1)

n(=0(a\.")
comes

example

Another

from substituting

For all constants

for f(n).
log\342\200\236A?

c>0

and

a>\\

= 0(n).

(\\ogany=0(aloSl\")
We

add and multiply

can

2) If fin)
Proof:
largest
Since

= 0(s(n))andg(n)= 0(r(n))then
= 0(s(n))
and g(n) = 0(r(n)) then

By

of N,
the

N2

O notation
divide.

or

subtract

and

can

are

there

definition,

for a?>/V,,

f(n)<C\\s(n)

3.1,

importance

which contains

algorithms

using

the

rules.

following

consume

constants

c\\,

and g(n)<c2r(n) for n>N2.

to show both claims.

corresponds to
That

of

+ g(n) = 0(s(n) +
= 0(s(n)r(n)).
fin)g(n)

fin)

N\\,
The

c2,
largest

r(n)).
such

and

N2,

of c,

and c 2, and the

that

be used

is,

it

is

that
g(n) = 0(r(n)) imply
f(n)-g(n)
I
Exercises
3.15and
0(s(n)
r{n)) (see

The

O notation

the

with

3.2

Lemma
\342\226\241

/. If fin)

(3.2)

concentrating

several

typical

for a problem

the

\342\226\2

to

\"<\"

not

it is not
however,
relation,
possible
that
true in general
f(n) = 0(s(n))
= 0(s(n)-r(n))
or that f(n)/g(n)

and
=

3.16).

the asymptotic
behavior is
times
and the time
running
on

of size n

= 1000

for different

illustrated
the

in Table

corresponding

computer speeds. The

differ by a constant
1000
of 2 from column to column, from
steps per second to
8000 stepsper second.We can clearly see the improvements we gain by speeding up the
we gain by
computer (or the algorithm)
by a constant factor versus the improvements
to
a
faster
An
the
changing
table).
(i.e., going up
exponential
asymptotic
algorithm
speeds

of years) to handle
astronomical
time (billions and
billions
the
base is very close to 1).
(unless
The O notation
is used to denote upper bounds
on the running times of algorithms;
All the algorithms in this book, for
however,
using only upper bounds is not sufficient.
more than
not require
times
of 0(2\.")
That is, they
do
example, have running
time.
of these
for
most
bound
crude
exponential
However,
0(2\") is a very
upper
\342\200\224
are
in
not
much
interested
faster
than
that.
We
are
algorithms
they
upper bounds,
only
but
also
in an expression that
as
time
is as close to the
actual
possible. In cases
running
where it is too difficult
to find the exact expression, we would
like to find at least upper
and
lower
bounds
for it. Obtaining lower bounds
is more difficult than is obtaining
upper
bounds.
An upper bound
time of an algorithm
on the running
only that there
implies
exists some algorithm
that does not use more time
A lower bound
must
than
indicated.
a
better
achieve
that
no
can
bound
It is impossible, of
for the problem.
imply
algorithm
to
consider
all
one
one.
We need mechanisms to model
course,
possible algorithms
by
that
a
in
enables
and
us
to
bounds
way
problems algorithms
prove lowerbounds. Lower
6.4.6.
There is a similar
are discussed further in Section
to handle lower bounds
notation

algorithm
n

= 1000

will

require

3.2 The O Notation

time |
2000

10
32

log2\302\253

\342\200\236..5

1,000,000

n3

1.1\"

Table 3.1

0.005

0.003

0.001

0.5

0.25

0.125

(in seconds)

1.25

2.5
8
250
250,000

1039

times

Running

8000 steps/sec

16
500
500,000

1039

time

steps/sec

4000

1,000

n2

time

steps/sec

0.010
1

1 log2\302\253

time

1000steps/sec

times

running

41

125

125,000

1038

1038

under

different

(n=1000).

assumptions

c and N, such that

for all n>N
the
ignoring constants. If there exist constants
of steps T(n) required to solve the problem
for input size n is at least eg (\302\253),
then
we say that T(n) = \302\243l(g(n)). So, for example,
n2=Cl(n2- 100), and also n=Q(n\302\260'9).
\">\"
The Q notation thus correspond to the
relation.
= \302\243l(g(n)), then
If a certain function
satisfies
both f(n) = 0(g(n)) and f(n)
f(n)
we say that f(n) = &(g(n)). For example,
5a? log2fl - 1O= 0(a? log a?). (The base of the
can be omitted in the expression
bases
the
&(n log/;), since different
logarithm
change

while

number

logarithm
part need

by

only

The 0,
need

notation

a constant

factor.) The constants

used to

the O

prove

part

and

the

the same.

be

not

Q,

and

corresponding

(loosely) to \"<\", \">\", and

correspond
to \"<\" and \">\". We say that/(\302\253)

we

Sometimes

\"=\".
=

(pronounced

o(g(n))

tt/(A7)islittleohofg(A?),,)if

= o.

lim^
For

g(n)

but n/\\0*o(n).

n/\\og2n=o(n),

example,

Similarly, we say

f (n)

= (o(g(n))

if

= o(f(n)).

We can

Theorem

strengthen

3.1 by

replacing big O with

o:

little

3.3

\342\226\241
Theorem

For all constants c >0 and a > 1, and for all

f(n), we have (f (n))c =o(aJ{n)).In other
than does a polynomial
function
grows
faster
function.
functions

The

that

growing

monotonically

words,

an exponential

\342\226\2

OO Symbol

The O notation

has

of course,

in

convenient

notation

that

received
reality

to forget

gives

only

a lot

constants

of criticism over the years. The main objection to it is,

do matter. The wide use of the O notation makes it

about constants altogether. It

As such,
approximation.

a first

is essential
it

serves

to remember

a useful

purpose,

that
and

the
its

O
use

Analysis of

42

has

was also

of complexity

development

efficiency.
to distinguish
important

however,

is,

are

notation

by

It

measures.

all

light on

sheds

which

theory,

book a

in this

large and

is efficient

new

in

It is

symbol.

where

case

the

not meant to be

constants

the

they are

this

make

To

practice.

where

case

the

between

prohibitively

algorithm

corresponding

introduce

are practical

that

algorithms

many

of algorithm

by the

ignored
the

the

in

instrumental

aspects
It

of

the development

prompted

many

Algorithms

small and
we

distinction,

a precisemathematically

(or at least
only to replace some prose that accompanies
whose
as
measured
times,
should
running
by the O
algorithms
accompany)
value
We
to
denote
theoretical
of
are
notation,
only.
suggest
by CO(f(n)) (pronounced
that is O (/(\302\253)), but with constants that are too large for
\"Oh Oh of/(\302\253)\") a function
should
be easy to remember since it resembles
a big
uses. (This notation
most
practical

meant

\342\200\224
it is

notation

defined

some

oo.)

The use of
not a certain
of

intention

the

reader

to tighten our

of

symbol is to stressthat

Time

3.3
How

do

need to

we
count

and Space

example, a

The

not

the

whole

of

main

the

Another

We have no

is to indicate to

purpose
goal

or

Whether

writer.

defined.

well

not

the

this

in introducing

story.

Complexity

analyze an algorithm's
the number of steps the

many different types of

judgment

use\" is

concise form.

is

O notation

to the

definition.

in a

writer

the

the

left

a \"practical

to

leads

attempting

opinion

should be

CD notation

the

constant

steps, and

time

running

algorithm
each

may

without

performs.

require

running the algorithm?

The problem is that there

We

of time.

For

amount

different

are

to compute than an addition

does. One way
to
an algorithm is to count
the number
of different steps separately. But listing
all
analyze
the types of steps separately
will
the
be, in most cases, too cumbersome.
Furthermore,
of the different
on
the
or
the
implementation
steps
depends
specific
computer
in
used
the
We
to
that
are
avoid
programming language
implementation.
trying
division

may

take

longer

dependency.

Instead of counting
all steps,
to us to
we focus on the one type of step that seems
be the major step. For example, if we are analyzing a sorting
then we choose
algorithm,
of sorting;
comparisons as the major step. Intuitively, comparing elements is the essence
all the rest can be regarded
that
sure
as overhead.
Of course, we have to make
indeed constitute
comparisons
constant
factors anyway, it

the
suffices

major
part of
to check that

the

algorithm.

the

number

Since

we

will

ignore

operations is
is a bound
for
and if 0(f(n))
for the total number of steps.
of

all other

to the number of comparisons. If this

is true,
of comparisons, then 0(f(n))
is also a bound
We
that the time complexity of the algorithm,
or the running time, is 0(f(n)).
say
This
the
of
also
solves
different
that
different
problem
approach
steps
require
computation time, as long as the difference is no more than a constant.
The space complexity of an algorithm
indicates
the amount of temporary
storage
In most cases, we do not count
for running the algorithm.
the storage required
required
as part of the
for the input or for the output
space complexity. This is so, because the
proportional

the

number

3.4 Summations

spacecomplexity

43

same problem, in which

do without
the input
or
to
we
want
count
the
and
that
be
saved.
We
also
do
not
count
only
storage
output,
may
for the program itself, since it is independent
of the size of the input.
the
required
storage
refers
to worst
denoted as
Like time complexity, space complexity
case, and it is usually
in the size of the input.
an 0(\302\253)-space
an
Thus,
expression
asymptotic
algorithm
of memory
a constant amount
per input primitive. An 0(l)-space
requires
algorithm
amount
of space independent of the size of the input.
requires a constant
the number
of major steps may
not
be easy.
In the next
sections
we
Counting
several
for
times.
mathematical
In
discuss
briefly
techniques
running
computing
the
of
a
is
contrast, estimating
complexity
particular
space
algorithm
usually
to compare

is used

different

straightforward, and,

in

most

cases,

we

will

for the

algorithms

requirements are fixed.

the input/output

case

not

cannot

we

Also,

it.

discuss

Summations

3.4

is composed of several parts,

then
its complexity
is the sum of the
it
In
this
is
as
as
The algorithm
of
its
not
sounds.
simple
complexities
parts.
many cases,
a different
We
many times, each time with
complexity.
may consist of a loop executed
in order to analyze such cases. Probably the
need techniques for summing
expressions
that the /th step (i<n)
such
case is a loop of size \302\253,
requires / operations. The
simplest
If

an

algorithm

of

number

total

is

operations

\342\200\242
\342\200\242
\342\200\242
+\302\253. We

1+2+

thus

denote

sums

with

the

term

/, where

sigma

sum is

above

The

notation.

goes from 1 to

As
\302\253.\"

can comparethis
observe that, by

sum

as

written

which

\302\243/,

seen in Section
case where each

we have
to the

time of the

the running

cutting

2.2,

\"sum of

means

this sum is equal to

step requires
/th

the

exactly

(n +

We
and we

l)/2.

n operations,

from n to /, we

step

save a factor

of

2.

about

3.1

\342\226\241

Example

now

Consider
other

words,

the
of executing
case
we are looking for the

a loop in

which

the /th

step requires i2 operations. In

summation

S2(/i)=ii2.

It is
step.
are

clear that

guess
induction

that

since

S2(n)<n3,

Judging from this

within a constant.
S2(n)
step

P (n

n3 is equal to running
we can guess that

the
the

for n2

loop

differences

We can prove our guess, and find the constants,

= an3+bn2 + cn+d. P(n) must
satisfy
+ 1)= P (n) + (n + 1 )2. The induction step implies that
+ l)2+c(\302\253 + l)

(since coefficientsof

^-(a\302\2533+/7\302\2532+c\302\253+^)

the

same

power of

must

operations

= \302\2532
+ 2\302\253
+ l,

be equal)

in

each

S2(n) and n3
by induction. We
and the
P(\\)=\\

between

= P(n)

+ l)3+/?(\302\253
a(\302\253

which implies

example,

that

Analysis of

44

3a

the coefficient of n2

= 1

+b-b

3a +

Algorithms

2b + c -

c =2

the

coefficient

of

the

coefficient

of 1 .

\302\253
,

and

= 1

a+b+c+d-d

that a = 1 /3, b =
These equations imply
initial condition (P (1) = 1),which
implies
all

S2(n) = T
it is

There

that by reducing

the

size

(3.3)

of the

ith

/2, we

n2 to

from

step

is another

\342\226\

at expression

to arrive

way

times. If we guess that

express S2(n) as a combination

an equation
for S2(n) by solving
try to
the

is a

S2(n)

of

such

(3.3).

is a

It

general

technique

we

that

third-degree polynomial,
We then arrive at the solution
then

we

can

polynomials.

involving

and

it

other

Consider

polynomials.

explicit

sum

(3.4)

S3(n)=i,i\\

We

the

3.

of about

use several

\342\200\242

gnote

to

interesting

savea factor
will

The value of d comes from

Hence d = 0. Combining

c = 1 /6.

that a+b+c+d=\\.

we get

the terms,

Again,

1/2, and

will

first

S3(n) =

write

(3.4)

in

a different

i,i3 = X(/-l

In other words, we shift

from
1 to n. This shift

right side of (3.5),and

the

way:
X0' +

l)3

X(/3+3/2

so that the sum

summation,

in Fig.

is illustrated

l)3 =

3.1.

We

can

goes
now

3/+l).

from

equate

(3.5)

0 to

the

left

- 1 instead of
side

expand:

X/3=I(/3+3/2+3/+l).

The /

terms

canceled.We

(0+1)3

(3.6)

/ ranging from 1 to
then
write an equation

for

the

and

- 1 are

common to
the

involving

I3

23

(l+l)3

(2+1)3 +

Figure

3.1

Computing

rest of

a summation

both

sides

(/i-D3

(\302\253-l+l)3

by

shifting.

and can

of (3.6),

the terms from

sides.

both

n3

be

3.4

45

Summations

n-\\

+
\342\200\2363=03

3/ +

(3i2 +

l).
n-\\

n-\\

We

know

already

difference is
n3 =

n3

the

in

l)/2,

clear that

it is

and

(the

\302\243/2=S2(n)-n2

only

Hence,

\302\253th
term).

solve for

- l)/2

(n

+ /i.

S2(n):

-3n(n-\\)/2-n=3(S2(n)-n2),

which implies
r/,

that

\302\2533
3\302\253(\302\253
-1)/2

S2(\302\253)

which

= n(n-

J)/

3(S2(n)-n2) + 3/i

now

can

We

that

-\302\253

2
+ \"2 =

is, of

course, exactly the

The main
example) in two

same

\302\2533

3\302\2532
+

\302\253

-T

/i(/i

as (3.3).
was to use a

+ 1)(2/i

+ 1)
\342\200\242

expression

this derivation

sum
in this
(S3(n)
each other. Many
other
the difference between a sum
exhibit the same behavior. If we consider
sums
'' '
f\\ +/2 +
+//i and a shifted sum f2+fi+ '' ' +/\342\200\236+i,we see that most of the
terms are left. We presentthree
cancel each other. Only the boundary
more
coefficients
of this technique.
examples

different

We want to

compute the

/r(/i) =

\302\2432'\"

would like to

and by canceling

2F(\302\253)

= 2

+ 4+

1+2

an

particular

cancel

to

another

whole

terms
F(n) by shifting
consecutive terms in F(n) is a
will allow
us to shift):
by 2 (which

expression

difference

The

expression

involving

between

\342\200\242\342\200\242\342\200\242

+2,?+2\"+l.

= 2\"+l

implies that

involving

F(n)

F(n):

-1.
= 2n+[

- 1.

\342\226\

3.3

\342\226\241

Example

now

the

following

slightly

G(/i) = 2'2'\" = l-2l +2-22+

We can

mostly

+2\".

the

expression

2F(n)-F(n)

Consider

they

\342\200\242\342\200\242\342\200\242

compare F(n)
most of them.

+ 4 + 8+

now get

But, this

that

sum:

following

factor of 2, so let'smultiply

We can

such

ways,

3.2

\342\226\241
Example

We

in

trick

apply

the

same

technique:

more difficult
3-23+

sum:

\342\200\242\342\200\242\342\200\242

+/i-2\\

Analysis of

46

Algorithms

incremented

(we simply

the effect of the

G(n)

By

subtracting

-(1-21

+ 1'22+

power).

= n-2n+l

= 2G(n)-G(n)

Finally, we
of heapsort:

-(2\"+I -2)

consider the

G(\302\253)

can apply

sum,

following

+2-2\"-2

\302\243i2#\"\"=l-2#|-1

the

\342\200\242\342\200\242\342\200\242

+1-2\

which

will

+ 3-2',-3+

in Section

appear

6.4.5

in

the

analysis

\342\200\242\342\200\242\342\200\242
+/i-2\302\260.

\342\200\242\342\200\242\342\200\242
+\302\253-2!.

Probably

we eliminate

expressions,

= 2n

+ l-2\"-2 +

\\'2n-1

the effect of the

/ factor:

\342\200\242\342\200\242\342\200\242
+ 1-21 -/i-2\302\260.

Relations

relation

recurrence

function.

two

= 2G(n)-G(n)

Recurrence

3.5

we eliminate

expressions,

the same technique:

by subtracting

G(n)

two

= (n-\\)2n+l+2.

+3-2\"-2+
2G(\302\253)=l\342\200\2422',+2\342\200\2422',-,
Again,

the

3.4

Example

We

the

+ n-2n^

i factor:

= n-2n+{

\342\200\242\342\200\242\342\200\242

+ 2-23+3-24+

2G(\302\253)=l-22

is a

the most

to define
a function by
famous recurrence relation

way

an

involving
one defining the

the same

expression

is the

Fibonacci

numbers

F(n) = F(n-\\)

+ F(n-2), F(l)=l, F(2)=\\.

expression
uniquely defines the function.
value of the
function
at every
number
F (4) = F (3)+F (2)= 3, and so on. However,

We can

This

the definition,
we
convenient to have an explicit
us to compute F(n) quickly,
called solving the recurrence
recurrence.
following

would

need k-2

(or closed-form)
and

relation.

compute from

For example,

k.
if

(3.7)

we

compute

the value

Recurrence relations appear frequently

of the

steps to compute F(k). It

expression for F(n). That

the

expression

F(3)= /r(2)+

F(n) to other known

We sometimes call a recurrence

to compare

this

= 2,

F(l)

function

is much
would

by

more
enable

functions.

This

is

relation

simply

in the analysis
of algorithms. We briefly
a useful technique for solving recurrence relations, and present
general
solutions of two classes
of recurrences
that are among the most common
recurrences
These recurrences will be used later in the book.
involved in analyzing
algorithms.

discuss

here

Recurrence

3.5

Relations

47

3.5.1 Intelligent Guesses

works

like a nonscientific
a wide class of recurrence

may seem

a solution

Guessing

well for

very

to find
is
useful
guessing
bound. Consider
are trying

our pride aside, it

even better when we
bound. The main reason that

but, keeping

method,

It works

relations.

the exact

solution, but only an upper

is valid is easier
proving that a certain bound
recurrence
for
which is defined only
following

not

is that
the

than

values

computing

the

of n that are

powers of 2:

7(2/i)

1.

7(2)=

+2/i-l,

<27(/i)

(3.8)

rather
than equality. This is consistentwith
bound
and with
(in the form of the O notation),
finding
only
We want to find a function
side represents the worst
case.
the fact that the right-hand
but we also want to make sure that /(/?) is not too far
/(/?) such that T(n) = 0(f(n)),
from the actual
T(n).
Given a guess for / (n), say f (n) = n2, we prove that 7(n) = O (/(/?)) by induction
on n. First, we check the base of the induction. In this case, 7(2) = 1 </ (2)= 4. We then
We

our

prove

The

wrote

modest

this

an

goal of

implies that

T(n)<n2

implies

proof is

as follows:

< 27

inequality

an upper

that T(n)<f(n)

7 (2/2)

as

recurrence

(n) +

< 2/22+ 2/2

7(2n) < (2n

-

2/2

1,

- 1,

<

7(2/2)

/ (2/2).

need

We

to prove

that

f.

(by

the

definition

of the

(by

the

induction

hypothesis)

recurrence)

<(2/2)2,

which is exactly what we wanted to prove. Thus, T(n) = 0(n2). Is n2 a good estimate
for T{n)l In the last step of the proof, 2/22 + 2/2 - 1 was substituted
by the greater An2.
two
which gives us
But there
these
between
is a substantial
2n2)
expressions,
gap (about
a hint that maybe n2 is a high estimate
for T(n).
Let's try a smaller
estimate,
say, f(n) = cn for some constant c. It is clear,
is no
than
does, since c2n=2cn, and there
however, that en grows more slowly
T(n)
room for the extra 2/2-1.
Hence, T(n) is somewhere betweenen and n2.
Let's

try

now

\\og2n. Clearly,

T(n)<n

7(2) < 21og22.Assume

that

T(n)<n

log2/2,

and consider 7(2/2):

7(2/2)< 27(n)+
< 2/2

log2/2

- 1,

2/2

2/2

- 1,

(by

the

definition

of the

(by

the

induction

hypothesis)

recurrence)

< 2/2(log22/2),
which

is exactly

are very close.

constant.

what we

Later,

we

wanted

will

The leeway
that this is actually

to prove.

prove

in

the
the

proof
exact

is only 1 now, so we
solution to within
a

Analysis of

48

a similar

define

can

(3.8) is defined only for

recurrence for all values of n in

relation

recurrence

The
We

Algorithms

The recurrence

of

of

Therefore, we already know

We now show

that

monotonically

increasing

2k is

where

the

< T(n)

7(2*_l)

power

< 7(2*). We

that

proved

to assume

sufficient

2, T(n) = O

implies

c.

Hence,

cxn\\og2n.

that

T(n)

power of

is a

that

for some constant

7(2*)<c2*log22*

<

c\\, which

constant

another

as (3.8).

same

the

exactly

that are powers of

that,
(n logn).
bound applies to all values of T(n). It is clear that T(n) is
If n is not a power of 2, T(n) is no more than
function.
7(2*),
n. That is, let 2*_l < n < 2k\\ clearly,
of 2 that is greater than

T(n)<c2k\\og22k<c(2n)\\og2(2n)
for

for integers.)
only
it is defined for all values

same

the

first

2, (3.9) is

powers of
for values of n

are

that

way:

defined

relation

values

for

but,
\302\253,

powers of 2.

(3.9)

floor symbol is necessary, because T(n) is

since
is more general than
(3.8),
(3.9)

the

that

(Notice

of n that are

the following

7(2)=1.

+ /i-l,

r(/i)<2r(L/i/2j)

values

= O(n

logn) for all n.

looking for an

are

we

when

It

is

usually

asymptotic

expression.

Let's summarize
relation. Suppose that

the

to a recurrence
proof of a solution
of the following
form:
recurrence relation

in an inductive

used

steps

a general

have

we

T(g(n)) = E(T9n)9

(3.10)

where g (n) is a function

of the recurrence), and E is some
of n (which defines the growth
and n. For example, in (3.8),
T(n)
expression involving
g(n) = 2n, and E(T,n) =
that
we guess that 7 (\302\253)</(\302\253),for some function f(n).
2T(n) + 2n 1. Suppose further
To prove our guess, we need to substitute
for n in/(\302\253), then to substitute f(n) for
g(n)
each occurrenceof T(n) in E. We then have to show that f(g(n)) is greater than or
for \302\243(7,n).
In other words, we have to prove that
equal to the value substituted

(3.11)

f(g(n))>E(f,n).

For example,

in

(2/i)(log2(2/i))

> 2(/i

than\"

We are

following.
if

we

substitute

trying

g(n)

repeated

times

several

reasonably

for

the

notation

even
through

though
the

solution

in

prove

to

try

that f
prove

intuitive,
that/(\302\253)

in /(\302\253), we

T(n).

But,

(n) = n

we

thus,

\\og2n;

had to show

'

that
the opposite \342\200\224
is, to replace
to
and
remember,
explanation
easy
more quickly than T(n) does.
grows

should

get a value larger

T(g(n)) = E(T, n)

replace T(g(n)) with

with different functions

\302\243(/,n).

than

(this

is

This

process

(guesses) until

the

'greater
is

the

Hence,

we get by
the recurrence

what

exactly

proof

that

may have to
of the inequality

be

tight.

Another common

guess that

An
to

substituting g(n) for n

relation); thus, we can
becomes

is to

than.\"

\"less

with

guessed

\\og2n) + 2/i - 1.

mistake

common

we

(3.8)

mistake

is O

(f

is

to use

and
(\302\253)),

the O notation
to substitute

cannot be used in that way. The problem

we do not care about the constants at
For example, if we try to prove that
proof.

That is, we
However, the
the O notation is that,
using
end, we cannot ignore them
of (3.8) is O (\302\253),
solution
by
when

we try

0(f(n))

with

the
the

guessing.
for n.

for

O (n)

substituting

+ 2n

< 2T(n)

T(2n)

< O (n) + 2n

following (the base case is trivial):

get the

we
\302\253,

- 1,

- 1,

49

Relations

Recurrence

3.5

(by

the

definition

of the

(by

the

induction

hypothesis)

recurrence)

= 0(/i),
as we

is wrong,

which

were used (or rather

to include

cf (n) for some constant

now
to solve the Fibonacci
try
it is

that

Let's

F{n) = F{n-\\)

is

is doubled every time;

c2n in (3.12), we get

that F(n)
Substituting

c, and

that different constants

fact

the

in

correct

The

''proof.\"

to

that the

guess

namely,

approach is
is 0(f(n)),

solution

the value of c later.

determine

by guessing.

we

Again,

are given that

(3.12)

F(2)=l.
previous values, a reasonable
it is approximately 2\".
Let's

of two

sum

the

the

want

relation

+ F{n-2), F(l)=l,

of F(n)

value

the

Since

stages of

When we

explicitly.

lies

error

The

at different

ignored)

constants

the

we guess

have seen earlier.

try

be

would

guess

F(n) = c2n.

c2\"=c2\"-1 + c2n~2.
and the left side is
impossible, since c is canceled
the
that c2n is too large, and
that
side. So we learned
constant c plays no role in the induction
step.

is clearly

This

equality

than

the right

The

next

could

attempt

Instead of guessingdifferent
its value

compute

a\" in

Substituting

through

be another exponential function,

it is easier to introduce
bases,
We

verification.

the

will

try

base.

base and
a is a constant.

as a

a parameter

F(n)

multiplicative
a smaller

with

but

greater

always

= a\", where

to

(3.12), we get

an=an-1 +an~\\
which

that

implies

+ l.

a2=a
The
now
find

that

know

F(n)
c such

a constant

If we
more

are ax =(l+\\5)/2 anda2= (l ~ V5)/2.

= 0((a\\)n), since {ax)n
satisfies
the recurrence,

for (3.13)

solutions

two

values

(3.13)

want

to

that

find

carefully.

combination of them

does.

c(a

{)n is

greater

Since

given

F(n), we

{ax)n

and

general

solution

both

So the

the

than

the exact value for

values for
will

{a2)n solve
of the

we
in particular,
and we can easily
= 1 and n = 2.

So,

the

to consider

need
the

recurrence,

recurrence

any

initial

linear

is

c,(a,r+c2(a2r.
We
and

exact

need
F (2).

to compute

a simple exerciseto
of the Fibonacci relation

It is

solution

the values of c \\

and

that

verify
is

so that

c, =

the expression
1

/\\5,

and c2

fits

the

values

of F (1)

=-1 / V5. Therefore,

the

of Algorithms

Analysis

50

1+

i-V?

V?

F(\302\273)=-p

^5

V5

The equation

+ 1, which
we encountered
relation (3.12), is called the characteristic
is the basis for solving
relation. The same technique

in our

= a

a2

recurrence

F{n) = b{F{n-\\) +
for a

In

and

Divide

\342\200\242\342\200\242\342\200\242

+ bkF{n-k)

b2F(n-2)+

Relations

Conquer

the problem is divided

algorithm,

a divide-and-conquer

recursively,and a combine
that there are a subproblems,

Assume

and that the

algorithm

for someconstants
T(n) =
We

0, b,

c, and k.

The running

time

simplicity,

that n =bm,
to expand

try

+ c(n/b)k)
T(n) = a(aT(n/b2)

T(n)

if we

expand all the

so

assume

that

T(\\)

= c

runs

original

problem,

original

in time

cnk,

satisfies

thus

algorithm

an integer
to get

(b is

an

integer

the feel of it:

+ c(n/b)k)+

cnk.

= 1, we get

to nlbm

\342\200\242
\342\200\242
+ c:/!*.
\342\200\242)

would

value

different

times

+ c(n/b2)k)

+ cnk=a(a(aT(n/b3)
way

(a

is always

nib

that

(3.14) a coupleof

\342\200\242
\342\200\242
T(n) = a(a( \342\200\242
T{nlbm)^c{nlbm^)k)

Let's

solve the

subproblems

of the

each

subproblems,

(3.14)

We first

for

than 1).

In general,

smaller

aT(n/b) + cnk.

assume,

greater

the solutions of the

to combine

used

into

algorithm is used to
each
of size Mb of the

subproblem is solved
problem.

recurrence

form

of the

recurrence

any

the

to

solution

of the

k.

constant

3.5.2

search for a

equation

change

the end

result

only

by

constant). Then,

T(n) = cam

T{n)

c^am-lbik

this

Case 1: a
this

constant

am = a
of

+cftm*,

both

fok

cam%(\342\200\224)

i
\342\200\242

geometric series. There are

than, greater than, or equal to 1.

a simple

is

is less

(bk/a)

/=o

/=o

In

\342\200\242\342\200\242\342\200\242

implies that

which

But,

+cam-2b2k

+cam-[bk

three

cases,

than

1, so

depending

on

whether

> bk

case, the factor of the geometric

even
if m goes to infinity.
\302\260Sh\"

(the last equality

=nlog\"a

sides).

Thus,

T(n) = 0(n^ha).

series

is less

Therefore,

can be easily

T(n)

= 0(am).
proven

converges to a

the series

Since

by taking

\\oghn,

logarithm of

we get

base b

3.5 Recurrence
Relations

In

a=bk

2:

Case
this

of the

the factor

case,

a = bk

51

that

implies

is 1, and

series
geometric
= kand
m-0(logn).

\\ogha

thus

m). Noticethat

= 0(am

T(n)

Thus,

T(n) = 0(nk\\ogn).

Case3: a < bk
In

this

the factor of the geometric

series
is greater
bkla
summing a geometric series. Denote
element
of the series is am, we obtain

case,

for

expression
first

the

T(n)

pm

_ i

+ l

am\302\243

F -1

are summarized

cases

three

These

= 0((bk)m)

-=0(amFm)

the

in

1.

by F

(F is a

= 0((bm)k) = 0(nk).
theorem.

following

The solution
relation 7(n) = aT(nlb)+
of the recurrence
b are integer constants,a > 1,b > 2, and c and k are positive

<j

standard

constant). Since

3.4

Theorem
\342\226\241

T(n)=

use the

We

than

0(n^a)

if

0(nk\\ogn)

ifa=bk.

0(nk)

ifa<bk

cnk,

a and

where

constants,

is

a>bk

\342\226\

of Theorem

result

The

should be memorized. This

to predict the running

3.4 applies to

result

used

time.

is also very

many

helpful

Generalizations

in

the

of

this

design
formula

It

algorithms.

divide-and-conquer

stage, since
are

it

be

can

in the

given

exercises.

Relations with Full History

Recurrence

3.5.3

not just

function,

relation

recurrence

full-history

on a few

of them.

is one
One

that

of the

on all

depends

simplest

the previous values

full-history

recurrence

of the

relations

is

r(/!) = c

a constant

c is

where

(3.15)

and

we used to

method

7(1)

history.)

We

We can solve
will

try to write

this
the

recurrence
recurrence

by using the
in such a way

terms will be canceled.

(This method is sometimes called elimination
For the recurrence (3.15),we compare
T(n + 1) to T(n):

T(n + \\)
If we

is given.

compute sums.

of the

most

which

+ \302\243r(i),

subtract

clearly

= c

+ ZT(i).

(3.15

) from

implies

that

same
that
of

(3.16)
(3.16), we get T(n + \\)-T(n) = T(n). So,T(n + 1) = 27(\302\253),
is true for 7(1), and, by
claim
T(n + l) = r(l)2\". (This

of Algorithms

Analysis

52

claim is

if the

induction,

This argument

7(1)= 1 and
be

should

induction

an

through

does

very

= 7(l)

7(2)

that

+
7(\302\253

simple. The next

the analysis of the average

For

example, set

of carelessly
example
The error results from the fact
another

at

case

but

we

it

is very

in

discuss

= 0.

7(1)

(for\302\253>2).

(3.17)

to

most of

cancel

both sides of (3.17)by

the

T(i)

+ 1):

(n>2).

-^yX7(/)

we multiply

want

for T(n

expression

corresponding

l)-l

We

terms technique.

canceling

the

+
(\302\253

convenience,

so simple,
of quicksort which

is not

is

relation

\302\2437(1),

one

i=\\

and

shifting

l) =

case.

is very

/i-1 + -

Let's look

terms.

the

double

canceled
7(1) is not necessarily
by c. One
when a parameter (c in this case) that appears in the expression
final solution.
the
To solve this problem
we note that
correctly,
and that the proof above is correct
for all n>2.
Hence,
definition),

use the

This is
base

proof ignoring
for T(2), since

in
important. It appears
Section 6.4.4. The recurrence

7(/i) =

7(2)

We can, for

incorrect!

is

it

= 6*27(1).
the

work

not

recurrence

This

We

but

+1), since we

for T(n

+ c)2''-1.

= (7(l)

+ l)
7(\342\200\236

''clear,\"

is true

it

suspicious

not appear in
+ c* (by

does

be

may

5, and see

c =

the proof

that

then

time.)

value every

going

T{n),

for

true

(3.18)
and
\302\253,

both

sides of

(3.18) by

n +

1:

n-\\

(/i +

+2

= n(n-\\)

nT(n)

l)r(/i + l) = (/i

(n>2).

\302\2437(/)

+ l)/i+2

now

(n +

which

(3.19)

\\)T{n +1)

(3.20), and

from
=

nT(n)

(n +

(3.20)

- n

\\)n

obtain

(n

- 1)+

2T(n)

= 2n

+ 2T{n)

(n

>2),

implies that
+
r(\302\253

This

subtract

(n>2).

2;r(i)
/=

We can

(3.19)

l) =

recurrence

+
^7(\302\253)
n+\\
is easier to

-^n+\\

solve.

(n>2).
First,

we

substitute

2 for
\302\253
+ l

, and get a close

approximation:
+
r(\302\253

If we

l)<-^_7(\302\253)
n +1

expand (3.21), we get

+ 2

(n>2).

(3.21)

3.6

\302\253
+ l
7\302\273<2

= 2 1 +

=2

1+

+ \342\200\224
+

+ l)(//(\302\253 + 1)-1.5),

1+

without

n-2

(1)

for T(n)

+ y+0(\\/n),

series

has

where

is

\\ogn).

several

proof,

Harmonic

equalities

and

inequalities

that

are

algorithms.

analyzing

series

Arithmetic

1+2 + 3

will

we

Harmonic series. The

not prove, H(n) = \\nn

is the

= O(n

we present,

n-\\

n-\\

+7- 1.5) + O

section,
in

solution

Facts

n-2

constant. Hence,the

+ l)(ln/i

< 2(/i

\342\200\242
\342\200\242
1/2+ 1/3+ \342\200\242
+ \\ln

which

n + \\
\342\200\242
\342\200\242
\342\200\242
\342\226\240

1
\342\200\242
\342\200\242
\342\200\242
\342\200\224

n-\\

n +1

Useful

this

n-2

n-\\

+ l
\302\253
\342\200\242
\342\200\242
\342\200\242
\342\200\224\342\200\224-

111

= 2(/i

7\302\273

In

n-\\

n-\\

+ l
\302\253

+ l)

a simple approximation,
7=0.577.. is Euler's

useful

+ l
\302\253

= 2(/i

where H(n)=

3.6

n-\\

4_

n-2

n + \\

n-\\

2+

n-\\

\302\253+l

\302\253+l

+ l
\302\253

2+

53

Facts

Useful

More generally, if an

=
\302\253

/!(/! +1)

(3.22)

+c\\ where

=an_\\

c is a constant, then

n(an+a{)

\342\200\242\342\200\242\342\200\242
=
+0\342\200\236

ax +02+03+

1+2 + 4+
generally,

\342\200\242\342\200\242\342\200\242

+2\"=2\"+1-l.

if

an

<c

< 1,then

=can_\\,

the

sum

of the

(3.24)

c*\\ is

where

'*' +an=a\\

ai+a2+fl3+
If 0

(3.23)

series

Geometric

More

a constant,

then

cn-\\

infinite

(3.25)

c-\\
geometric

series

is

Analysis of

54

2>
/

Algorithms

(3

-TT-

=1

of squares

Sum
\"

+ 1)(2h

h(h

.,

+ 1)

(3

h\"

Harmonic series
=

//\342\200\236
\302\243
j

= ln/i+Y

+ 0(l/\302\273),

is Euler's

constant.

y= 0.577..

where

(3

Basic rules involving logarithms

1

\\ogha

(3

\\ogab

loga* =

log/,*

(3

>og/,a

(3
(3
of logarithms

Sum

ILlog2'J

+
=(\302\273

!)Llog2\"J

-2Llog!\"J

+ 1

+2

(3

0(wlog\302\253).

1=1

a summation

Bounding
If

is a

f(x)

integral

monotonically increasing continuous

\342\200\236

function,

then

.v=n+l

(3

J /(*)</*.

I/cos

/=!

,v=l

approximation

Stirling's

n! =

by an

\\2tw

In particular,

(3

(1+0(1//i)).

Stirling's approximation implies that

log2(w!)

= 0(w

logn).

BibliographicNotes
3.7

Further

and

55

Reading

Summary

Niels Bohr

that

said

once

of an

it is

but

algorithm,

We

characteristics

important

predict; especiallythe

not that
The main

It is

future.\"

from

far

being easy.
to extract only the
many details and attempt
of the algorithm.
The O notation is useful
in that respect,
is
a
it
first
On the other hand,
the
only
approximation.

method we use is approximation.

most

hard to

is very

\"it

the behavior

to predict

hard

but we must never forget that

in analyzing algorithms
attempting this task. It is essential

ignore

to get

designer

from

efficiency

of an

algorithm

of

indication

some

least

at

the

deter

not

should

difficulty

the

algorithm.
In

The first
This will

many

cases,

thing

we should
us

give

The first

do with
some idea of the

terms

few

when recursion is

especially

help

in

of the

behavior

relation,

but

relation.

a recurrence

get

is to look

first pass at guessing

times, as we did in

the

making

used, we

relation

a recurrence

at

first

the

few terms.

by no means enough.
a solution.
Another useful
it is

Section
3.5.2.
Guessing and
good technique for solving recurrence relations, but it is usually just a first
\342\200\224
We must be careful not to \"overguess\"
that
is, to try an upper bound that is
step.
There are many
other
most
correct, but too pessimistic.
techniques.
Fortunately,
in
lead
to
one
of
a
few
of
classes
recurrence
that
practice
very
appear
algorithms

is to

step

expand

verifying

is a

relations,

most

a first

several

are described

of which

that n

step,

recurrence

the

has a specialform

this

It is

chapter.

Further

to assume,

sufficient

usually

that n is

particular,

and

Notes

Bibliographic

in

\342\200\224
in

a powerof

as

2.

Reading

analysis was promoted in the early 1970's, and it was met with
There are
efficiency.
by now the major measure for algorithm
\342\200\224
on
and
that
cover
several books \342\200\224
discrete
mathematics
combinatorics
mainly
recurrence
and
other
needed
for
summations,
relations,
expressions
techniques
evaluating

The

of asymptotic

idea

some resistance.

for

analyzing

Knuth,

and

It

is

Brualdi

algorithms.
Patashnik

[1989]

devoted entirely
to algorithm
Additional books and survey
Purdom

and

Knuth

Brown
[1976]

[1985a],
discusses

[1977], Bavel

are just
analysis.
papers

latter

Greene

include

Flajolet and Vitter

the relatives of

solving recurrence relations can be found

Saxe [1980]. (The

contains

[1982],Roberts

[1984],

and

a few examples. There

are fewer
books
Knuth [1973a] providesa rich source of

in

[1987],
the

Lueker

the solutions to

and

Knuth

[1982],

and Hofri
notation.

[1980],

Lueker

Graham,
that are
material.

[1980],

[1987].

techniques for
and Bentley, Haken, and

Additional

Exercises 3.23 and

3.24.)

Tarjan

discusses amortized complexity, which

is an elegant
method for analyzing
the
in a more
times of certain algorithms
if a certain
running
precise way;
part of the
is performed
several times, each time
with
a different
algorithm
running time, then,
case
instead
of taking the worst
The
time, we amortized the different
costs.
every
3.19 is from Manber [1986]. Exercise3.21is from
in Exercise
recurrence relation
Purdom
and Brown [1985a].
[1985]

56

of

Analysis

Algorithms

Exercises

Drill

(n) is a polynomial

3.1

Prove

that,

if P

3.2

Prove

that,

if f(n)

3.3

Prove, by

using

= o(g(n)),

in /i, then
then

Theorem

3.1,

that

Theorem

3.1,

that

O (log (P (/i)))

Is the

= 0(g(n)).

f(n)

= O

(log n).

opposite

true?

/i(log3/i)5=0(/i12).

3.4

Prove,

by

pairs of functions
Compare the following
whether f(n) = 0(g(n)),f(n) = Q(g (n)),

in

terms

of order

100/1+log/i

b.

log

n +

n2

(log /i)2

n (log

(log/i)log\"

nL

(log/i)5

f.

n 2\"

3\"

Solve the following

recurrence relation.
T(\\)=

Give

an exact

solution.

Give

an exact

solution.

1.

Solve the following recurrence relation.

T(n) = ST(n-\\)- \\5T(n-2); T(\\)=\\\\

Prove that

T(n),

which

is defined

+ 2/ilog2/T,
7,(/7)= 27,(L/i/2j)
satisfies T(n) = 0(n
The following

matrix

)2

log /i

e.

T(n) = T(n-\\) + n/2\\

3.9

case, say

/i

d.

3.8

In each

log (/i2)

log/i

3.7

magnitude.

S(\

a.

c.

of

= Q(g (n)).

and/or f(n)

fin)

3-6

>0

constants a, b

for all

= 0(nb).

(\\og2ny

3.5

using

multiplication

= 4.

7(2)

the recurrence

by

T(2)

relation

= 4,

log2/?).

recurrence
([Pan

relation

describes

1978]). What

the running time

is the

asymptotic

of a

running

recursive

time of this

algorithm
algorithm?

for

T(n)= \\43640T(n/10)+ O(n2\\

3.10

Find the mistake

57

Exercises

Creative

T(\\)=\\.

Let A

works
on complete
depth). Suppose that
k is a parameter that has to do with
A performs
0(k) steps for each leaf in the tree, where
of the tree), and
the amount of information stored in the leafs (but is otherwise
independent
node.
We claim that
the total running time of the
time c per each internal
constant
trees

binary

the

in

following

binary trees

(namely,

analysis.

in

which

be an

all the leafs

algorithm

are at

that

the same

is 0(k).

algorithm

'
is by induction on /i, the number of nodes in the tree. If n = 1,
'proof\"
Assume that the claim is true for all
number of steps is obviously
0(k).
<
Such a tree consists
and
consider
a
tree
with
with
n
n nodes.
trees
nodes,
complete binary
the
induction
of a root and two subtrees, each of size (n - l)/2.
By
hypothesis, the running
time for the tree is 0(k) + 0(k) + c.
time for the two subtrees is O(k). Hence,the running
But
this is equal to O (&), and the proof is complete.
Wrong proof: The
then

total

the

3.11 Solvethe

= max

inO),

7\302\273

where 7(1)=

3.12

Solve

recurrence relation:

full-history

following

1.

the following

full-history recurrence relation:

n-\\

r(/i)

/i

+ Sr(/)f

where T(\\)=

3.13 Use(3.34)
2i*

1.

to prove

that, for every

positive

integer

&,

= 0(/i*+I).

/=i

3.14

Use (3.34)

to prove that,

for every

positive integer k,

= 0(nk+l\\ogn).

Zik\\og2i

i = \\

Creative

3.15

Find

Exercises

that f(n)-g(n)

3.16

Find

that f

to the following claim:

= 0(s(n)-r(n)).

f(n) = 0(s(n))

and

g(n)

= 0(r(n))

imply

to the following claim:

f(n) = 0(s(n))

and

g(n)

= 0(r(n))

imply

a counterexample

a counterexample

(n) I g(n)

= 0{s{n)I r{n)).

3.17

Find two

functions/(n)

and g(n)*0(f(n)).

and

g (n),

both monotonically

increasing,

such

that f(n)*0(g(n))

58

3.18

of

Analysis

Algorithms

Consider the

relation

recurrence

T(n) = 2T(n/2)+\\,

T(2)=\\.

attention to powers of 2). We guess

and substitute en in the expression
c,
T(n)<cn
Section 3.5.1). We have to show that en >2e(n/2) + 1. But this is clearly not true. Find
correct solution of this recurrence
(you can assume that n is a power of 2), and explain
this attempt failed.
We

to prove

try

that

= 0(n)

T(n)

for some (as yet

3.19

where m and

3.20

c are constant

our

the
why

the following recurrence relation:

satisfies

(The solution

solution for

that

(see

1,

5(2)=

parameters.

the asymptotic

that

Prove

which

cm log 2mS(n) + 0(mn\\

S (mn)<

limit

constant

behavior of S (n),

the asymptotic

Find

(we

unknown)

should be a function

of /i, m,

and c.)

relation

the recurrence

T(n) = 2T(n-c) + k,
both c and k

where

3.21

The

are integer

relation

recurrence

following

problem is divided

into

is T(n) = O

constants,

unequal

(dn) for

some

in divide-and-conquer

appears

constant

d.

algorithms

in

which

the

size parts:

it

T(n)

+ cn.

2,aiT(n/bi)

i = \\

All

the

tf,s

6,s are

and

constants, and

satisfy

they

i-2>iA->\302\2601=1

Find
3.22

the asymptotic

behavior of this

Solve the following two

relation

recurrence

recurrence relations.

It is

(by guessing and

sufficient to

find

the

verifying).

asymptotic

behavior

of T(n).

a. r(/i) =
b.
(Hint:

3.23

Prove

+ l;

4rf[^l]

= 27\"

7\"(/i)

H-2/i; T(2)=

rr^TJ]

Substitute

another

the solution

that

T(n) = kT(n/2)+f(n\\

T(2)= 1.
1.

variable for n.)

of the

recurrence

relation

T(\\) = c

is
T(n)

nlogk(c+g(2)

+ g(4)+

\342\226\240\342\226\240\342\200\242
+*(/!\302\273,

where
You can assume that n is a power of 2. (This
is a
g(m) is defined as /(m)/wlogA.
more general solution
than the one given in Section
to any function
3.5.2, since it applies
/(\\342\200\242)

3.24

Prove that the

solution of the

recurrence

relation

Exercises

Creative

T{n) = kT{nld)+f{n\\

= c,

T(\\)

T(n) = nlogdk(c+g(d) + g(d2)+

where g (m) is defined

3.25

T(n) = T(nl2)+ T(\\y[n\\)

3.26

can

Find

the asymptotic

7>) = 7>/2)

+ Vrt~,

can assume that

T(n)

power of d.

by

the recurrence

relation

by

the recurrence

relation

by

the recurrence

relation

1, T(2) = 2.

are powers

that

defined

n is a

function

T(n)

of 2.
defined

= 1.

T(\\)

of n that

values

= T(n/2) +

T(n)

T(\\)=

behavior of the

the asymptotic

Find

. You

function

behavior of the

You can consider only

3.27

+ n,

only values of n

consider

You

+ g(n)\\

behavior of the

the asymptotic

Find

\342\200\242\342\200\242\342\200\242
og

/ (m)lm

as

59

are powers

function

T(n)

defined

T(2)= 2.

T(\\)= 1,

+ n (n>2),

T\\

of 2.

\\og2n

3.28

You

can

consider

only values of n

that

Find

the

solution

of the following

recurrence

behavior of
satisfies

T(n)

T(n). You should

(n) = Q(T(\302\253)).
+ 3/i

2T\\

are powers

of 2.
evidence

T(\\)= 1, T(2)

(n>2),

It is

relation.

give convincing

sufficient to
that

the

find

function

the

asymptotic

f(n)

you

find

= 2.

log2fl

3.29

Although

is not

in

it is

general

the case.

always

T(n/2)+

T(n) = 2T((n-\\)/2)
with

that

the

solution

powers of 2, T (n)

b.

Show

that,

= O

3.30

this

if

of 2, that

of this

recurrence

prove

relative

of

behavior

that

satisfies S(n) = 0(n).

the following

for

powers

of 2 is

T(2k) = k+

(namely,

for

(log n)).

recurrence.

Use (3.34)to

Compute

for powers

relation:

odd,

s(/o=i;riog2(/i//)i

3.31

recurrence relations only

is even

n is

for an infinite number

assumption about the

for

if n

following

1.

T(\\)=

a. Prove

to evaluate

sufficient

Consider the

sum precisely:

of /i, T(n) = Q(n). Discuss why

the usual
for powers of 2 and nonpowers
of 2 breaks down

values

Analysis of

Algorithms

5(/i)=snog2(/i/oi.

You can

assume that

The Fibonacci
and so

numbers
F(/i+2)

definition:

n is a

Prove that

G(/i)

to solve

can be extended
+l) + F(/i), and F(l)=l,

F(n)

= F(/i

on). Let G(n) be

suggest a way

power of 2.

defined

it.

= (-l)\"+,F(/0.

as F(-n).

to

negative
F(0)

=0

values of

using

(e.g., F(-l)=l,

Write a recurrence

relation

for

the same

F(-2)=

-l,

G{n),

and

CHAPTER

TO

INTRODUCTION

BRIEF

STRUCTURES

DATA

is nothing

Science

but trained

and

common

organized

sense.

T.H.Huxley, 1878

/ hate

they are

intellectuals;

I am from

bottom

the

Frank Lloyd

4.1

the

top down;

Wright

(1869-1959)

Introduction

Data structures
are
is like a design of
most
about

effective

the building
a building.

for the

functionality,

intended

efficiency,

blocks of computer algorithms.

One has to
use

not

techniques. Putting
Other ideas
possible.

design of

an

algorithm

must

of the

form,

construction
is

from

up.

and

put

all

the rooms

building. To
beauty.

in midair may
but too
possible,

a room
may

be

One

be based on a

thorough

of an algorithm
that is the

A design

together

do that,
needs

it is

a way

not enough to
desired
In

know

of

knowledge

thorough

achieve the
expensive.
understanding

in

the

of

effect,
same

data

but

it

way, a

structure

techniques and costs.

this short chapter,
we review only the basic data structures used throughout
the
to provide a comprehensivetreatment
We do not intend
this chapter
of data
structures.
That would
are
(at least) a whole book, and indeed, there
require
many
that most readers have already studied
excellent such books. We expect
data
structures
in some depth. This chapteris intended
for quick review.
mostly
In

book.

\302\243^k

Structures

Data

62

useful

notion

in the

we

write a

when

Normally,

is that of an abstract
data type.
study of data structures
program, we have to specify the data type (e.g., integers,

reals, characters). But, in some cases, the data type is not important for the design of the
first-out
(FIFO)
queue of
algorithm. For example, we may want to maintain a first-in
into
the
of
of
items
and
are
insertions
removals
queue,
items. The required
operations
the
items
must
be
in
the
same
removed
order
In
case
of
the queue.
removals,
from
items
the
convenient
and more general to design
which
in
they were inserted. It is more
the data type of the items.
We specify
without
these
for
specifying
operations
algorithms
data type that supports these operations
We call the abstract
operations.
only the required
of
an
abstract
data type is a list of operations
that
The most important
a FIFOqueue.
part
data type is a queue in which
the
Another example of an abstract
we want to support.
have priorities.

items

according to the
highest

priority

queue.

Again,

to

have

are totally

The removals
That

priorities.

the items

among
we

do not

the data

specify

in

specify
of

type

and that

ordered

are

is, the
the

the

first

item

queue.

data

to the
according
to be removed

not

type

the priorities;

but
insertions,
is
item
of
the
step
of

order
each

in

This abstract data

type

is called

a priority

items. (In this case, we do not even

we need only to assume that the priorities

of the

we can determine that

order.)

and not on a precise

structure,
we
make
the
more
The
particular
problem,
design
general.
implementation
a priority
techniques for implementing
queue, for example, are for the most part
our
data type.
If we realize that
needs
to the
independent of the exact
correspond
definition
of the abstract data type,
we can immediately
use it. Abstract
data types allow
us to make the algorithm-design
more modular.
process

on

By concentrating

operational

nature

of a data

for

4.2

the

Data Structures

Elementary

4.2.1 Elements
We

use

the

notion

of

an

element

throughout

this

book as

generic

name

for

an

or
can be an integer, a set of integers,
a file of text,
unspecified data type. An element
data structure. We use this term whenever
the discussion is independent
of the
the algorithm
If the only
type of data. Consider,for example,
sorting
steps
algorithms.
takes are comparing
elements
and moving them around,
then
the same algorithm can be
used for sorting integers or names (strings of characters). The implementation
(that is,
the program) may be slightly different, but the ideas are the same. Since we often
concentrate
on the ideas rather
to ignore the
than
on the implementation, it is reasonable
of
the
elements.
types
The only assumptions
we make about elements are the following:

another

can be comparedfor equality.

a totally ordered set, and it
from
are taken
another.
element is \"less than\"
We usually are
\"less than,\"
definition
of the relation
as long as it is

1.

Elements

2.

Elements

3.

Elements

can

be copied.

is possible
not

concerned

a valid

to tell whether one

with the exact

total order.

4.2 Elementary

Data Structures 63

are counted as taking

one
unit of time. Although
the unit is relative
actual elements, we usually
will
these
regard
operations as taking
time.
Most of the time, it is easier to think
of an element
constant
simply as an integer,
work
for
more
also
even though the algorithm
structures.
may
complicated
these

All

to

the

size

4.2.2

operations
of the

Arrays

An array is

elementsin

a row

of

same type. The sizeof an array

Since the size of the
be fixed.

of the

elements

This size

must

modified

if

of

is the number

and
are of the same type,
the amount
of memory that should
be allocated to
all the elements
are names with
8
store the array is known a priori. For example, if the elements
if
each
character
1
of
if
of
and
the
size
the
is
characters
each,
requires
byte
storage,
array
to store
the array. The storage for an array is always
100, then 800 bytes are required
then the fcth
consecutive. If the first byte of the array is stored at location x in memory,
of the array is stored at location
x + k- 1. Consequently,it is easy to compute the
byte
in the array. In our example,
if the
of the storage of each
element
location
starting
of
the
is
at
then
the
55th
name
starts
at
the
location
433rd
10000,
starting
array
byte,
which
at location 10432, assuming
locations
is stored
are numbered
by bytes. (This
that

calculationcan
can

be
are

Arrays

array

array.

easily
very

and

efficient

be accessed

in

constant

are numbered

locations
very

time.

common

data

The algorithm

array

is fixed,

differently.)
Every element

structures.

designer

who

uses

of an

a high-level

calculations \342\200\224
are done by the compiler.
they
for
As
a rule of thumb, arrays
should
be used whenever possible. The main
drawbacks
cannot
to
elements
of
different
are
their
restrictions.
be
used
store
using arrays
Arrays
We deal with
types (or sizes), and the size of an array cannot be changeddynamically.
these two restrictions
in the following subsections.
language is

4.2.3
Records

rarely

with

concerned

location

Records
are

similar

to arrays,

except

that

we

do not

assume

that

all

elements

are of

the

The exact combination

of different
types.
type.
of types is fixed. Like that of an array, the storage size of a record is known in advance.
Each element
in
a record
can be accessed in constant
time.
This is accomplished by
same
an
with
the
number
of
such
that
for
each element the array
elements,
keeping
array
contains
its starting location. This array
access
is needed
time
only to enable a constant
to any record
element.
Such access is achieved by consulting
of
the array for the location
the element. The exact program
that
maintains
the array is created automatically
by the
same

A record is

thus

a list

of elements

compiler.

For example, a record

more integers,and 2 names

consist
of 2 integers, 3 arrays
of 20 integers
each, 4
may
each containing 12 characters. (Note
that the two array types
in the record
are considered now to be elements by themselves.) This recordis defined
in
with the record contains the starting
stored
relative
locations of all the
Fig. 4.1. The array
elements.
in the
Thus, if each integer is stored in 4 bytes, Int6, which is the ninth element
Since the sizes of all the
record, starts at byte number 261 (2-4 + 3-20-4 + 3-4+1).
to compute the location
it is possible
of each element
in the records are known,
elements

Data Structures

64

recordexample!
begin

: integer

Intl

Int2 : integer;
[1 ..20]

Arl

: array

Ar2

: array

[1..20]

Ar3

: array

[1..20]

lnt3

: integer;

Int4

: integer;

Int5

: integer

Int6

: integer

Namel

: array

Name2

: array

of integer ;
of integer ;
of integer ;

[1..12] of character
[I. .12] of character

end

Figure

in

constant

similarly,

4.2.4

time.

Like that for

it is

not possible to

Linked

Lists

There are many

applications

the algorithm

progresses.

4.1

arrays,

Definition of a record.
the

storage

for a

record is always consecutive;

add elementsdynamically.

as
the number of elementsis changing
dynamically
as arrays (or records)
to define all the elements
to ensure
a good
sufficient storage space. This is often
solution, but, of
large enough
to the worst case (and, in
course, it is not very efficient to demand storage according
there are cases where there is a
cases, the worst case is unknown).
Furthermore,
many
need for insertions
and deletions in the middle of the list.
If we use arrays
and we need to
insert
an element
in the middle, we have to shift all other elements.
This inefficiency is
inherent
in the
insertions
and
consecutive
thus,
representation of arrays;
arbitrary
in which

It

is possible

deletions are very costly for

structures.
We use dynamic
familiarity

with

them

large
data

arrays.
structures

For these cases we

extensively

need

throughout

data

dynamic
this

book;

is essential.

Linked lists are the simplest

form of dynamic data structures. Suppose we have a
elements
and we want
and to delete old elements
to be able to insert
new
elements
The
idea
is
of arrays. Instead,
to
abandon
the
consecutive
efficiently.
representation
each element
is represented
the use of
are
connected
through
separately, and all elements
A
of
another
is simply a variable that
as its value the address
holds
pointers.
pointer
element. A linked
and a pointer, such
list is a list of pairs, each consisting of an element
that
each
contains the address of the next pair. Each such pair is represented by a
pointer
record.
A linked list can be scanned by following the addresses
in the pointers. Such a
\342\200\22
scan
must
be a linear scan. That is, it is not possible to access each element
directly
in
order.
we must traverse the list
list

of

4.2

are

There

more

the

space.
30th

two

major drawbacks to

There is one

additional

the

Data Structures

Elementary

linked-list

First,

representation.

it

Second, if we want
beginning and look at

per element.

pointer

65

requires

to

look

at

we need to start at the

29 pointers,
we could make a simple calculationand find the 30th element
there is one major
to this representation.
advantage
Suppose

for example,

element,

time. With arrays,

directly. On the other hand,
we find the 30th element and we now
want
to insert a new
31st
element.1
All we
that
need to do is to set the pointer associated with the new 31st element to the address of the
is stored in the 30th pointer), and set the 30th pointer
previous 31st element(this address
two
are required. With
to point to the new 31st element (see Fig. 4.2). Only
operations
would need to be moved. A delete
all elements
following the 30th element
arrays,
to delete
the 31st element, we simply
is also simple. If we want
set the 30th
operation
one at a

pointer to point
(see Fig. 4.3).

The
complicated.

the

at

31st

the

pointer

two

details

important

stored

32nd element, by copying

the address
are
Only
operations
required.
in
linked
discussion of insertions
and deletions
to

that

tend

to make

problem is

The main

the

implementation
to

how

detect

lists

the end

of

the

ignored several
lists a little
more

has

linked

of

list.

a special

Usually,

30th
N

ooo

30th
M

Figure

4.2

Inserting

-1

OOO

new

element

into a

>

linked list.

30th
1

\342\200\224
n

ooo

\342\200\224H

30th
\342\200\242I1\342\200\224\342\200\224M
I J

Figure

The

Deleting

\342\200\224H

an element from a linked list.

to talk about
the new 31st and the old 31st
system is consecutive in nature. It is thus confusing
uses \"30a\" to denote an insertion after 30. This notation causes many problems.
If we
(30aa?). This is a good example of the need for dynamic
again after the 30th, we may run out of notation

element.
insert

4.3

OOO

number

We often

data structures.

66

Data Structures

is a pointer
to nowhere;
it can be
introduce a regular
but
record,
that the search will
end
there.
This additional
to include in it a key that will guarantee
makes the program simpler, since there
are
record,
record, sometimescalled a dummy
a
are
for
of
data
structures.
records
useful
cases.
fewer special
variety
Dummy

address,called
used to

4.3

The
they

indicate

nil,

such

is provided,

an end

of a list.

a nil

that

Another

pointer

is to

possibility

Trees
only

structure

represent.

that
There

lists
can capture is
arrays and linked
are numerous applications that
require

represent hierarchical structures.

for certain operations on linear
hierarchical trees, also known

They

can

structures.

also serve
In this

as a

the

more

section, we

will

order

more
efficient
be

of the

structure.
data

concerned

elements
Trees
structure

with only
is a set of

as rooted
trees or arborescences. A rooted
tree
with
a
set
of
that
connect
the
nodes
elements,
(or vertices),together
edges
One
node
the
in
is
root
of
the
tree
of
a
the
elements
(the top
way (see Fig. 4.4).
special
The root is connected to other nodes, which
are at level 1 of the
hierarchy).
hierarchy;
at level 2, and so on. All the connections are thus
they, in turn, are connectedto nodes
and their direct unique
between
nodes
called parents after
\"supervisors\"
(usually
the
root
has
no
The
main
of trees is that
trees).
Only
genealogical
parent.
property
they
do not have cycles. As a result, there is a unique
between
two
nodes of a tree.
path
any
A node
is connected
to its parent
and to several underlings
the
(again,
following
we
will
call
the
latter
The
maximal
number
of
children).
genealogical terminology,
children of any node in the tree is called the degree
of the tree. We usually
order
the
children of every node, then identify
them by their index
in that order (the first,
second,
and so on). In the special
case of trees of degree2, called
we identify the
trees,
binary
children by left (for first) and right
A node with no children
is called a leaf
(for second).
the terminology
comes from real trees). A node that is not a leaf is called an
(this
time,
internal
node.
The height of a tree
level of the tree, namely,
is the maximal
the
which

we call

Figure 4.4

rooted

tree.

Trees

4.3

67

distance
between
the root and a leaf. Each node has a key, which
comes
from a
set
We
ordered
a
number
or
an
will
real
(for
totally
example,
integer).
interchangeably
no confusion
can arise. For convenience,
refer to the key and the node as the same when
assume
that we deal with
we can link together
we
all the
Otherwise,
keys.
unique
list and have one node with
a pointer
to that list.
elements with the same key in a linked
has a data
field containing
the data (or a pointer
to the data) that
Each node usually
is
with the node. The data
field
on the application, and
we
will
associated
depends
generally not deal with it.
In this section, we concentrate on two
uses
of trees: search trees, and
In
heaps.
with
of the representation
trees
are used. We start
a discussion
of trees
both cases, binary
maximal

in memory.

4.3.1
There

are

two

main representations

representation. In
is done by a pointer.
(In

of Trees

Representation

some

the

A node

applications,

of trees, an

representation,

explicit

with

k children

also contains a

a node

representation

implicit

the connection of one

is a record containing
pointer

to

tree
an

its parent.)

and an explicit
to another

node

array

It is

of k
usually

pointers.
more

same type. Hence, all nodes have m pointers,

it is possible to
where m is the maximal
number
of children in the tree. Alternately,
associateonly two pointers per node in the following way. The first pointer points to the
the two
to the next sibling. Figure 4.5 illustrates
first
and the second pointer
child,
points
is that,
of the same tree. The main drawback of the second representation
representations
list.
to get hold of all the children
of a node, we have to traverse a linked
An array is used to store all the
No pointers are used in the implicit
representation.
of the nodes in the
nodes
the connections
are implied by the positions
of the tree, and
tree
an implicit
The most common way
of implementing
representation is the
array.
a binary tree T. The root of T is stored in y4 [1]. The left and right
Consider
following.
of the left child of the
children
of the root are stored in A [2] and A [3]; the two children
of the tree
root are stored in A [4] and A [5]; and so on. The array represents a traversal
convenient

to have all

the

nodes

Figure 4.5

Binary

of the

representation

of a nonbinary

tree.

68

Data Structures

from

left

level

to right,

root is storedat
is stored

in

A [1]

A[2i],

can define

We

level.

by

the

representation

node v

(the base case). (2) The left child of a

and v's right child is stored in >4[2/+l].

(1) The

by induction:
that is stored

The

in

A [i]

of

advantage

this

saves
On the other hand,
if
no pointers are required, which
storage.
representation
much
from
if
some
leaves
are
farther
the
root
than
is
unbalanced,
away
namely,
the tree
nodes must be represented. An unbalanced
tree is
others are, then
nonexisting
many
in
node
indicate
its
the
An
below
each
shown in Fig. 4.6; the numbers
position
array.
or
8 nodes. The implicit representation thus
may
array of size 30 is neededto represent

is that

may

not save

operations

storage, depending

the

in

can be reasonably

on

of the

middle

if they

supported

Also, since

tree.

the

tree are costly.

other

the

On

are limited to nodes that

are
used,
dynamic
hand, dynamic operations
to the end of the
correspond
arrays

array.

4.3.2

Heaps

A heap is a binary

tree

whose

keys satisfy

node is greater

The key of every

the

than

property:

heap

following

to

or equal

the

of

of any

key

its

children.

By the transitivity
or equal
to

law,

than

a priority

implementing
two

the

the

of

all

queue,

which

keys

node's

that

is an

data

abstract

type

in

by the following

defined

operations:

insert a key x

Insert(x):

Remove{)\\ remove the

can be

Heaps
will

of every node is greater

descendants. Heaps are useful

that the key

implies

property

heap

use

into

largest

the

structure.

data

key from

the data

structure.

implemented with either the explicit or the

representation, since we can ensure

implicit

the implicit

We assume

elements

the

the

that

heap

is
array
will ever

Figure 4.6

i4[l..\302\243],

where

contain (if

Implicit

an

representation

k is an
upper

upper

bound

of an

tree

representation.

the heaps

that

bound

is not

be

will

on
known,

unbalanced tree.

the
then

We

balanced.

number

of

a linked

4.3 Trees 69
Let n denote
required).
A
[\\..n
] is of interest
array

is

representation

only
how to implement insert
namely,

the

We
key

largest

from

The

root.

the

first

[n ],

:= A

We

the leaf

take

two

have

still

decrement

and

heaps

separate

we

property,

now

jc

propagate

This is done by

maximum.

the

proceed

the heap;
to describe

heaps.
the node

property,

heap

in

always

operation

the

with

removes

the key

the

property

heap

on top, which
or may not satisfy
the
may
the
at
this
is
if
the
satisfy
heap property
point
was
contained
the value jc.) To restore
the heap
a value

plus

(The only way

property.
path from the root to where

whole

By

use of

now

after the key of the root has

have
to two separate heaps.
an array A [2..n ], which
corresponds
it in place of the
root.
That is, we let A [1]
[n ], delete it, and put
the value of n by one.
Denote the value of the new A [1] by jc.
for

heap

the

with

A[\\]. So, a Remove

is to restore

problem

We now

been deleted.
We

efficiently

operation.

is the root,

a heap

in

Remove

the

with

start

remove

and

We

moment.

at any

of elements

number

current

the

jc
jc

to

the tree,

down

jc

comparing

a subtree for

it reaches

until

with

the

values

of

its

which

children

two

it is

04 [2]

and

A [1] with the largest

and, if jc is not the maximal among the three,
by exchanging
of them. Assume that A [2] is the maximal. Then,A [2] is clearly the maximal key in the
rooted at A [3]
whole heap, so it can be put in the root position.
Furthermore, the subtree

A [3])

remains unchanged,
the subtree

about

continue
key

jc

again

thus

rooted

at

inductively

is now

if

jc

is

in

the

it also
A [2]

same

satisfies the

now it
Assume that

(because
way.

We have to

property.

heap
has
we

jc

in

its root).

But

only

worry

now

we

for / steps, and that

satisfy the heap property.

continue

Only the tree rootedatA[j] may not

jc to
its two new children, A[2j] and
terminates
not the maximal. The algorithm

atA[j].

compare

exchange

and

A[2j+\\]

either

(if they
when jc

exist),
becomes

can
the

We
and

the

number
of comparisons
a leaf. The maximal
of a subtree,or when
it reaches
tree.
The algorithm
the
of
the
for
which
is
twice
a deletionis 2|\"log2\302\253~|,
required
height
for removing
a heap is given in Fig. 4.7.
a maximum element from
the new key
n by one, and insert
An Insert operation is similar. We first increment
as the new leaf A [n ]. We then compare
and exchange if the
the new leaf with
its parent,
new
of its subtree
the new key is the maximal
leaf is larger than
its parent.
At this point,
it was
found to be larger). We
was the maximal and
(since the parent
assume,
and
A
the
tree
satisfies
that
the
rooted
condition,
[n ])
heap
?xA[j] (initially
inductively,
that if we remove
We
continue
the
this tree the rest of the heap satisfies
heap
property.
this
its
the new key up the tree, until the new key is not larger than
process,
promoting
The
it reaches
the root). At this point, the whole tree is a valid
parent
(or until
heap.
maximal number of comparisonsrequired
is riog2\302\253l, which is the height
for an insertion
of the tree. The algorithm for inserting
in Fig. 4.8.
into a heap is given
an element
in time
of
and
Insert
Remove
Overall, we can perform
sequence
operations
any
O (\\ogn)
On
is
the
other
it
other
not
to
hand,
per operation.
possible
operations
perform
a heap. For example, if we want
with
to search
for a given key, the hierarchy
efficiently
A Heap is a good exampleof an implementation
of an
given by the heap is not useful.
a limited
abstract data type. A heap supports
number of specific operations
very
we need these particular
we can impose the
Whenever
operations,
efficiently.
heap
structure
on the data whatever its type is.

maximal

70

Data Structures

(A, n);

Remove_MaxJrom_Heap

Algorithm

Input:
Output:

of size

array

(an

maximal

Top_of_the_Heap (the

heap), and

new

(the

begin
ifn = 0 then

size of

the

is

\"the heap

print

a heap).
element

representing

if n

heap;

of the

= 0, then

heap), A

the

heap

empty\"

else

:=A[\\];

Top_of_the_Heap

A[\\]:=A[n];
n := n -1 ;

parent :=

1;

2 ;

:=

child

-1 do

while child <n

ifA[child]

child := child + /

;
then

> A[parent]

ifA[child]

swap(A[parent],
parent := child;

child := 2*child
child :=

else

then

< A[child+J]

[child])

the loop

stop

{to

end
Figure 4.7 Algorithm
_Heap (A,n,x);

Insert Jo

Algorithm

Input: A
Output:

of size

(an

RemoveJAaxJromJieap.

array
A (the new

heap),

representing
and

n (the

a heap), and x (a
new size of the heap).

number).

begin

n :=

A [n ] :=

while

assume that

the

array

does not

overflow

x;

child := n
parent

; { we

+ 1

:= n div 2
parent

> 1

;
do

if A[parent] < A[child]

A[child]);

[parent],

swap(A
child

then

:=

parent

{ see alsoExercise4.6}

parent := parent div

else parent := 0 {to stop

2 ;
the

loop }

end

Figure 4.8

Algorithm

Insert

jojieap.

(the

new

is empty).

4.3

Binary Search Trees

4.3.3

trees

search

Binary

implement

there (for simplicity,

the

delete(x):deletethe
data

Abstract

insertions

holds

(or to

nil).

are

leaves
node

be removed

may

simplicity,

Binary

section, since dynamic

We do not want to
trees.
parts
binary
of elements.
We assume that each
for the number
at least three fields: key,
left, and right, such that
are pointers
to other nodes
node, and left and right
in this

search

of

may be

pointers

that all keys

assume

we

the

and

dictionaries.

called

as other more complicated

as well

a given upper bound

tree is a record
containing
the key associated with the
search trees are more complicatedthan heaps,
Binary
whereas in
added or removed and keys exchanged,

in the

already there).

is there.

if it

structure

efficiently,

limit ourselves to

key

it is

(unless

these three operationsare

are important

deletions

and

structure

explicit representation of trees

use the

will

We

operations.

structure, or determine that x is not

that each key appears at most once).

the data

dictionaries

implement

operations:

following

the data

x from

key

that handle

types

search trees

x into

key

data

the

in

will assume

we

insert

insert(x):

the

efficiently

find the key x

search(x)\\

node

71

Trees

binary

in

manipulated

in heaps
trees,
other ways.

because

only

search

many

any

For

are distinct.

Search
tree
is a structure to facilitate searching. The structure
search
that we have a key x
Assume
search procedureis understood.
and we want to know
whether
it is currently a key of a node in the tree, and if it is, we
x against the
want to find that node. This operation is called a search. We first compare
=
If
x
root
we
of the tree, whose value
If
x
then
are
done.
r.
<r, then we
is, say,
r,

name suggests,a

As its

once the

becomes

clear

continue

the search from

Each

child.

in the

keys

it.

keys satisfy

this

tree is presented in

than

the

continue

the

range

search

of the

from the

keys below

right

it:

the

the keys in the right subtree are all

say that the tree is consistent if all the

and

it,

search trees.

We

recursive program for searching

A simple

condition.

divide

tree serves to

rule defines

This

we

otherwise,

child;

all smaller

are

subtree

left

greater than

search

the

in

key

left

the

in

search

a binary

4.9.

Fig.

Insertion

Insertions
for x is

into

is already
performed
(We assume that we
If x

first.

be aborted.

will

Otherwise,
be

then

value

the

that

which

we

ends

search

The tree

find

is given

remains

the
in

leaf.
Fig.

4.10.

in

do

the

Given

simple.

then

tree,

it

will

not want several

at a

(as either a

consistent,

the new
We use this

it to

through

also quite

(unsuccessfully)

below that leaf

inserted

of x).

same leaf and

so

trees are

search

binary

right

since

leaf. A
child

nodes

with

containing

the

a search
insertion

same

key.)
the new key can

child, depending on the

searches for a will get to the

or a left

subsequent

node. The search

opportunity

node

a key x to insert,
be found and the

must be changed slightly

nonrecursive search program,

program

to write a

72

Data Structures

Input: root (a pointer

to a

(a pointer to

node

Output:

x) ;
root of a

(root,

BST_Search

Algorithm

search

binary

node

the

and x

tree),

the key

containing

jc,

(a number).

or nil

if no such

node exists).
begin

if root

= nil or

{root\"

is the

that

record

else
ifx < root\".key

then

the

:= root

then node

= x

root\".key

is pointing

root

pointer

x)

BST_Search(root\".left,

else BST_Search(root\".right,

to.}

x)

end

4.9

Figure

Input:

root

(a pointer

to a root

tree is

changed by

The

Output:
the

(root, x);

BSTJnsert

Algorithm

child;

pointer

BST_search.

Algorithm

search

inserting

a node

and x

tree),

the

with

a node with

is already

there

if

of a binary

key

begin

if root = nil

then

node pointedto

a new

create

by

child

child ;

root :=

:=

root\".key

else

node :=
:=

child

while

root;
{ to

root;

node *

nil

and
= x

if node\".key

that it is

it so

initialize

not

nil}

* nil do

child

then child

:=

nil

else

parent

:= node

then
ifx < node\".key
else node := node\".right

if child

create

nil

:= node\".left

node
;

then

a new

node pointedto
;

by

child

child\".key := x
child\".left

:=

nil;

ifx < parent\".key

child>ight
then

else parent\".right

:= nil;

parentleft
:= child

:= child

end

Figure 4.10

Algorithm

BSTJnsert.

jc,

(a number).

key x
then

pointed to

child

= nil.

by

Trees

4.3

73

Deletion
It is easy to delete a leaf;
we need
complicated.
be
to
nil.
It
also
not
is
hard
to
delete
a
node
that
has
pointer
change
to
the
is
the
node
to
to
its
child.
if
the
child;
However,
pointer
changed point
has
two
then
we
need
to
delete
find
a
for
to
the
two
want
children,
place
pointers.

Deletionsare

more

generally

to it

the

only to
only one
node

we
Let B

be a node with two children whose key we want to delete (see Fig. 4.11). In the first step,
we exchange the key of B with a key of another node X, such that (1) X has at most one
child, and (2) deleting X (after the exchange) will leave the tree consistent. In the second
now has the key of B which
we wanted
to delete. We can easily
step, we delete X, which
the consistency
of the tree, the
delete
X, because it has at most one child. To preserve
as
as
all
the
X
must
be
at least
in
the
left
subtree
of
of
B, and must be
large
keys
key
the
all
subtree
of
B.
Notice
that
the
of
X in Fig. 4.11
than
the
in
smaller
right
keys
key
satisfies these constraints: it is the largest among the keys in the left subtree of B. X is
X cannot have a right
of B in the tree.
it
since otherwise
called the predecessor
child,
The deletion algorithm
not have the largest key in that subtree.
in Fig.
is presented
would

4.12.

Complexity
the tree
take

The

the

and

us all the

constant

time

to

way

of search,

of the relevant
the

bottom.

the actual

(e.g.,

worst-case running

times

running

location

node.

In

All the other

insertion,

the

and

insert,

the

steps

on the shape of
the
search
would
case,
path

delete

worst
in

the

depend

algorithms

of keys in the
exchange
of a path from the
root

require only
So, the

deletion).

is
maximal length
to a leaf, which
tree is reasonably balanced(we will define balance shortly),
then its height is approximately
log2fl, where n is the number of nodes in the tree. All the
If the tree is unbalanced, then these operations
case.
are
are
efficient
this
in
operations
much less efficient.
If
a binary
the
search tree in a random
order, then the
keys are inserted into
\342\200\224
more
2ln n. In this case, the search
expected
height of the tree is 0(\\ogn)
precisely,
and
insert
of the tree can
are efficient. In the worst case, however, the height
operations
be n (when the tree is a simple linked list). Trees with long paths can result, for example,
from insertions
in a sorted, or close to sorted,order. Also,
deletions
may cause problems
even if they occur in a random order. The main reason for that is the asymmetry
of
the
If
to
a
deleted
node.
there
are
always using
predecessor
replace
frequent deletions,

the height

of

the

is the

time

tree.

If the

\302\243

Figure

4.11

Deleting

a node

with

two

children.

74

Data Structures

root (a pointer

Input:

The

Output:
(

to a

tree is

changed by

a binary

search

a node

deleting

never deleted,and

that the root is

assume

We

x);
root of

(root,

BST_Delete

Algorithm

and x

tree),
the

with

that

(a number).

key x,

all keys

if

it exists.

are

distinct

begin

:= root

node

;
and

* nil

node

while

:= node

parent

ifx < node\".key

:-

node

else

if node

if node

* root

if

then

node

node\".right

then

nil

is not

print(\"x

parent\".right

else if node\".right
ifx < parent\".key

the

tree\")

; halt

then

else parent\".right
two

node! :=

\".right

:= node\".right
- nil then
:= node\".left

parent\".left

:=

node\".left

case

children

node\".left;

parent! := node
nodel

parent I

do

* nil

\".right

:= node!;
\".right ;

nodel

:=

nodel

{ now

in

then

:= node

parent\".left

while

node\".left

then

= nil

ifx < parent\".key

else {the

:-

then

node\".left

else

*xdo

node\".key

the actual

comes

:=

parentl\".right

node\".key :=

deletion }

nodel

\".left;

nodel \".key

end

4.12

Figure

followedby
and

deletions.

insertions,

This

have a height
of O
can be avoided if,

the tree may

asymmetry

Algorithm BSTjdelete.

(\\\302\253),even

instead

of

for random

always

insertions

choosing

the

deleted node, we alternate

and the successor
between
the predecessor
is a smallest key in the right subtree).
are
to prevent
the
(which
there
ways
Fortunately,
creation of long paths in binary search trees. We describe
one such method in the next
predecessor

section.

of the

4.3 Trees

4.3.4

75

Trees

AVL

trees (named after

AdePson-VePskii
and Landis
[1962]) were the first data
to guarantee 0(\\ogn) running
time
for search, insert, and delete
in the worst
case (n is the number of elements). The main idea in AVL trees (and in most other tree
time
when inserting and
structures that achieve
bounds) is to spend additional
logarithmic
to
the
such
that
the
of
balance
the
tree
is
bounded
tree,
deleting
height
always
by
The time devoted to balancing
must not exceed 0(\\ogn), or else insertions
0(\\ogn).
will be too expensive. The idea is to define balance in such a way that it is
and deletions
easy to maintain.
AVL

structures

Definition:

of

This

definition

an

a maximal

an AVL tree

h of

height

Proof: Left
theorem

This

as an

We

property.

every node,

tree

such that, for

and

right subtrees is

at

most

as 0).

of O

height

(logn),

as is

in the

shown

next theorem.

<

1.4404

with

start

key

log2(>z+ 2) -0.328.

exercise.

that search in
to perform insertions

Let x be a new
bottom of the tree in

nodes satisfies

n internal

with

implies

is how

problem

left

its

4.1

Theorem
\342\226\241

The

empty

guarantees

search

binary

the heights of
tree is defined

between

(the height

is a

tree

AVL

An

difference

the

\342\226\2

AVL

an

and

insertions;
that

the usual

again,
we wish to
way.

we assume that
into

insert

0(\\ogn) comparisons. The

still to
the keys

all

tree.

an AVL

the insertion,

after

If,

tree requires
and

deletions

the

tree

First,

AVL

the

maintain

are distinct.
we

remains

insert

x at the

an AVL

tree,

Otherwise, we need to rebalance the tree. There are four possibilities

\342\200\224
are symmetric
two of them are illustrated
in Fig. 4.13; the other two
(to the right).
In part (a) of Fig. 4.13, the new
node was inserted into the left subtree, making the
we
this unbalance,
of B equal to h +2, whereas
the height of C is h. To remedy
height
to
the
We move B to the top and change the rest of the tree according
perform a rotation:
then

we

binary

h + 2,

are done.

search
which

(Fig. 4.14).
same as the height

property
is the

The

of the new subtree, rooted at B,

original subtree before the insertion.
rotation
is called a single rotation. It

height

is now
As a

of the

result, no more balancing is required. This

help in part (b) of Fig. 4.13; a doublerotation

subtree has the same height as the original

not

the new
(Fig. 4.15). Again,
so no more balancing is required. An
rotation
(single or double) is always

is required
one,

trees is that one

important
property
sufficient
after an insertion. We omit
the proof.
The node A in both examples
is called
of AVL

will

the
critical
node.
It is the root of the
subtree as a result
of the insertion.
To perform
the
critical
node
find
the
and
we
to
to
have
determine
of
which
the
cases is
insertion,
in each node a balance factor,
is equal to the difference
involved. We maintain
which
For AVL trees, the
between
the heights of the left and right subtrees of this node.

smallest subtree

that

becomes

a non-AVL

Data

76

Structures

h-I

h-l
/\302\273+/

new

new

(a)

4.13

Figure

balance factor of
rebalancing if
of

the

(b)

Insertions

node

each

balance

that

invalidate

1, -1, or 0. An
was either 1 or -1, and

is

factor

the AVL

insertion
the

property.
a subtree

into

increases

insertion

requires
the height

in the ''wrong\" direction. That

a
that the critical node must
have
implies
balance factor. Moreover,if a lower node has a nonzero balancefactor,
then,
the
the heights
from
be the same as they
were
before
that node will
balancing,

a subtree

nonzero
after

h-l

h+1

h+l
h-I

new

(b)

(a)

Figure

4.14

single

rotation:

(a) Before,

(b) After.

4.3

77

Trees

new

(b)

(a)

Figure 4.15

double

rotation:

(a) Before,

(b) After.

insertion
the old heights from the critical
Hence,
node).
(recall that the balancing retains
the critical node is the lowest ancestor of the new node with nonzero balance factor. On
the
one.
down the tree, we look at the balance
factors, remembering the last nonzero
way
to the \"right\"
or
When we reach the leaf, we can easily determine whether
we insert
\342\200\22
or
down
bottom
We
then
another
direction.
make
top
up
pass (either
\"wrong\"
less nodes), readjust the balance
bottom
involves
factors,
preferably
up, since that usually
and perform a rotation
if necessary.
We omit the details.
It is no longer true that the tree can be
more
Deletions
are, as usual,
complicated.
There are cases
after
a deletion.
rebalancedwith only one single or double rotation
where
rotations
are required, where n is the number of nodes in the tree.
0(\\ogn)
number of steps; thus, the worst-case
each rotation
Fortunately,
requires
only constant
time
of
a
deletion
is
still
we omit the details.
O
running
(log n). Again,

Comments
worst
in

the

trees

AVL

case, requiring
case.
average

at

most

form
45%

Empirical

an efficient data

structure.

more comparisons

studies have

They

than

shown

optimal
the

average

perform
trees,

well

in

the

and even better

search

time to be

main
[1973], pp. 460]). The
and the
factors,
disadvantages
storage for the balance
fact that the program that implements them is rather
complicated.
Many other schemes
been
for balanced-search trees have
2-3
trees, B-trees, weightproposed,
including
balance
trees, and red-black trees.
approximately

log2fl+0.25

of AVL

comparisons

trees are the

need

(see

for extra

Knuth

78
4.4

Structures

Data

Hashing
is

Hashing

algorithms.
can also be
for

structure

one

of the

most (if

not

Hashing is used mainly

used for
storing

The

deletions.

data

with

keys

the

most)

idea behind

numbered

structures

is

hashing

from 1 to

computer
of

it

a data

Designing
The data can be stored

simple.

is easy:

for

some variations

/. Any key can thus

be accessed
1
to
If there are n unique
In, for example,then it is still
keys in the range
immediately.
in an array of size 2w, even
the storage utilization
is
though
usually best to store them
it is usually worth
that
the extra
now only 50 percent. The accessis so efficient
space.
However, if the keys are integers, say, in the range 1 to M, where M is the maximal
to allocate
that can be representedin the particular
computer, we cannot afford
integer
if
there
students
identified
their
For
are
250
M.
size
of
example,
by
social-security
space
to store
information
about them
number, we will not allocate an array of size 1 billion
in an

array of

size n,

such

that

key / is

stored at

data

useful

and searches, and

insertions

for

location

numbers). Instead, we can use the last three

case we need only an array of size 1000. This is not a
There may be students
with
the same last three digits
with
method.
(in fact,
foolproof
of that is quite
will
the probability
We
show
how
to
such
handle
250 students,
high).
We can also use the last four digits, or the last three digits and the
first
duplicates shortly.
even
further.
letter of the student's name, to minimize
more
However, using
duplicates
utilization.
digits requires a larger-size table and results in a smaller
We assume that we are given a set of n keys taken from a large set U of size M,
such
M is much larger than
that
n. We want to store the
in a table of size m, such
keys
that m is not much
than
a
n.
The
ideais
to
use
called
a hash function,
to
function,
larger
the
which
in
are
the
1
in
1
to
to
new
the
to
so
we
can
M,
m,
map
keys,
range
keys
range
store everything
in an array
of size m. Taking
the last three digits of a large
is
integer
such a function. It maps
a large set U of size 1 billion
to a set of size 1000. Each
to store
possible
key is thus given a place (index) in a table of size m. We will attempt
the
in that particular place in the table.
If the function is easy to compute, then
key
the key is also easy. However,
since
the set U is large and the table is small,
accessing
no matter
what
function
we use, many
will
be mapped into the same place in the
keys
table. When two keys are mapped to the same location in the table, we call it a collision.
We are thus
faced
with two problems: (1) finding
function that minimizes the
a hash
likelihood
of collisions, and (2) handling
collisions.
Even though the set U is much
the actual set of
than the size of the table,
larger
is usually
should map the keys
not too large. A good
hash function
keys we handle
in the table. Of course, no hash function
sets of keys
can map all possible
uniformly
without
collisions.
If the size of U is M and
table
is w, then there
the size of the hash
must
be at least M/m keys that
are
is
into the same place. If the mapping
mapped
Hash
it.
uniform, each location will have
M/m
into
mapped
approximately
keys
functions should transform
a set of keys uniformly
to a set of random locationsin the
1 to m.
The uniformity and randomness are the essence
in hashing.
For example,
range
instead of taking the last three digits of the social-security
we could take the last
number,
three digits of the student's
hash function,
year of birth. It is clear that this is an inferior

(there are

digits of

the

billion

numbers,

possible

social-security

in which

79

4.4 Hashing

since it

more likely that many students

same
last three digits of

is much

many students have the

same year

in the

social-security

the

size

than

it is

that

number.

Functions

Hash

We assumethat

the

the table

cannot

size that

is

keys

function

hash

effective

and

were born
the

be

easily

adjusted

a power

of 2, for

) mod m,

= (xmodp

are integers, and that

is h (x) = x mod m,

to be

a prime

example),then

of the

where m
(it
the

is a

hash

prime

is convenient

hash

following

is m. A simple
table
number. If the size of
sometimes to have a
function can be used:
be sufficiently
larger

a prime, and p>m

(p should
smaller
than | U |).
than m to be effective,but it should also be sufficiently
function
As
we have already mentioned, no hash
can be good for all inputs.
Using
since most data in practice
as described is fairly
have no structure related to
safe,
primes
it is always
numbers.
On the other hand,
that, in a
possible (although
unlikely)
prime
on integers
made
certain application, one will want to store results of some experiments
r. All these numbers of course will
are of the form r + kp for a constant
have
all of which
data
the same hash values if p is usedas described.We can take the idea of scrambling
function!
For
with hashing one step further,
and use a random procedure to selecta hash
from
list
in
the
be
selected
at
random
of
the
can
a
prime p
primes
appropriate
example,
list of primes, however, is not easy.
Another
is the
a large
possibility
range. Finding
a
let
At
select
two
numbers
a
such
and
and
that
and
/?,
a,b
*0,
random,
<p,
following:
=
the
is more complicated to compute than
[ax + bmodp]modm. This function
h(x)
for all inputs.
one is, but it has the advantage that it is very good on the average
previous
Of course, the same hash function must be used for all accesses to the same table. In
tables, or tables that are
cases,
however, there is a need for many
independent
many
function
can be used
In those
hash
created
and destroyed frequently.
cases, a different
is
The
functions
described
above have
time
random
hash
a
different
table
created.
every
h

(x)

where

p is

certain other desirableproperties.

Handling Collisions
The

way

simplest
in

entry

hashed

the
into

that

called
collisions is to use a method
chaining.
separate
all the keys
serves as a head of a linked list containing
search
To access a key, we hash
it and then perform linear

to handle

hash table
entry.

Each
that are
on the

of the list (but the

appropriate linked list. A new key can be inserted into the beginning
list must be searched to ensure
A search may be
that the key is not
a duplicate).
inefficient
if some
lists are long. The lists will be long if the size of the table is small
is
is bad. Thus, hashing
compared to the actual number of keys or if the hash function
not a good dynamic structure.
on the number of
to have a good estimate
It is important
with separate
keys. The main problem
dynamic memory
chaining is that it requires
allocation and more space for the pointers (even if the number
of keys is not too large,
and the pointers are not used).
On the other hand,
if for some reason the estimate
of the
will still work, whereas other static
separate
appropriate table size is wrong,
hashing
methods

will

fail.

Another
are

no pointers.

simple method is linear probing. The

The

hash

function

determines

size

the place

of the

of

the

table is fixed, and there

in the table. If that

key

80

Data

Structures

then
the first empty place after
it is
place is already occupied, that is, if a collision occurs,
is
table
considered
the
the
same
A
for
follows
search
(The
instead.
procedure.
key
taken
order; if the last place is reached and it is full, then the first place is considered
in a cyclic
is
thus ends at the first empty place. When the table
search
unsuccessful
An
next.)
the
is
there
will
works
well.
If
table
method
this
relatively full,
simple
relatively empty,
that are caused by keys with different
are collisions
collisions, which
be many secondary
with
that have the same hash
avoid collisions
function,
hash values. We cannot
keys
such keys are mapped into the same place. We should, however, try to minimize
because
Suppose that the /th place is full and that
secondary collisions. Let's look at an example.
A new key, which is mapped to /, will
cause
a collision, and
the (/ + l)th place is empty.
the collision
is resolved with
This case is efficient,
since
into /+1.
be inserted
will
to / + 1, there will be a secondary
effort.
However, if a new key is now mapped
minimal
it
is
not
full
become
full
collision and /' + 2 will
(if
already).
Any new key mapped to /, to
/ + 1, or to /+ 2 will not only encounter secondary collisions, but will also increase the
effect
is called
This
size of the full segment, causing more secondary collisionslater.
clustering. When the table is almost full, the number of secondary collisions with linear
to linear search.
will be very high, and the search will degrade
probing
with
linear probing. If an insertion
Deletions cannot be implemented
efficiently
if
on
its
to
then a
a
an
and
that key is later
deleted,
key
way
empty slot,
4'passes\"
through
will
future search
be unsuccessful,
since it will stop in the new empty
slot.
If deletions
are required, we must have a collision-resolution
scheme using pointers.
The clustering
effect can be reduced with double
When a collision
hashing.
occurs, a second hash value h2{x) is computed. Instead of searchingin a linear order,
namely, / +1, /' + 2, and so on, we searchthe places i + h2{x), i + 2h2{x),and so on (all in
a cyclic order). When
another
will be
key y is mapped to, say, / + h2(x), the next attempt
at
i + h2(x) + h2(y), instead of at i + 2h2(x).
of h2(y),
then
If h2{x) is independent
is
eliminated. We must be careful, however, to choose the second hash value
clustering
such that the sequence
/ + h2{x), i + 2h2(x)...,i + nh2(x)
the whole table (which
spans
will happen
if the numbers h2{x) and n are relatively prime).
The main disadvantage
of double hashing is that
it requires
more computation
of a second hash
for
the
One
to save extra
search.
(namely, the selection
value)
way
is
to
select
a
second
hash
is
not
of the
value
that
computation
completely independent
first hash value, but that still reduces clustering. One such method
is to set h2{x) = 1 if
= 0, and
that m is prime and that
assume
h\\(x)
h2(x) = m-hx{x) otherwise
(we

=xmodm).

h{(x)

The

4.5
The
the

union-find
use

Union-Find

(also
problem
of nonstraightforward

Problem
known as the
data structures

problem)

equivalence
to

is the following. There are n elements

groups. Initially, each element is in a group by itself.
of operations
and the groups in an arbitrary
performed on the elements

The

problem
divided
into

is a good

example of

the efficiency of algorithms.

jc,, jc2, ...,xn. The elements are

improve

There are two

order:

kinds

4.5 The

find

(/):

returns

union 04, B): combines

unique name (any name
The

goal

is to design

operations as efficiently
is

group

A with

distinct

from

an

ahead

X[\\..n]

array

the problem is to store the

A find operation is thus
trivial

solving
X[i].

that
takes more time. Assume
is necessary to change all the

We

now

xt

group B to form
names

will

do).

will support

any

sequence

a new

the other

that

structure

elements are known

to allocate

possible

data

that contains

group

group

with

these

of

two

as possible.

Since all the

it

of the

the name

81

Union-FindProblem

present

union(/l,
entries

a different

identity
\342\200\224
we

of time

(and

for them.
of the

group

this

the

containing

B) results in a combined
Bio A.
containing
problem.

indexed

from 1 to

n),

method

of

straightforward

look at the

simply

approach to

are

they

The

array.

/th element

A union

group called
Instead

in

operation
A.

of making

it

Then,

the

find

we make the union operation

We use indirect
simple.
addressing.
is a record with
the
of the element and a pointer
to
array
identity
all pointers
are nil. We perform the operation
another record. Initially
union(/l,
B) by
in the record for B to point
to the record containing ,4, or vice versa
changing the pointer
this
is a set of
will
discuss
choice
After
several
unions, the data structure
(we
shortly).
trees as in Fig. 4.16. Each tree corresponds to a group,
and each node corresponds to an
To find the
The element at the root of each tree serves
element.
as the name of the group.
we reach
the
that
element G, we follow the path from G's pointer until
contains
group
is a record whose pointer
is nil. This process is similar to someone changing
root, which
\342\200\224
to leave a forwarding
of notifying everyone, it is simpler
address.
addresses
instead
Of course, finding the right
is more difficult now, namely, the find operations
are
address
less efficient. They are especiallyinefficient
if the union operations form tall trees.
The idea behind the efficient
union-find
data structure is to balance and collapse
effort
the trees.
is
to expend additional
to
We have already seen that it
worthwhile
balance the data structure.
Consider
B) in Fig. 4.16. We have two possibilities.
union(/l,
We can set ZTs pointer
to point
to B. It is clear
to point
to /I, or we can set /Ts pointer
in
is formalized
the
that the first
idea
leads
to a more balanced tree. This
option
simple,

operation
Each

entry

in

the

nil

nil

11

Figure

4.16

The representation for

the

union-find

problem.

Data Structures

82

of

the

store

We

way.

following

group,

when a

Balancing:

group is set to
The

arbitrarily).

the

appropriate

shown

as is
log2\302\253,

size

of the

utilizes

the

in

the name

only

the

the

combined group

is also

the smaller

of

pointer

(ties are

group

larger

broken

and placed

computed

in

root.

the

in

of

record

not

it.

is performed,

operation

the

to

field

union operation

If the

union

point

a root

that corresponds to

record

each

with

the number of elementsin

but also

of

the height

then

balancing,

more

is never

trees

the

than

theorem.

following

4.2

Theorem
\342\226\241

// balancing is

used, then

tree

any

of height

at least

contain

must

2h

elements.

on the number of union

The theorem
The proof is by induction
operations.
results
in a tree of height
1 with two elements.
is clearly true for the first union, which
that A is the larger
so that B will point to A.
Consider union(/4,
B)y and assume
group,
Proof:

Denote

and

h(A)

by

elements;

twice as

hence,

ZTs

find

operations, such that

It is possible to improve

union

final

At that point,
final destination,

destination.

stations

about

traverse

the pointers from

path to

point

Traversing

the

complexity
time a find
every

of a

both

steps

the

log2\302\253pointers.

would

be

so

they

that

If several

analogy.

address to

a good idea
can forward

until

another

to

the

all

notify

changes of
it reaches the
forwarding

the mail directly.

After

we

on

the

number

is performed.
the worst case

of steps;

The

therefore,

the

We can use path

the same.

remains

which

theorem,

following

asymptotic
compression

we

will

not

complexity.

4.3

balancing
in

operation

operation

\342\226\241
Theorem

//

it

only doubles

find

gives a good bound

than

\342\226\2

a recordto the root of its tree, we changethose pointers

on the
the root (see Fig. 4.17). This
is called
path compression.

again

path

time

prove,

to

directly
the

mail

the

its

time. Consequently, any sequence

of m either find
>ny takes at most O (m \\ogn)
steps.
the efficiency
of the union-find
with the
data
structure

again the mail-forwarding

will go from one

Consider

idea.

following

and

is satisfied.

more

follows

never

operation

than ,4),

smaller

be

constant

take

addressesoccur, then

(since B was assumedto

Again, the theorem

height.

original

that a

4.2 implies
union operation always

Theorem

or

obviously

as ZTs tree

more than

is one

height

tree

combined

theorem

the

the trees corresponding to groups A and By

tree is the maximum
of h{A) and h(B)+ 1. If
more
has the same height
as A's tree with
even
holds. Otherwise, the combined
tree has at least
of

heights

the combined

of

elements

many

the

h(B)

The height
respectively.
is
h(A)
larger, then the

worst

and

path compression
case for any sequence

is 0(m\\og*n),
union)
defined below.

where log*n is

are used, then

ofm>n
the

the

operations
iterated

total

number

(either

logarithm

find

of
or

function,

\342\226\2

4.6

(b)

(a)

Figure 4.17
function

The

\\og*n

log*\302\253is

defined

1 +log*([log2\302\253]).

= 1+log*

log* 60000
all

virtually

find

operation

0(n

\\og*n)

is almost

finds

We

as follows,

log*

(b)

1 =

4.

For

any

we have

number

such
\302\253,

<5.

log*2 =

\\ogn

linear (and is linear

that

for this

any

in

each

O(n)

2,

analysis,

an

open

sequence
particular

of them require
involves
which

step separately.

problem is still

n >

log*4 = 3, and
a?<265536, which covers

the complexity of any

Notice
that one
practice).

still require 0{\\ogn) steps, but, overall,

This is an excellent example of amortized
bounding

1. For

14 = 1 +

Thus,

may
steps.

After.

example, log*4 = 1+log*2= 2, log*

than
counting all steps together rather
time
possible to design a linear
algorithm

4.6

(a) Before,

compression:

recursively

purposes,

practical

and

of unions

Path

For

16 =

83

Graphs

Whether

it

is

problem.

Graphs
chapter (Chapter 7) to

a whole

devote

used to

algorithms.

graph

section, we discuss
of a set V of
consists
E)

In this

represent graphs. graph

to a pair of
nodes), and a set E of edges. Each edgecorresponds
vertices. The edgesrepresent
the vertices. For example,the graph
relationships
among
a set of people,and the edges may connect any two persons who know
may represent
each other. A graph can be directed, or undirected. The edges in a directed graph are
the

data

vertices

structures

G =(V,

(also called

In
the two vertices
the edge connects is important.
arrow pointing
from
one vertex (the tail) to another
(the
The edges in an undirected
head).
graph are unordered pairs. Trees are simple
a hierarchy
to indicate
of graphs. If we want
in a tree,
we can orient
all the
examples
trees
Such
are
root.
the
sometimes
called
from
it
rooted
since
edges to point
trees,
away
is enough
of all the edges. We can also
to specify the root in order to define the direction
do not correspond to a
which
consider undirected trees (sometimescalled free
trees),

pairs

\342\200\224
the

case we

specify

ordered
this

hierarchy.

order
an

edge

between

as an

84

Structures

Data

use

will

We

representation

G is an

matrix of

the matrix is

example,

of edges

number

the

size n2

space of

require

they

which

no
a tree

in

(depending
pointers per vertex
With adjacency matrices, each vertex
most
if the number of edges is small,

tree).
words,

if (v,,

only

the

in

The

|V|=\302\253.
v7-)

yth position

The

e E.

first

adjacency
of

row

/th

if there is

an

edge

Adjacency matrices have one major drawback

matter
how many edges are in the graph.
For
is n - 1, and these edges can be represented by

or two

one

= 1 if and
has a 1

The

book.

this

in

Let

otherwise.

a 0

and

that a^

such

graphs

of a graph.

matrix

of size

array

v7-,

of

representations

adjacency

n matrix

an

from v, to

leading

the

n x

thus

main

two

uses

on

we

whether

has

entries

of the

in

to

want

the

or

go up

array of

an associated

other

In

matrix

adjacency

the

down

size n.

be

will

Os.

Instead

of

an

having

representation

explicit

Is (representing

of

in a

for all of those Os,

linked list. There will

link the

can

we
be

one

pointer
the adjacency list. In the adjacency-list
each vertex is associatedwith a linked list consisting of all the edges
representation,
vertex. This list is usually sorted according to the labels of the heads of
this
to
adjacent
number

actual

per edge.

the

the

edges)

This secondrepresentation

The whole

edges.

corresponding

in the

array includes

its list

of edges.

label

the

the

If

is called

graph

by an array of
a pointer to

is represented

graph

(or index) of the vertex,

is static \342\200\224
that
is, if no

and

or

insertions

lists. Each entry

the

deletions

of

beginning

are allowed

following way. We assignan array of size

Each such entry
(in order).
| V | + | E |. The first
| V | entries
correspond to the vertices
contains
the index in the array
where
the list of edges emanating
this vertex is
from
started. For example, if there are 20 vertices and 50 edges and vertex 1 has 4 edges
and the second
| V | +1),
emanating from it, then the first entry will be 21 (it is always
will
be
25.
The
entries
to
the
contain
the
of these edges.
heads
entry
corresponding
edges
In the
if
the
vertex
to
the
fifth
second
of
the
second
above,
vertex,
example
edge
points
then entry 26 is equal to 5. The edgesare usually
stored in a sorted order,although
this is
not always required. All three
are illustrated in Fig. 4.18.
representations
Adjacency
matrices are usually
easier
to handle than are adjacency lists, and
the programs
using
them are usually
the
lists
are
more
efficient
when
However,
simpler.
adjacency
graph
has few edges. In practice,
most graphs have much fewer than
the maximal
n (n - l)/2
or n (n - 1) directed
undirected
lists are more common.
Thus, adjacency
edges.
\342\200\224
the lists

4.7
Data

can

structures

we start

They

represented by

arrays

in the

Summary

structures.
efficient.

can be

The size
using

it,

Linked

can support

and

be divided

into

of an

or at

and

dynamic

least a good

structures.
on

bound

it, has

static

are

Arrays

to be

known

before

be extended. On the other hand, accessing an array is very

and reduced in size.
be extended
dynamic. They can easily
the constraints of the total available
(within
memory).

it cannot

lists are
any

array,

static

size

Data structures
can
multidimensional structures.
structure
they represent is the

also
Arrays

possible

be

divided
and

order

into

linked

among

one-dimensional

structures

one-dimensional. The
the elements. Trees represent
a

lists are

and
only

little

Notes and Further

Bibliographic

1
1

(b)

(a)

13

12

Reading 85

13

10 11 12
(c)

4.18

Figure

more than
more

even

Graphs can

hierarchy.

represent

they

Of course, we

structures.

elaborate

\342\200\224

structure

one-dimensional

Graph representations.

can also build

multidimensional

represent
or

arrays

multidimensional linked lists.

The concept
operations
are

from the data

required

dictionaries, priority
If we need only
best option.
the explicit key of
hash table, then the
is the

The

study

and

Knuth

include

to

whole

table

We

structures

data

a result,

any

structure

be used if the

now

considered

many books on data structures

contain a wealth of information

[1980],
[1984],

and

algorithms

Aho,
and

on them,

imposed

access dependson
to

the

find

the
that

for

[1983].

then

something

minimal

hashing

besides

key

in

Reading
a

basic
have

about

part of computer
been
data

science

written. Knuth [1973a]

Other books
structures.

and Ullman [1983],

Hopcroft,
Wirth [1986]. A more
advanced

is by Tarjan

concentrate on

data structures.

if we
wish
For example,
must still be scanned.

is

us to

allows

It

to postpone implementation details

have
described
implementations

Notes and Further

[1983], Gonnet
structures

cannot

element.

the

of

[1973b]

data without

store

Hashing

As

Standish

and union-find

queues,

Bibliographic
education.

and

structure,

specific data type.

on the

dependent

data types is very useful.

abstract

of

Reingold and Hansen

monograph

on data

86

Structures

Data

study of many
Jones's
paper also

A comparative
[1986].

Jones

The algorithms
others,
described, among
empirical study
was performed by

random deletions
comparison

Knuth

in binary
length of the

the

that

search trees

average

certain

under

that,

were

average path
[1973b]. An

deletions

and

Culberson[1985]proved

path

conditions,

A
of the average
path to be O(yn).
Schwab
in
Baer
and
is
schemes
presented
balancing

the length

cause

insertions

and

different

between

the

by

priority

trees

search
that

see

information,

who conjectured

[1983],

Eppinger

as 0(log3A?).

be as high

and deletions in binary

[1962]. This paper proved

the effects

on

may

by Hibbard

on

bibliography

comprehensive

insertions is 21n/?. For more

of random insertions

n random

after

length

insertions

for

queues.

includes a

was done

for priority-queues

structures

data

Sleator
in Knuth [1973b] and Tarjan
[1983].
[1977]. Balanced trees are also described
for
methods
new
maintaining self-adjusting trees.
and Tarjan [1985] present several
node to the top after
accessed
is to adjust the tree by moving the most currently
idea
The
balanced,
access.
they exhibit good performance
Although the trees are not always
every
be slow, but over a
in the amortized
sense; namely, a singleoperation
characteristics
may
is
small.
each
for
time
the
operation
average
long
period,
in Knuth
can again be found
about
More information
[1973b], and in
hashing
Gonnet

A book

[1984].

by

and

Vitter

describes

Chen [1987]

in

detail one strategy

called universal hash

great

called coalesced hashing. Classesof

functions are describedby Carter and Wegman [1979]. Several interesting
applications
and Upfal
in Wegman
and Carter [1979], Karlin
of this concept can alsobe found
[1986],
that allow
schemes
Manber
and
and Kurtz
[1987]. There are also extendible
hashing
and Strong
of the tables; see, for example, Fagin,
Nievergelt,
Pippenger,
dynamic growth
Litwin
and
[1979]
[1980].
hash

random

The union-find data

also

Fischer

by

[1972],

in Theorem

mentioned

0{ma{my a?)),where
than

log*a?.

compression

7 and

Hopcroft

a(n) is the

achieve

the same

[1984] studied several simpler

on
running time. For information

variations

graphs,

and
result

of path
see Chapter

Exercises

4.1

Write

a program

to delete an

4.2

Write
should

a program
all point

to reverse
backward.

Convert

the

simple

element
the

from a linked

direction

recursive search

list.
list.

of a given linked

procedure

for

binary

In

other

words,

search trees

to

procedure.

4.4

the

its bibliography.

Drill

4.3

[1964],

[1973]
(who
time to
[1975] improved the running
function, which grows even slower

Tarjan
Ackerman's

inverse

Fischer

and

obtained

Ullman

and

others.

among

by Galler

studied

and van Leeuwen

Tarjan
that

and

4.3),

first

was

structure

functions

Design

an algorithm

to list

in

order

all the

keys in

a given

binary search tree.

the pointers

a nonrecursive

Creative Exercises 87

^5
4.6

A [1.. 16] be an array

the minimal number of

that represents a heap (using

the implicit representation).
elements
that can occupy an array of size 16?

Let

to Heap may
at most one

Insert

Algorithm

so

algorithm

that

is

What

heap

many times up
(0(log/i)

elements

swap

swap

be

will

the

performed

Modify the

heap.

comparisons are

still

allowed).

4.7

Suppose that we want to use

complexity of all the operations?

4.8

Show the

4.9

Show

tree formed

AVL

tree

AVL

an

resulting tree
the tree, specify

an AVL

be made

the

whose deletion

a node

with

cannot

the numbers

inserting

by

and explain

node,

as a priority-queue

trees

AVL

tree

the

to 20

is the

What

in order.

in a

results

non-AVL tree,

resulting

the

that

such

or double)
rotation.
tree cannot be balanced

one (single

only

by

why

data structure.

Draw
with

one

rotation.

Exercises

Creative

4.10

of an abstract data

an implementation

Design

Insert(x):

In

other

only one of them

This abstract

4.11

the pool data

element can appear

Another

now

the

Design

from

If there

most

in

the

data

assign

it to

same

element

y.

be removed.

of Exercise
once

4.10

in

in the data

the same operations

each operation

already

duplicates.

the data structure and

are several copiesof the

called a pool (or a bag). It is useful

inserted into the pool, and when
operations should take 0(1) time.

type
at

Implement

for storing
a worker

in the

the

way: Assume

following

structure.

as before,

worst case?

but
What

jobs, for example.

becomes available

with

is

now

that

every

now check for

What is
duplicate
checking.
a good data structure for the

insertion

An

must

pool data type (see Exercises 4.10 and 4.11) is the following: Assume
in the range 1 to /i, and that n is small
elements are identified
by integers
can allocate memory of size O (n). Each element
can appear at most once.
for insert and remove (as defined
in Exercise
4.10) that work in 0(1)

of the

variant

that all the

enough that

do.

the following operations:

x is

hold

and

Modify

the complexity of
average case?

4-12

element

if

type is

data

are generated
is removed. All

duplicates.

any
will

should

New jobs
job

be performed even
the data structure should

words,

Removeiy): remove
any element
Again,

supports

should

insertion

the

structure.

that

type

you

algorithms

time.
4J_3

node

one heap that contains all the elements

an algorithm to construct
of two given heaps
The heaps are given in a linked-list representation (each
and m, respectively.
links to its two children). The running
has
of the algorithm should be
time

O (log

(m +

Design

of sizes n

4.14

Design

n))

in

the worst

an algorithm

What is the

complexity

case.

to construct one heap

of the algorithm?

that

contains

all the elements of

given

heaps.

88

4.15

Data Structures

x into

the key

insert

lnsert(x)\\

find

Find_Next{x):

should take
operations
the data structure.

these

key

only

structure (if

0(\\ogn) time

is not already

if it

than x.

is greater

that

worst case,

in the

there.

is there).

it

structure

the data

in

the following operations:

supports

structure

the data

the smallest

that

type

the data

delete the key x from

Delete(x):

All

of an abstract data

an implementation

Design

n is

where

the number of

elements in
4.16

of an abstract data

an implementation

Design

insert

Insertix):

Find_Smallest{k):

the fcth smallest

find

should take
operations
the data structure.

these

only

structure (if

the data

key

0(\\ogn) time

in the

is not already

if it
it

there.

is there).

structure.

the data

in

the following operations:

supports

structure

the data

delete the key x from

Deleteix):

All

x into

the key

that

type

worst case,

n is

where

the number of

elements in
4.17

of an abstract data

an implementation

Design

Insert(x): insert

Find_Next(x, k): find

in the data structure

are

these operations should take

the data structure.

All

structure (if

&th \"right\"neighbor

the
that

the data

larger than

is not already

if it
it

there.

is there).

smallest

the keys

among

key

.v.

time in the

0(\\ogn)

the following operations:

supports

x into the data structure only

the key

delete the key x from

Delete(x):

that

type

worst case, where

n is

the number of

elements in
*

4.18 The AVL

algorithms

that were

possible values, 1, 0, or -1.

these

implementing

presented in

Torepresent
(with

algorithms

only

4.3.4 require

Section

three

values

a slight

balanced factors

we need 2 bits.

modification)

with

only

three

with

for

a method

Suggest
1

bit per

extra

node.

4.19

take two sets, such that all the keys in one set are smaller than
set, and merges them together.
Design an algorithm to concatenate
time
should
binary search trees into one binary search tree. The worst-case running
O (/?), where h is the maximal
of
two
the
trees.
height

concatenate

the keys

4.20

Design

valid
height

in

all

operation

the other

an algorithm
to concatenate (as defined
AVL tree. The worst-case running
time
of the two trees.

4.21 Consideran

in Exercise

should

4.19) two AVL trees

be 0(h\\

where

is the

into

two

be

one

maximal

and deletions.
a fairly random sequence of insertions
by
a
balance factor appears
the same probability
with
(namely,
Prove that the average length of the path from the
probability of 1/3 for each possibility).
critical node to the place of insertion is a constant
of the size of the tree.
independent

Assume

4.22

4.23

that

AVL

tree formed

each

possible

Determine the general structure

order. What is the height of this
Find the \"worst AVL tree.\"

of the

AVL tree

formed

by

inserting

the numbers

to n

tree?

That

is, construct

an AVL tree of

height

h with

the

minimal

in

4.24

4.25

number of

nodes. Usethis

the maximal

height of an

Let 7, and
at most In
A

AVL tree

worst

AVL

tree with

T2 be two arbitrary trees, each

rotations to 7, so that it becomes

of two undirected graphs

W = V \\^j U (namely, the

Theorem

(Hint: Try a
n nodes.

having

equal to

Prove

it is

that

regarding

sufficient to apply

T2.

vertices of the

the
and D=E\\<jF
\\^jVxU (namely,
from each vertex in V to each vertex in U). Suggest
to be performed efficiently.
allows join operations
4.26

4.1 (Section 4.3.4)

recursive construction.)

E) and H = (U, F) is a new graph J = (W, D) such

new graph include the vertices of both
graphs),
edges include all the previous edges plus an edge

G = (V,

join

that

to prove

nodes.

89

Exercises

Creative

a good

representation

for

that

graphs

[s\\,s2,..., sm) be a very large set, and assume that S is partitioned into k blocks.
block
that you have a procedure called which
such that given an element
s,,
block works in constant time
whichblock(Sj) = number of the block that contains s,; which
to all street addressesin the United States, and the blocks may
(e.g., S may correspond
You
want to maintain a small subset of S, T, and to perform
the
to
codes).
correspond
zip
T:
on
operations
following

Let S =
Assume

Insert(Sj).

Delete(Sj).

Delete_block(j): delete all

in T

elements

T is empty. Each operation

Initially,
the number of elements currently

should

belong

take O

(logn)

Delete block

T.

in

that

to block
time

j.

in the

worst case,

n is

where

the
(disconnects)
one of them.
every

removes

only

elements from the data structure; it need not physically remove each and
Both m and k are too large, so you cannot afford to use a table of size m or k.
relatively small, and you can use O (n) space.

n is

However,

4.27

Let A [\\..n ]

be an

array

of real

numbers.

to perform

algorithms

Design

any

of the

sequence

following two operations:

y):

Add{i,

add the

Notice

4.28

that the

y to

the

/'th

number.

return the sum of the

Partial_sum(i)\\

only

value

first

fixed
number of elements remains
values. Each operation
of size n as a work space.

are to the

changes

one more array

/ numbers,

\302\243[/'
i

].

(there are no insertions

should take 0(\\ogn)

or deletions);

steps.

can

You

the
use

4.27 to support insertions

and
data structure for the problem in Exercise
Each element now has a key and a value. An element is accessedby its key. The
addition operation applies
to the values (but the elements are accessedby their keys).
The
is different.
Partial sum operation
the

Extend

deletions.

Partial
value

The

worst-case

operations.

return

sumiy):

is less

than

running

y,

the sum

of all

the elements

currently

in

the

set whose

\302\243a,.
\342\226\240V|<V

time should

still

be

0(n\\ogn)

for

any

sequence

of

O(n)

90

4.29

Data Structures
a data

Design

structure

operations

following

Find_value(x):

to

maintain

a set

of elements, each with

find

be supported:
the value associated

a key

and a value. The

should

with

the

element

(nil

if x

is not

in

the

set).

Insert(x, y).

Delete(x).
Add(x,

y):

add the

value

y to

the current

value

Add_all{y)\\ add the value y to the values of all

time should be O (log/i) for
The worst-case running

4.30

of the
the

each

element

elements

of these

with

key x.

set.
operations.

in the

A programmer named Guy once encountered

an error message from
a new
story.)
that the compiler had run out of memory space while
indicating
compiler he was using
was baffled,
since the program did not use much
a program. The programmer
compiling
able
to
to
He
was
the
a
certain
case statement, which
is given
problem
pinpoint
space.
the program
With
the
below. Without this case statement,
it,
compiled flawlessly.
compiler
out of space. Determine what
data structure
the compiler was using that was causing
ran
the problem.
the problem lies with the compiler,
(The case statement is correct and valid;
to compile the case statement.)
which was unable
(True

case

/' of

1:

Statement(l);

2:

Statement(2);

4:

Statement(3);

256:

Statement(4);

65535: Statement(5);

CHAPTER

ALGORITHMS

OF

DESIGN

BY INDUCTION

invention,
than

important than to see the sources of

more interesting
are, in my opinion,
inventions themselves.

is more

Nothing

the

which

G.W.Leibniz (1646-1716)
Invention

breeds

invention.

R.W.Emerson

5.1
In

(1803-1882)

Introduction

this

we introduce

chapter,

our

approach

to algorithm

design

using

the

analogy

to

We
include
relatively simple examples, and present the basic
method
is based. The analogous induction
the
techniques on which
have
been described in Chapter
When appropriate, we repeat the
2.
techniques
discussion
here to make this chapter
self contained.
Mathematical induction
is based
we have a
on a domino principle. Imagine
that
line
of upended
and that we wish to knock down
all of them by knocking
dominoes,
down
all dominoes
the first. To make sure that
will fall down,
we need only to
only
one
and
first
that
the
that
each
we
have
domino
will
the next one as
verify
pushed
topple
it falls.
We need not collapse the whole
we
time
add
a
new
domino
to
arrangement
every
work.
will
The
can
that
the
same
new
be
to
verify
arrangement
principle
applied

mathematical
principles

algorithm

induction.

and

design.

ai

Designof

92

by Induction

Algorithms

It is not necessary to design the steps required

from scratch; it is sufficient to guarantee that (1)
of the problem (the base case),and
instance
small

problem can be constructed

inductive step).
this

With

in mind, we

principle
(or to a

problem

set of smaller

The

problems).

the problem to a smaller

usually not easy to find

on reducing

concentrate

should

problem

possible to solve a
to every
(2) a solution
of smaller problems (the

solutions

from

to solve the
it is

is

trouble

that

it is

to facilitate
chapter, we present severaltechniques
were chosen not because
of their importance
this process. The examples
chapter
but because they are simple and yet they
(some of them have limited applicability),
we want to emphasize. We will present
numerous other examples
illustrate the principles
the
book.
of this approach
throughout

to reduce

a way

the

In this

problem.

this

in

5.2

a simple

with

start

We

Polynomials

Evaluating

\342\200\224

algebraic problem

a given

evaluating

polynomial

at

a given

point.

Given a sequenceof

Problem

The

a real number x, compute the value

\342\200\242\342\200\242\342\200\242
+ a{x + a0.
an_\\Xn~l +

and
+

This problem
We

problem.
it

of the

an, an_\\,...,

a0,

an inductive

candidate

for

can lead directly

simple (almost trivial)

to a

natural

a \\,

polynomial Pn{x) = anxn

that induction

the most

with

start

to be

seem

show

will

to better

lead

that

not

may

Nevertheless,we

numbers

real

approach,

approach.

to the
very good
then find variations of
solution

solutions.

The problem

of a

problem

of evaluating

The

numbers.

n+2

involves

in terms

inductive

is to

approach

solve

this

problem. In other words, we try to reduce the

solve
problem to one with smaller
size, which we then
or, as we call it, by
recursively,
induction.
The first natural attempt is to reduce the problem
We are left
by removing a\342\200\236.
problem

with

the

Pn-\\(x)

This is
it

by

polynomial
+ a\\X+a0.

+tf\342\200\236-2-*\"~2+

problem,

that it has

except

one less parameter.

we can

Therefore,

solve

induction.

Induction
by

We can
solve

the

smaller

'\342\200\242\342\200\242

an-\\xn~{

same

the

to a

solution

the

now

the base

input

use

hypothesis:
an_\\, ...,
the

case,

We
a\\,

hypothesis
which

know

a0,

how

at the

to solve

is computing

to evaluate a polynomial
point x (i.e., we know how

the

by

problem

a{); this is

trivial.

induction.
Then,

represented
to compute

First, we have
show how

we must

to

to

5.2

solve the
problem

is the

compute xn,

At this

an, and add the

it by

multiply

Pn{x) = Pn_x{x)

the

Although

multiplications
the

This induction
(since

to compute

x\"~l

by

xn, then

that, even

not

is

it

return to

coefficients

an,

problem becomesthe
is
an_{,..., a {, which

an is

that

(Notice

on.) So we

have

now

a new

the

evaluation

polynomial

This

hypothesis

how

the

way.
straightforward
first

coefficient,

polynomial represented

by

the

an_{

is

the

(a?

\342\200\224

2)th

and so

coefficient,

induction hypothesis.

represented
to compute

is more

by the coefficients

suited

P'n-\\

to

to compute Pn(x)
following
expression:

from

how

know

a,nan_{

to evaluate
the point

,...,#

\\ at

it

is easier

the

x (i.e.,

(x)).

our

PnW=X'P'n-\\(x) + a{). Therefore,

required

different

\342\200\242\342\200\242\342\200\242+*!.

l)th coefficient,

(n-

It is

a better

However,

implement.

of the

need

n additions.

requires more computation,

This algorithm looks good

in yet another

Induction hypothesis (reversedorder):We

we know

to

addition

since we

strong,
and

the last coefficient, an, is the

the problem by removing
the only possible reduction. We can also remove the

= anx\"-{+an_{x\"-2+

P'n_{(x)

later.

point

only one
one

then

to

is easier

it

perform

an xn,

too

not

hypothesis

this

induction

using

by

The smaller

a{).

to get

simple, and easy

It is efficient,
We discover

Reducing
it

though

We will

overall.

work

induction

to

need

We

hypothesis is
there are 2n multiplications

the

xn~\\ but

computing

induction

Overall,

the

xn~x.

to compute

one more multiplication

(The

anyway.)

note

exists.

but

step,

requires
use

now

the value of

to compute

how

know

how

since it requires
to compute x\.")

easier

computation.

measures.

all

algorithm

We

is stronger,

to compute
the

to less

It
We

additions.

solution.

a better

hypothesis:
we know

now

is

it

to complete

it leads

and

improvement

hypothesis

extend

interesting

preceding
In a

written.

is

it

efficient.

not

is

it

multiplications

Pn_x (jc), and

multiplication

as

left

\342\200\224
it

powers

induction

Stronger
polynomial

to

correct,

+ 1)/2

obtain

the

by

implied
to

right

is frivolous

problem

by observing that there is a great deal of redundant

of x are computed from
scratch.
We can save many
when
we compute xn. We make
this change
by using the value of xn~x
xk
in
of
the
induction
computation
hypothesis.
first

The

by including

to

in this

of our approach.

is

algorithm

differently

from

polynomial

the power

+1=a?(a?

We make the
computation:

the

\342\200\242\342\200\242\342\200\242

+
a little

case; simply

(x):

use of induction
a very simple solution. The algorithm

+n-\\+n-2

induction

to Pn_{

result

in this

smaller

the

to

solution

step is straightforward

seem that the

it may

point

discussion is merely
evaluating
moment, however, we will see
n

the

aid of

the

with

+ anxn-

complicates

just

(computing Pn(x))
value of Pn_\\(x)). This

problem

original
(which

93

Polynomials

Evaluating

purposes,
only

P'n-\\(x).

one
The

because

to extend.

and one
multiplication
complete algorithm can be

addition

Clearly,
are

describedby

Design of Algorithms

94

anxn+an-xxn
This
(It

+tf1*

mentioned

also

evaluate the

\342\200\242
\342\200\242
\342\200\242

is known

algorithm
was

polynomial

as Horner's

rule after

the

English

by Newton, see [Knuth

is given in Fig. 5.1.

a=a0,a^a2,

-)x + a\\)x + ao.

= (('-((anx+<*n-\\)x+<tn-2)'

+ tf0

1981],

mathematician
467.)

page

W.G. Horner.
The program to

(a, x);

PolynomialJEvaluation

Algorithm

Input:

Induction

by

..-,an (coefficients

of a

and x (a real

polynomial),

number).

Output:

P (the

value of the

polynomial

at x).

begin

for i := 1ton do

p :=x*P+an-i
end

Figure

5.1

Algorithm

The algorithm
Complexity
requires
the
memory location. Even though
it worthwhile
efficient, we have found
algorithm

faster

than

only

Polynomial^Evaluation.

n multiplications,

additions,

previous solutions seemed

to pursue a better algorithm.

those described previously,

but

also

very

and one

extra

and

very
is this

simple
Not

its corresponding

only

program is

simpler.

Comments

Induction
allows
us to concentrate on extending
solutions
of smaller
to
those of larger problems.
that we want
to solve P (n), which
is a
subproblems
Suppose
on a parameter n (usually
its size). We start
with
problem P that depends
an arbitrary
instance of P(n\\ and try to solve it by using the assumption
that P(n -1) has already
been solved. There are many possible ways to define the induction
and many
hypothesis
to use it. We will
several
of
these
and
their
possible ways
will
show
methods,
survey
power in designing
algorithms.
This simple example illustrates
The
the flexibility
we have when
we use induction.
trick
that
led to Horner's rule
was merely
the
from
left
to
instead
considering
input
right,
of the intuitive
is comparing top down
versus
common
right to left. Another
possibility
bottom
is involved).
It is also possibleto go in increments of 2
up (when a tree structure
(or more) rather than 1, and there are numerous
other
Moreover, sometimes
possibilities.
the
best
induction
sequence is not the same for all inputs. It may be worthwhile to design
an algorithm
We will see examples of
just to find the best way to perform the reduction.
all

these

possibilities.

5.3

Maximal Induced

Subgraph

95

Maximal Induced Subgraph

5.3

You are arranging

a conference
of scientists from
have
a
list
of
want
to
invite.
You
assume
that
you
people
you
the
list
will
on
to
come
under
the
condition
that
there
will
be
everyone
agree
ample
the names
of all other
you write down
opportunity to exchange ideas. For each scientist,
scientists on the list with whom interaction is likely. You would like to invite as many
on the list as possible,but you want to guarantee that each one will have at least k
people
to interact (k is a fixed number,
of the number of
other people with whom
independent
in particular,
invitees). You do not have to arrange the interactions;
you do not have to
make sure that there is enough time for them to occur. You just want to lure everyone to
the

problem.

following
and
disciplines

Consider

different

decide

whom
to invite?
This problem corresponds to the
An induced
Let
be
an
undirected
G
=(V, E)
problem.
following
graph-theoretic
graph.
of
is
a
such
that
V
and
F
includes
all
G
i/c
F)
H=(U,
graph
subgraph
edges in E both of
vertices
are in U. A degree
of a vertex is the number
of vertices adjacent
whose incident
The vertices of the graph
to the scientists, and two vertices are
to that vertex.
correspond
for the two corresponding
scientists to exchangeideas.
connected if there is a potential
to
a
subset
of
An
induced
the
scientists.
subgraph
corresponds
the

How

conference.

induced

an

find

induced

subgraph

undirected
//=(\302\243/,

degree > k

(in

G =(V, E) and an integer

of maximum size such that
or conclude that no such

graph
of G

F)

//),

exists.

approach to solving

vertices are

an

subgraph

of H have

all vertices

A direct

Given

Problem

The
k,

do you

this

problem

is to

remove verticeswhose

degree

is <

k. As

of other vertices may be

edges, the degrees
vertex should be
vertex becomes less than
k, that
we remove all the
order
Should
however, is not clear.
vertices
of degree < k first,
reduced?
Should
then
deal with vertices whose degreeswere
we remove first one vertex of degree < k, then continue
with affected vertices? (These
two
which
are
to breadth-first search versus depth-first
search,
approaches
correspond
discussed in detail in Section 7.3.) Will both approaches
lead to the same result?
Will
the resulting
be of maximum size? All these
are easy to answer; the
graph
questions
we will describe makes answering
them
even easier.
approach
Instead of thinking
our algorithm as a sequence of steps that
about
a computer
has
to take to calculate a result,
of proving a theorem that the algorithm
think
exists. We do
not
a formal proof (at least not at this first stage). The idea is to
suggest
attempting
a theorem, in order
imitate
the steps we take in proving
to gain insight into the problem.
We need to find the maximum
induced subgraph that satisfies
the given conditions. Here
removed

reduced. When
removed. The

is a

\"proof\"

the

by

with

their

of a
degree
of removals,

induction.

adjacent

96

Design of

Induction

all of

hypothesis:
vertices

whose

Induction

by

Algorithms

We

to find

how

know

> k,

have degrees

provided

maximum induced subgraphs

that the number of vertices

is < n.
We

that this

to prove

need

' 'theorem\" is true

for

a base

case,

and

that

its truth

for

- 1

base case occurs when n =k + 1, because if

If
than
k.
n -k + 1, then the only way to have all the
less
are
n <k,
degrees
all vertices
are connected),
have
a
is
to
k
to
complete graph (namely,
equal
degrees
>
is
k + 1 vertices.
now
that
a
with
n
assume
G
detect.
we can
So,
which
=(V, E)
graph
>
whole
satisfies
the
conditions
and we
have degrees k, then the
If all the vertices
graph
are done. Otherwise,there exists a vertex v with degree < k. It is obvious that the degree
of v remains < k in any induced subgraph of G; hence,v does not belong to any subgraph
its

implies

that

the conditions of

satisfies

edges
n-\\ vertices
without

The first nontrivial

n.

for

truth

all the

then

the

affecting

Therefore,

problem.

the conditions

of

the

theorem.

we can
After

remove v

and

its adjacent

v is

removed, the graph has

to solve the problem.

induction hypothesis, we know how

to the questions
The
and
the answers
we raised
now
We
are
algorithm
The order of removals
earlier are now clear.
Any vertex of degree < k can be removed.
after all these removals must be of maximum size
The
is immaterial.
graph remaining
It is also clear that the algorithm
are
is correct,
because these removals
mandatory.
because we designedit by proving its correctness!
\342\200\224

and,

by the

done.

The best way to reduce a problem is to eliminate some of its elements.

Comments
In this example, the application
of induction was straightforward,
it was
because
mainly
clear which
vertices
we should eliminate and
how
we should eliminate them. The
reduction
follows
In general, however, the elimination
immediately.
process
may not be
We will see examples of combining
two elements
into one, causing the
straightforward.
number
of elements
to be reduced (Section 6.6); of eliminating
restrictions
on the
than
of
the
and
of
problem rather
7.7);
(Section
eliminating
parts
input
designing a
to
find
which
elements
can
be
eliminated
Another
special algorithm
5.5).
(Section
of
the
elements
is
next.
It
is
to
note
that,
example
eliminating
right
presented
interesting
\">\" with
\"<\"
if we replace
in the statement of the problem
(that is, if we look for a
maximal
induced
all of whose degrees are at most k), the problem becomes
subgraph
much
more
difficult
(see Exercise 11.12).
5.4

Finding

Let/be

to another
n. We

function

One-to-One
that

maps a

finite

Mappings
set

A into

itself (i.e., every

elementof

A is

mapped

For simplicity, we denote the elements

of A by the integers 1 to
assumethat the function /is represented by an array / [ 1..n ] such that / [i ] holds
the
value
between
1 and
(which is an integer
of/(/)
n). We call /a one-to-onefunction
for
if,
to j. The function
every element y, there is at most one elementi that is mapped
/
as shown in Fig. 5.2, where both sides correspondto the
can be represented by a diagram,
the mapping. The function
indicate
same set and the edges
a
in Fig. 5.2 is clearly not
one-to-one

element

function.

of ,4).

5.4 Finding One-to-One

A mapping

5.2

Figure

6,/

- 6

7^^

.7

from a set into

Given a finite

itself

(both

sides represent

Mappings 97

same

the

set).

a mapping /from A to itself,

find a subset 5c A with the maximum number of elements,such that (1)
the function / maps every element of S to another element of S (i.e.,/
elements
of S are mapped to the same
S into itself), and (2) no two
maps
to 5).
element
when
restricted
one-to-one
(i.e.,/is
Problem

The

set

A and

set A satisfies the conditions of the problem,

the other hand,
for some i*j, then
S
f(i)=f(j)
cannot
the problem given in Fig.
contain both / and j. For example, the set S that solves
5.2 cannot contain both
choice
of which one of them
2 and 3 since / (2)=/(3)= 1. The
to eliminate
to eliminate
3.
for
that
we
decide
cannot be arbitrary.
Suppose,
example,
Since 1 is mapped to 3, we must
1 as well (the mapping must be into
eliminate
S and 3 is
no longer in 5). But if 1 is eliminated, then 2 must be eliminated as well (for the same
since
it is easy to see that
we
could
have
reason).
But, this subset is not maximal,
eliminated 2 alone. (The solution
The problem is to
for Fig. 5.2 is the subset
{1,3,5}.)
find
a general
to include.
method to decide which
elements
in
how to reduce the problem
to a
we
some
have
deciding
Fortunately,
flexibility
smaller one. We can reduce the size of the problem
either
an
element
that
by finding
to
S
or
an
that
does
not
to
We
will
do
element
S.
the
latter.
We
use
the
belongs
belong
If / is originally
and A is definitely

maximal.

induction

straightforward

then

one-to-one,

Induction

the whole

If, on

hypothesis.

hypothesis:

We

know

how

to solve

the

for

problem

sets of

n-\\

elements.
The

base

is trivial: If there is only

one-to-one mapping.
for a subset S that
looking

case

itself,
and we are
which

is a

one

element

Assume
satisfies

now

in the
that

set, then

it must

we have a

the conditions

of

the

set

problem.

be mapped to
n elements
We claim

of

Design of Algorithms

98

that

words,

an element

cannot

be

in

is such
n

Otherwise,

5.)

which

elements,

say, k

therefore,
remove it from the
into itself;

/maps

the

by

is not

belong to S. (In
connected to any

elements, then
cannot

mapping

We

set.

now

other
edge,

are
If there

k elements

those

be one-to-one.
a set

A'=A-{i) with
we know
how
to

have

induction

hypothesis,
the mapping is one-to-one,

then

exists,

it cannot

which

diagram,

S has,

the

problem for A'. If no such /

solve the

the

k-\\ elements;

we simply

/, then

an

element mappedto

right side of
if / e S and

in the

at most

mapped into

has no other

/ that

element

any

Induction

by

are

we

and

done.
remove
/. We proved that
/ cannot
is that we must
The essence of this solution
Once
we
remove
an
and
of
induction:
element
reduce
is
the
This
S.
to
strength
belong
done.
We
have
to
be
that
the
reduced
we
are
the
careful,
however,
the size of
problem,
The only condition
problem is exactlythe same (except for size) as the original problem.
is still
function
on the set A and the
/ was that / maps A into itself. This condition
that
was mapped
to /. The
for the set A-[i}, since there was nothing
maintained
when no more elements can be removed.
terminates
algorithm

step,

we found

a recursive

as

algorithm

that no other

such

element

an

the

described

We

Implementation

procedure.

element is mapped to

it,

removed

In

each

it, and

need not be recursive. We

can
continued recursively. The implementation,
however,
a counter c [i ] with each element /. Initially,
c [i ] should be equal to the number
maintain
of elements that are mapped to /'. We can compute
c [/], for all /, in n steps by scanning
the
and incrementing
the appropriate counters. We then put all the elements that
array
have
a zero counter in a queue.
In each step, we remove an element
j from the queue
if
in
we
the queue. The
(and the set), decrement
and,
c[f(J)],
c[f(j)]=0,
put
/(/)
terminates
when
the
is
in
The
is
algorithm
queue
algorithm
empty.
given
Fig. 5.3.

Complexity
put on

the

queue

from a

without

was

The

time. The total

of steps

number

is

operations.
in

removing

thus

O (n).

Every

the

element

example, we reduced the size of the problem by

set. Therefore,we tried to find the easiest way to remove
of the problem.
A
into
itself, the
maps

function

be

can

from

an element

the

eliminating

an element

the only requirement we made

of an element to which
no other

Because
choice

is mapped is natural.

element

The

5.5

the

and

the conditions

changing
that

requires 0{n)
steps involved

part

once,

In this

Comments
elements

at most

constant

take

queue

initialization

The

next

Problem

Celebrity

example

is a popular

exercise

in

algorithm

design.

It is

a nice example

of

the data (or even a

not require scanning
all
of them).
a celebrity
is defined as someonewho is
significant
Among
persons,
part
known
the celebrity,
by everyone but does not know anyone. The problem is to identify
'
if one exists, by asking
'Excuse me, do you know the person
only of the form,
questions
is that
all the answers are correct,
over there?\" (The assumption
and
that even the

problem

that

has a

solution

that

does

celebrity will

answer.)

The

goal is to

minimize the

number

of questions.

Since there are

5.5 The Celebrity Problem

Input: /(an

of

array

S (a

Output:

(f,

Mapping

Algorithm

);
whose values

integers

subset of the

99

from

integers

are between1 and

1 to

n, such

1 to

n (

that

/is

n).

on S).

one-to-one

begin
from
S := A; { A is the set of numbers
:=
:=
1
ndo
to
0;
c[j]
for j
for j := 1to n do increment c [f [j ]];

for j

:=

1 to

n do

ifc[J]=0
while

then

remove/ from
S:=S-{i);

j in

Queue;

empty do
top of

the

the queue;

c [/\"[/]];

decrement

if elf

put

is not

Queue

]] =

0 then

put

/ [/

in

Queue

end

5.3

Figure

is potentially
l)/2 pairs of persons,there
are asked arbitrarily.
worst case, if the questions
the worst case.

a need

n(n-

We can
the

use a graph-theoretical
to

vertices

Mapping.

Algorithm

It is

to ask

We

formulation.

that

we

a directed graph
person A to person

persons
corresponding
knows B. A celebrity
to a sink of the graph
(no pun intended). A
corresponds
vertex with indegree
that
a graph can have at
n -1 and outdegree 0. Notice
sink. The input
to the problem
(whose
corresponds to an n x n adjacency matrix
is 1 if the /th person knows the yth person, and 0 otherwise).

The Problem
there exists
/7th
entry)

an

entry) are
are 0.

Given

the

an

n xn

/ such

that all the

1, and

all

adjacency

entries

the entries

in

the

in

the

/th

matrix,

determine

in

with

B if A

sink
most

is a
one

ij entry

whether

(except for
row (except for the //th
/th column

the

in

do better

can

can build

an edge from

and

n(n- 1) questions,

not clear

the

is simple.
case of two persons
Consider as usual
the difference
between the
problem with n -1 persons and that with n persons. We assume that we can find the
Since there is at most one celebrity,
the first n - 1 persons
by induction.
celebrity among
is among the first a?- 1, (2) the celebrity is
there are three possibilities:(1) the celebrity
the
rtth person,
and (3) there is no celebrity. The first case is the easiest to handle. We

The base

need only

to check

that the

nth

person

knows

the celebrity,

and

that

the celebrity

does

not

of Algorithms

100

Design

know

the Mh person.

whether
2(/7

mh

the

in the

questions

1)

The

are

then

consideration,

need

is probably

it

step,

avoid).

We

consider

the

to

easier

we reduce

to eliminate

of

number

to determine

because,

2(n

- 1)questions. If
will be

questions

ask

we

n{n-\\)

approach.
It may be

''backward.\"

problem

to identify

hard

After all, there

If we eliminate someone from
the problem from n to n - 1. Moreover,we do
as a noncelebrity.

someone

identify

celebrities.

of

size

difficult

to ask

need

another

need

than

the

more

are

the total

then

noncelebrities

more

definitely

not

is to

here

trick

but

celebrity,

mh

we tried to

is what

(which

The other two cases

celebrity, we may

is the

person

\342\200\224

Induction

by

someone specific;anyone

do.

will

Suppose

Alice

ask

we

that

she cannot
be a celebrity; if she does not, then
If she does, then
Bob.
whether she knows
of
We
can
eliminate
one
them
with
one question.
a
be
Bob cannot
celebrity.
we
with
which
the
three
cases
started.
We do not just take
We now consider again
We use the idea in the last paragraph
to eliminate
an arbitrary person as the mh person.
solve
the problem
for the other
n -1
either Alice or Bob, then
persons. We are
not occur,
since the person eliminated
cannot
be the celebrity.
that case 2 will
guaranteed
- 1
is
there
no
the
n
Furthermore, if case 3 occurs \342\200\224
namely,
celebrity among
persons \342\200\22
1
the n persons.
case
this
case
is easy.
but
there is no celebrity among
then
remains,
Only
it takes two more questions to verify
that
If there is a celebrity among the n - 1 persons,
is
whole
set.
there
is
no
this
a celebrity for the
Otherwise,
celebrity.
The algorithm
as follows.
We ask A whether
she knows \302\243,
and eliminate
proceeds
either A or B accordingto the answer.
Let's assume that we eliminate
A. We then find
the remaining
If there is no celebrity, the
n - 1 persons.
(by induction) a celebrity among
we
check
A
that
knows
the
terminates;
otherwise,
algorithm
celebrity and that the
does
not
know
A.
celebrity

As was the case with the algorithm in the previous

section, it is
to implement the celebrity
rather than recursively.
algorithm
iteratively,
The algorithm
is divided into two phases.
In the first phase,
we eliminate
all but one
this
is indeed the
candidate
candidate, and in the second phase we check whether
We start with
n candidates,
of this discussion, let's
and, for the purpose
celebrity.
assumethat they are stored in a stack. For each pair of candidates,we can eliminate one
candidate by asking one question \342\200\224
whether
one of them knows the other.
We start by
the
first
two
candidates
from
the
and
one
of
them.
in each
taking
Then,
stack,
eliminating
we
have
one
is nonempty,
we take
step,
candidate,
and, as long as the stack
remaining
one additional
candidate
from
the stack, and eliminate
one of these two
candidates.
When
the stack becomes empty,
We then
check
that this
one
candidate
remains.
candidate is indeed the celebrity.
The algorithm
is presented in Fig. 5.4 (notice that the
stack is implemented
/, j, and next).
explicitly
by the use of the indices
Implementation
efficient

more

Complexity
phase

most

At

to eliminate

-1

candidate is indeed a
n (n

- 1)(the

identify

a priori

number

a celebrity

questions

3(a?-1)

and

persons,

Notice

celebrity.

of

of entries

by looking

the solution may

at

the

only

be sensitiveto

be

will

at

then

most

that the
matrix).

O(n)
each

n -1 questions in
- 1) questions
to verify

asked:

2{n

size of

This

the

solution

input

shows

entries in the adjacency

of the n (n - 1)entries.

is not
that

but
\302\253,

it is

matrix,

first

the

that the
rather

possible

even

though

to

The

5.5

Know (an nxn Boolean matrix

Input:

Problem

101

(Know);

Celebrity

Algorithm

Celebrity

).

celebrity.

Output:

begin
i :=

1;

j:=2;
3 ;

:=

next

{in
while

the

all but

we eliminate

phase

first

if Know[i, j] then i := next

else j := next;
:= next + / ;
next
{ one of either i or j is eliminated
= n + 1 then

ifi

one

candidate

+ 1 do

< n

next

:= j

candidate

else

candidate := i;

that the

check

we

Now

Know [candidate,

{a

if Know
if not
if

if

to pass

the test}

k] then wrong := true

Know[k, candidate] then
:= true ;
* k then
candidate
wrong

wrong

else celebrity
end

and k<ndo

wrong

[candidate,

k:=k + l
not

celebrity

candidate]:=false ;

variable

dummy

not

while

candidate is indeedthe

wrong .-false
k := J ;

then
:=

celebrity
0 { no

:= candidate

celebrity }

Figure 5.4 Algorithm

solution is to reduce the size of the problem

elegant
This example shows that it sometimes
pays to expend
someeffort (in this case \342\200\224
one question)
to perform the reduction
more effectively.
Do
- 1 and
not
an
start
of
size
n
to
extend
arbitrary
input
by simply considering
attempting
- 1. We will
it.
see more
Select
a particular
input of size n
examples where we spend
the right order of induction \342\200\224
substantial
time
and
that time is well
just constructing
Comments

from

spent.

to

The key idea in

a clever way.

Celebrity.

in

this

of Algorithms

102

Design

5.6

A Divide-and-Conquer

Induction

by

The Skyline

Algorithm:

Problem
have

we

far,

So

with

deals

example

Given

Problem

The

these

in a

buildings

rectangular

An

seen examples from

a problem of drawing

the

exact

locations

draw

the

in

Fig.

and shapes
(in two

skyline

of several
dimensions) of

lines.

hidden

is given

of an input

example

shapes.

city,

eliminating

buildings,

computation. This

and numerical

theory

graph

5.5(a);

the corresponding

in Fig.

is given

output

5.5(b). We are interested in only two-dimensional pictures. We assume that the bottoms
lie on a fixed line (i.e., they
share
a common horizon). Building
of all the buildings
Bt is
represented

by a

and

the

in

5.5(a)

Fig.

(2,6,7), (3,13,9),

(1,11,5),

denote

Rt

the

height.

building's

to

correspond

and

the left

them arranged

connecting

heights

the buildings

example,

//, denotes

and

respectively,

building,

coordinates

/?,). L, and

//,,

(L,,

triple

the

order

in

of the

x coordinates

right

A skyline is
from left to

list

right.

of
For

input:

following

and
(12,7,16),(14,3,25),(19,18,22),
(23,13,29),

(24,4,28).

(The

numbers

in

boldface

type

are the

heights.) The skyline

in Fig.

5.5(b) is

represented

as follows:
(1,11,3,13,9,0,12,7,16,3,19,18,22,3,23,13,29,0).

(Again,

the numbers

The straightforward
time to the skyline.
know how to solve

in

The
the

algorithm

are heights.)
for this problem is

based on adding

induction

hypothesis is

simple

boldface

type

problem

for

-1

the

and

buildings,

one.

then

one

building

We assume

we add

the

nth

that

at a
we

building.

15

10 15 20 25 30

15

(a)

Figure 5.5 The

20

15

10

(b)
skyline

problem:

(a) The

input,

(b)

The

skyline.

25

30

5.6

The

for one

is trivial

problem

intersectit

from

to

left
to right
x coordinate

skyline
appropriate

the one

5 is

\"covers\"

a building

add

To

building.

skyline (see Fig. 5.6).

the existing

with

from

this example).
to 9, and its height

we need to
We first scan the
(i.e., we search for the
the horizontal line that
the skyline,

Bn be (5,9,26).

Let

left side

the

where

find

\342\200\224
5 in

to

Bn

103

Problem

The Skyline

Algorithm:

Divide-and-Conquer

of

fits

Bn

this

In

is 13.

case,

We can

scan

now

the

skyline,

line after another,

and
whenever the height
of Bn is
looking at one horizontal
adjusting
we reach
an x coordinate that is greater
higher than the existing height. We stop when
side
of Bn. For this example,
than the right
we do not adjust
from 3 to 9, but
the height
we do adjust it all the way
from
it
once
more
from
22
9 to 19, then
to 23. The new
adjust
is represented

skyline

by

(1,11,3,13,9,9,19,18,22,9,23,13,29,0).

This

algorithm

scan for

is clearly

correct,but

requires

0{n)

To

called

the performance of
and conquer. Insteadof

for n

solution

not necessarily efficient.

total
number
Hence, the

this

improve

divide

(Again,

it is

steps.

In

the

worst

of

steps

case, the
be

will

\342\200\242\342\200\242\342\200\242
= <9(\302\2532).
+<9(1)

+ 0(/i-1)+

0(/i)

the

Bn

the base

- 1 to a solution

case of

one

for

building

we

algorithm,

use a

the simple induction

using
we
extend
a solution for
\302\253,

is

well-known

technique

principle of

extending

n/2 to a solution
trivial.) Divide-and-conquer algorithms

for

n.

divide

each subset recursively, and

the
inputs into smaller subsets, solve
(conquer)
merge
solutions together. Generally, it is more efficient to divide the problem
into subproblems
of about
size.
As we saw in Chapter
relation
3, the solution of the recurrence
equal
=
=
=
+
+
of
is
whereas
that
2T(n/2)
0(n) is
T(n)
T(n-\\)
0(n)
T(n)
T(n)
0(n2),
the

T(n) = 0(n logn).

subproblems,

if

Therefore,
combine

the

we

divide

solutions

in

the

of

two

into

problem

equal-sized

the algorithm runs

time,
and we will
useful,
very

linear

The divide-and-conquer technique is

0(n\\ogn).
examples

then

in
see

time
many

it.

Bn

10

15

Figure5.6 Addition

of a

building

15

20

(dotted line) to the

skyline

25

30

of Figure

5.5(b) (solid lines).

104

of

Design

The

key

in this
is the
the divide-and-conquer
algorithm
example
with
the
it takes linear time to merge one
case,
building
In
same
we
about
the
time to merge two different
time,
skylines.

behind

idea

in the

that,

observation

by Induction

Algorithms

worst

skyline, and also linear

Two skylines can be merged
with
the
the latter
achieve more using
basically
approach.
a
We
scan
one building into
the
two
that
same algorithm
(Fig. 5.7).
skyline
merges
x
match
and
when
from
left
to right,
coordinates,
adjust
heights
together
skylines
in linear time, and
therefore
the complete
necessary. The merge can be achieved
in
case.
This
is similar
in time
the
worst
to
runs
O(n
algorithm
logn)
algorithm
in Section
in
6.4.3.
we
do
not
is discussed
detail
the
which
Therefore,
give
mergesort,
here.
algorithm
precise algorithm for the skyline
Comments

technicalabout

this

required, consider

to get

try

Always

algorithm includes a
to a more complicatedpart of the problem.
The
is so useful
is that it uses the combine
approach
step to its
in Section 3.5.2 cover the most
common
dividegiven
should
memorize
these recurrence relations.

this

a binary tree

Let

T be

the

farthest

leaf

step

Balance

Computing
with

the

down

is nothing mysterious or
step that is more general than

There

money.

your

If the

principle.
applying

reason the divide-and-conquer

fullest. The recurrencerelations
and-conquer algorithms. You

5.7

more for

root

r.

tree.

Factors in Binary Trees

The height
The

balance

of a node v is the distance between v

factor of a node v is defined as

and
the

the height of the node's

of the node's right
left subtree and the height
subtree
are labeled
or
In Chapter
left
(we assume that the children of a node
4,
right).
by
we discussed AVL trees, in which all nodes have balancefactors
of -1, 0, or 1. In this
we consider arbitrary
trees.
section,
binary
Figure 5.8 shows a tree in which each node is

difference

labeled with
factor.

between

numbers

representing

hlb, where

10

15
Figure

is the

15

5.7 Merging

node's

25

20

two

skylines.

height

and

30

b is

its balance

5.7

Figure 5.8

balance

in trees

nodes

The base caseof

(by

now know

rather on

their

the

know

the

problem

the

within

original

Stronger

induction
of all

the base

Again,

factor

Furthermore,
the two

The
computing

be an

all

of

remain.

that

on only

of

the root,

we remove

n > 1 nodes,

with

depends

We

chose to

then

remove

the nodes belowthat

node.

The root's

nodes, except
but
factors of the root's
children,
induction does not work in this case. We need to
the height-finding
the root. The idea is to include
for

root.

the

balance

the

on

hypothesis:

We

that

trees

in

trivial.

difference

the
the

to compute

height

balance factors

nodes.

we consider

when

Now,

how

know

have <

the

root,

we can

determine

the heights of its

\342\200\224
of the root
it is the maximal
between

its

children.
of

height

1.

plus

is that it solves
a slightly extended problem. Instead of
key to the algorithm
turns
only balance factors, we also compute heights. The extended
problem
easier

one

to solve,

solving a stronger problem is

of a small problem to a solution
problem

simple

easily by calculating
we can also determine

children

balance factors

hypothesis.

problem:

nodes

case is

to compute

of all the

factors

children

the

and heights
balance

out to

of

the

induction

subtrees

factor of a node

Hence,

height.

heights

tree

the two

for

balance

the

how

Given a

induction)

the balance

because

root

nodes, compute

straightforward

the

with

know

balance factor, however, depends not

the

height and b is

nodes.

< n

have

that

n = 1 is trivial.

the problem

solve

T with n

tree

inductive approach

Induction hypothesis: We

We

is the

105

Trees

Binary

the nodes.

of all

factors

the regular

use

Given a binary

Problem

The

the

in

factor.

balance

We

represent hlb, where h

The numbers

tree.

binary

Balance Factors

Computing

is extended),

then

because the

easier.
of
the

With

a larger
induction

heights
induction,

problem.
step

are easy to
we need

If

may be

the

compute. In many
only to extend a

solution

is broader

easier, since we

have

cases,
solution

(because
more

with

Design of Algorithms

106

which to work.

is a

It

by

5.8
The

one

forget

should be

is from Bentley

problem

there

that

are two

computed separately.

different

in

parameters

will

We

several

present

book.

in the

Consecutive

the Maximum

Finding
following

error to

common

this problem, and that each

examples of such errors later

Induction

[1986] (it

also

appeared

Subsequence
in Bates and

Constable

[1985]).

The Problem

necessarily positive)

negative,

then

subsequence is 0).
the sequence

The

in

subsequences

order

only

like to

Induction hypothesis:We
sequences of size <n.
If

n =

1, then

nonnegative,

the

maximum

or

the

is empty (by

have

an

definition,

that

algorithm

the

solves the

sum

sequence

(2,

is 3.5.

There

numbers

are

of the empty

problem and

reads

once.
induction

straightforward

subsequence. For example,in the

is (1.5, -1, 3); its sum
subsequence
in a given sequence. If all the

subsequence

would

We

numbers

the

(not

xi+l, ...,jcy- (of consecutive

in it is maximum over all

elements.

maximum

the

maximum

the

of

a maximum

a subsequence

3, -2, -3, 3),

be
several
maximum
may

-3, 1.5, -1,

xh

subsequence

of consecutive

subsequences

We call such

find

the sum

that

such

elements)

of real numbers
jc,, jc2, -~,x\342\200\236

a sequence

Given

hypothesis

is as

how

to find

know

subsequence
empty

S = U,,*2, \342\200\242\342\200\242\342\200\242,-*,,)
of size
n>\\.

consists

of

follows:
the

the

maximum

single

otherwise.

subsequence

in

subsequence

number

if that number is

Consider

sequence

how
to find a maximum
By induction, we know
If that maximum
subsequencein S' = (x,, x2,..., *\342\200\236_i).
subsequence is empty, then all the
numbers in S' are negative, and we need to consider
only xn. Assume that the maximum
found by induction
in S' is S'M = (xh xi+l,...,
subsequence
jcy), for certain / and j such that
1
1. If j = n- 1 (namely,
the maximum subsequence is & suffix),
then it is easy
^i^j<n-

to extend the
maximum.

solution

to S:

However,

if j

\\i

xn

< n-

is positive,

1,

then

there

then
are

extends
S'M\\ otherwise,
S'yv/ remains
o m remains
two possibilities. Either
is not maximum in S\\ but is maximum

it

maximum, or there is another

which
subsequence,
in S when
xn is added to it.
The
first
illustrate
key idea here is to strengthen the induction hypothesis. We
the technique by using it to solve the maximum-subsequence
then discuss it in
problem,
more
in
next section.
The problem we had with the straightforward
the
generality
induction hypothesis
that xn may extend a subsequence that
was
in S\\
is not maximum
and thus may create a new
maximum
subsequence.
Knowing
only the maximum
in S' is thus not sufficient.
However, xn can extend only a subsequence that
subsequence

5.9

ends

\342\200\224
that

n -1

at

hypothesis

of S'.

a suffix

is,

the

include

to

the Induction

Strengthening

Suppose that
of

knowledge

we

the induction

strengthen

maximum

the

107

Hypothesis

denoted

suffix,

by

S'e =(xk'Xk+ \\'\342\200\224'Xn-\\)Stronger

<n,a

a
If we

know both

maximum
It

could be

subsequences, the

global

of size

sequences

that is

subsequence

set as

The algorithm

maximum

new

done

need to find
the previous

xn to

to the

maximum

we have

we retain

We also

add

simply

xn

then

Otherwise,

suffix).

yet.

We add

subsequence,

the

new

previous

maximum

new

the

suffix.

maximum

It

suffix ending at xn is negative. In that case, it is better to take

that later xn+{
will
be considered
the maximum suffix
(such
by itself).
is given in Fig. 5.9.
subsequence
finding the sum of the maximum

maximum

the

that

empty

clear.

becomes

algorithm

than the

is more

sum

the

(as well as
subsequence
We are not
subsequence.
is not true that we always

maximum

suffix.

in

suffix.

suffix. If

the

induction
We know how to find,
hypothesis:
maximum subsequence overall,and the maximum

for

(X, n) ;

Maximum_Consecutive_Subsequence

Algorithm

Input: X (an array of size n).

Output: GlobalJAax (the sum

of the

maximum subsequence).

begin

GlobalJAax := 0 ;
:= 0

JAax

Suffix

fori := J to n do
ifx U ] + Suffix
Suffix

.= Suffix JAax

+ Suffix

Suffix

else

+ x [i ]

:= Suffix JAax

GlobalJAax

else ifx [i ]

then

> GlobalJAax

JAax

JAax

JAax

> 0 then
+ Suffix

JAax
:=x[i]

JAax

:= 0

Suffix JAax

end
Figure 5.9 Algorithm

5.9

induction

the

theorems

mathematical

proving
often

the

Strengthening

Strengthening

the

encounter

can

hypothesis

In many

easier. That

is,

it

can add

by

jOonsecutive_Subsequence.

Induction

Hypothesis

is one of the
hypothesis
with induction. When

following

be denoted

cases, we

Maximum

scenario.

Denote

and
\302\253),

the proof

P (<

another

is easier to

call

assumption,

prove [P

and

most
attempting

theorem

the

important
techniques
an inductive
proof,
by

must conclude
it

Q](<n)

Q,

under
=>

that

The

P.

for
we

induction

P(<n)=>P(n).

which the
than
P(n)

proof becomes
it

is

to

prove

Design of Algorithms

108

by

Induction

is not clear how we can prove it.

now
have
to prove that [P and
We
trick
Q
hypothesis.
The
but often stronger
than
theorem
P
is
a
and
and
P,
=>
just
Q
stronger
Q](n).
[P
Q](<n)
with
the right added
be
can
This
to
easier
and,
are
repeated
process
prove.
theorems
The
tractable.
becomes
maximum-subsequence problem is a
assumptions, the proof
of how this principle is used to improve
algorithms.
goodexample
is a well-known
A nice analogy to this principle
phenomenon: It is easier to add $1
to
of sales,
than it is to add $1 thousand
that are based on $100 million
to profits
million
of sales.
on
based
are
$10
that
profits
while
The most common error people make
using this technique is to ignore the
In other
and
to
was
added
additional
an
that
fact
forget to adjust the proof.
assumption
that
even
without
and
that
Q was
[P
noticing
Q](<n) =>/>(\302\253),
words, they
prove
suffix
to compute the new maximum
to forgetting
assumed. This oversight
corresponds
to
it corresponds
In the balance factorsexample,
in the
example.
maximum-subsequence
\342\200\224
is
a
error.
common
the
to
which,
unfortunately,
heights
separately
compute
forgetting
this fact:
We
cannot
overemphasize

/>( <

h)=> P(n).

The

in

the

will

more complicated
6.13.1, 7.5, 8.3, and

present

Sections

6.11.3,

5.10

Dynamic
that

Suppose

may be
the
silicon

chip,

hypothesisprecisely.

examples of strengthening
12.3.1

induction

hypothesis

Problem

The Knapsack

Programming:

in

others).

(among

given a knapsack and we want to pack it fully with items. There

and our only goal is to pack
items of different shapes and
sizes,
to a truck, a ship, or a
full as possible. The knapsack
correspond
may
of this
variations
the problem is to package items. There are many

different

as
and

problem; we consider only a simple one dealing

variations of the knapsack
are presented in
problem

The

the

we are

many

knapsack

it

induction

It is crucial to follow the induction

We

but

correct,

seems

assumption

is to include

Problem

that the

/th

item

sizes sum to

Given an
has

integer

an integer

exactly K, or determine

K and n

size kh
that

find

no

with

the

items.

one-dimensional

exercises,

and

in

Chapter

sizes
items of different
of the items
such subset exists.

a subset

Other
11.

such
whose

of items and K
K), such that n denotes the number
problem by P(n,
denotes the size of the knapsack. We will implicitly
that the n items are those
assume
in the
that are given as the input to the problem, and we will not include their
sizes
a
notation of the problem.
with the first / items and
Thus, />(/, k) denotes the problem
we
first concentrate
of size k. For simplicity,
on only the decision
knapsack
problem,
a solution
which is to determine whether
exists.
the straightforward
We start
with
We

denote

the

induction approach.

5.10

Induction hypothesis(first

The

Programming:

Dynamic

how to solve P(n

We know

attempt):

109

Problem

Knapsack

- 1,K).

if the single element is of sizeK. If there

only
there is a way to pack some of the n - 1 items
\342\200\224
then
we are done; we will
the knapsack
not use the nth item.
into
simply
Suppose,
for P{n-\\,
result?
Yes
however, that there is no solution
K). Can we use this negative
\342\200\224
that the nth item must be included. In this case, the rest of the items must fit
it means
the problem to two smaller
into a smaller knapsack of size K-kn. We have reduced
K
To
and
we have
to
P{n-\\,K)
P(n -1, -kn).
complete the solution,
subproblems:
We need to solve the problem
not only for knapsacks of size
strengthen the hypothesis.
The

case is

base

K,

also

but

easy; there is a

to P(n

is a solution

- 1,K)

for knapsacks

Induction

solution

\342\200\224
that

is, if

of all sizes at

hypothesis (second

most

K.

attempt):

We

know

how

to solve

P{n-\\,k)

forallO<k<K.
The

did not

reduction

previous

of K\\
value
depend on a particular
for
0<k
<K.
all
The
P{n, k)

it will

work for

any

k.

case P(\\, k) can

Otherwise, there is a
now
reduce
if
solution
P(n, k) to the two problems
only
and P(n- \\,k-kn). If k-kn
Both
<0, then we ignore the second
P(n-l,k)
problem.
a
now
can
be
solved
induction.
This
is
valid
and
we
have
an
reduction,
by
problems
a
of
the
be
inefficient.
We
reduced
size
n
to
algorithm
problem
algorithm; however,
may
of
in
of
also
the
value
k
one
two
size
n-\\\\
reduced
(We
subproblem.)
subproblems
Each of these two subproblems
leading to an
may be reduced to two other
subproblems,
We

can use

be easily

this

solved:

exponential

of

may

not

hypothesis

then there is always

the first item is of size k. We
If k=0,

a (trivial)

base

solution.

algorithm.

Fortunately,
kinds

to solve

problems.
be too

it is

possible in many cases

observation is that

The main

high.

In

fact,

we

introduced

to improve the running

time for these
of possible problems
the total number

the

notation

of

P{i,k)

especially

to

and K
demonstrate this observation.
There
are n possibilities for the first parameter
possibilities for the second one. Overall, there are only nK different possible problems!
The exponential running
of problems after every
from doubling the number
time
resulted
but if there are only
nK different
reduction,
problems, then we must have generatedthe
same
all the solutions and never
is to remember
problem
many many times. The solution
solve the same problem twice. This
a
of strengthening
the
is
combination
approach
induction
the assumption
that all
induction
(which is using
hypothesis and using
strong
solutions to smaller cases, and not only that for n - 1, are known). Let's see now how to
implement this approach.
We
all the known results in an n xK matrix. The (/, k)th
store
in the matrix
entry
contains the information
from the secondabout the solution of P(i, k). The
reduction
in the nth
attempt
hypothesis
computes the nth row of the matrix. Each entry
basically
row is computed from
two
of the entries above it. If we are interested in finding
the
actual subset, then
add to each entry
we
a flag
can
that indicates whether
the
item was selected in that step. This flag can then be traced back from the
corresponding
The
is given in Fig. 5.10.
and the subset can be recovered.
(\302\253,
K)th
algorithm
entry,
matrix for a given input.
Figure 5.11 shows the complete

of Algorithms

Design

110

(S, K);

Knapsack

Algorithm

Input: S (an

of size n

array

size of

K (the

and

Induction

by

the

storing

of the

sizes

the

items),

knapsack).

that P [/, k ].exist = true

such
P (a two-dimensional array
with the first / elements
exists a solution to the knapsack
problem
=
true
if the /th element
P
k
and
of
size
k,
[/,
].belong
knapsack

if there

Output:

to that

solution).

( See

Exercise

5.15 for suggestions about

and a

belongs

this program.

improving

begin

P [090].exist:= true
1 to K do

fork:=

P[0,k].exist :=false

P
need to initialize
P
[0,0]}
computed from

is no

{ there

be

fori := J to

[/, 0]

for / >

1, becauseit

will

n do

fork:=0toKdo

; {thedefault

P [/', k ].exist .-false

value

ifP[i-\\,k].existthen

:= true

P[i, k].exist

:=false

k].belong

P[i,

else if k-S[i]>0

then

[i ]].exist then
:= true ;

ifP[i-\\,k-S
k].exist

P[i,

P[i, k].belong:= true

end

5.10

Figure

\"I\":

*4=6

then

last line,

a solution

been found;

it 3 =5

An example

of four

*2=3

consists

5.11

Figure

*,=2

O
O

of

the

Algorithm Knapsack.

table

for the

constructed

items of sizes2, 3,5, and 6. The

containing this item has been found;

\"-\":

there

no

solution
is no

has not

yet

been

10

\"O\":

solution for a knapsack of this

12

13

14

15

16

knapsack problem. The

in

symbols

found.

11

the

table

a solution
(If the

size.)

without

symbol

input

are the following:

\"-\"

this

appears

item

has

in the

The method
programming.

we

used is

just

technique called dynamic

of a general

instance

an

programming is to

of dynamic

essence

The

111

Errors

Common

5.11

build

with

tables

large

all

The tables are constructed

Each entry is computed
iteratively.
previous
or
of
in
of
above
it
to
it
the matrix.
other
entries
the
left
The main
a
combination
from
of
in
the
matrix
most
efficient
is
to
the
construction
the
organize
way. Another
problem
results.

known

example of dynamic

is presented

programming

There are nK
Complexity
from two other entries.

exponential
approach
determining

too

not

Hence, dynamic
is not too large.

running

this

constant

of the

not

but

this

small

can work

the

subproblems.

enough,

computation

when

problem
All

for

using,
(n) time.

can

possible

a large matrix.
maintaining
total number of possible subproblems
by

Errors

we briefly mention

section,
We

have

a recursive
error

problem for n
Changing the hypothesis
of it. A graph
example
of the

into

partitioned

two

same subset. If the

the proof

of this

The
determine
accordingly.

some common errors in

already discussed common

have analogous errors here.

2.13. All those errors

is common. In the case of
recursion. Another
common

typical

is effective

approach

programming

We do

programming

Common

algorithms.

instance

in

sizes

the

If

only if the
and
Even then, dynamic programming requires building
matrices,
large
in Fig. 5.10, it is
a large space. (In some cases, as in the program
usually
requires
at any moment.) The
to use less space by storing
only a small part of the matrix
times
are usually at least quadratic.

5.11
In

O (nK).

subset, then we can trace back from the (\302\253,

K)th
entry,
the subset in O
and recover
the knapsack program,

are computed.

subproblems

it

is computed

one

each

time is

flag in

belong

The dynamic
Comments
be reduced to several smaller,

possible

and

running

then K cannot

large,

actual

the

finding

example, the

thus

table,

total

6.8.

Section

be too large and nK is much better than an

is
in
K
n.
(If
very large or if the sizes are real numbers, then this
expression
will not work; we discuss this issue in Chapter 11.) If we are interested only in
K ]. If we are interested
whether a solution exists, then the answer is in P [\302\253,

items are

in

in the

entries

Hence,the

time

in

+ 1,

For

induction

example,

procedure, a base caseis

is to extend a
instead of

an

for

solution

arbitrary

induction to design
in Section
proofs
the base case
forgetting
essential
to terminate the
to a solution of a special

use of

the

errors in

instance.

common mistake. Here is a

E) is called bipartite if its set of vertices can be
from the
subsets such that there is no edge connecting two vertices
is connected
the partition is unique
and bipartite,
then
(we omit
graph
unintentionally

is another

G = (V,

fact).

Problem
whether

Given a connected undirected

if it is, partition
and,
bipartite

it is

graph

the

G=(V,

vertices

\302\243),

of Algorithms

Design

112

solution: Remove a vertex

A wrong

induction. We call the

only red vertices,add
red subset. If

the

to

Induction

by

The main

after we have removed a

of the problem is
instance

used. Had

that does

if possible,

graph,

If

to

want

cannot be

graph, this solution

does not depend on

think
such
to

make
it

that we

form

can

find

other

simple

it

is included

unless

assumption

avoid

is sometimes very
tempting.
hypothesis
such
\"we know how to find
and such,\"

the
the

the hypothesis
changes to that black

changing

any

the

with

things

in

specifically

box,

same

effort.

the

induction

is to

unintentionally

think of

it

graph

similar

to

related

to redefine

(namely,

explicitly).

5.12

Summary

Several techniques for designing algorithms,

all of which
are
were
in
this
These
are
no
introduced
means
approach,
chapter.
by
for designing
the

would
the

If the hypothesis
is
to
then we are tempted
But we cannot use any
One way
hypothesis.
as a black box. Do not

are ready to open it

unless you

smaller

the

which
This problem has a better solution,
have
that
solution
to
the
reader
we
leave
connected;
(Exercise 7.32). For
being
of
this
common
see
Section7.5. A result
discussion
error,
example and further
in Exercise 5.24.
is included
this incorrect algorithm

something of

is that

illustrate,

induction

and

instance,

been valid.

Changing

by
to

is connected

be connected. Hence, the

disconnect

not

we

the one

graph may not

same as the original

the

not

a vertex

removed

we

both

and

solution,

the

blue.

to only blue vertices, add it

then the graph is not
subsets,

is connected

the

vertex

of

subset

second

the

to vertices from

is connected

the partition is unique).

in this attempted
error

(since

bipartite

the rest

partition

subset. If v

the blue

it to
v

and

red, and

subset

first

algorithms.
The rest of

presented

in

can

\342\200\242We

of

this

achieved,

and

main

idea

to one

learn

is devoted

numerous

to precisely

that

purpose.

The

principles

follows:
of

use the principle

a problem

The

are as

chapter

to

way

book

this

same

methods

known

examples are presentedin

these techniques is to use them
to solve
and

techniques

The best

chapters.

following

problems.

Additional

all the

the

of

variations

or

induction

more

of

to design

size.

smaller

algorithms
If the

by reducing
can

reduction

an

instance

always

be

case can be solved,then the algorithm

follows by induction.
concentrate on reducing a problem, rather
than
on solving it

base

the

is to

directly.
\342\200\242One

of the

elements.

easiest
That

ways

technique

to

reduce

the size

should

be the

of a problem is to

first

line

of attack.

eliminate

The

some
elimination

of its
can

do not
to simply eliminating elements that
many forms. In addition
clearly
contribute (as in Section
into one, to find
5.3), it is possible to merge two elements
elements
that can be handled
a new element
by special
(easy) cases, or to introduce
that takes on the role of two or more original
elements
(Section
6.6).
\342\200\242We
can reduce the size of the problem
in many ways. Not all reductions,
however,
As a result, all possibilities for reductions should
lead to the same efficiency.
be
In particular, it is worthwhile
considered.
to consider different
for the
orders
take

Notes and Further

Bibliographic

induction
element

have seen examples

We
sequence.
first. Sometimes,
of starting from

examples

the

middle

it is

where

to take

is better

it

the

6.2).

(Section

better to take the

first. We

element

smallest

113

Reading

We also

see

will

largest
will

see

of

examples

induction on trees in which the root is removed first (top down), and examples in
which
the leaves are removed first (bottom
up) (Section 6.4.4).
of the most efficient ways
to reduce
the size of a problem is to divide it into
\342\200\242One
(or more) equal parts. Divide
can be divided such that the output
two

for

output
Sections
\342\200\242Since

whole

the

6.4, 6.5,
a reduction

itself, we

and

works

conquer

of the

the

if

problem

generate

algorithms

Divide-and-conquer

problem.

8.2, 8.4, 9.4,and

can change only

effectively

subproblems can easily

are

given

the
in

9.5.
the

size

of the

problem,

but

not

the

problem

as possible.
subproblems
of finding some ordering among several items
For example, the problem
can
be
reduced to finding
the item that is first in the order; the relative
(and
removing)
of the first item (see Sections 6.4 and
is independent
order of the rest of the items
for smaller

look

should

are

that

as independent

7.5).

is one

\342\200\242There

be

must

important

very
the

hypothesis

the complete
\342\200\242

Finally,

that the reduced problem

way, however, to overcomethe limitation
to the original
the
problem:
Change
problem statement. This is a
it is better
to weaken
method that we will use often.
Sometimes,
and to arrive at a weaker
algorithm, which can be usedas a step in

identical

(see

algorithm

we can

example, we can use

induction

Section

6.10).

use all these techniques together,

hypothesis,

the

approach

divide-and-conquer

so that the

different

or in

subproblems

various

with
become

combinations.

For

strengthening the
to combine

easier

(see Section8.4).

Notes and Further

Bibliographic

Reading

in this chapter was developedby the author (Manber [1988]). It is

presented
means new. The use of induction,
and in general mathematical proof techniques,
in the algorithms
area has its origin in the flowcharts of Goldstineand von Neumann
(see
von Neumann
but
was
first
[1963]),
by Floyd [1967]. Dijkstra [1976],
fully
developed
similar
Manna
to
Gries
[1980],
[1981], and Dershowitz [1983] present
methodologies
ours to develop programs
with their proof of correctness. Their
approach
together
addresses
fashion
than the
program
design in a much more rigorous and detailed
in this chapter.
in Section 2.12, can be
described
The use of loop invariants,
presentation
considered,in some sense, to be equivalent to the use of induction in this chapter.
in algorithm
Recursion, of course, has been used extensively
design (see, for example,
The

method

by no

Burge [1975]and
is

Paull

The celebrity
to save
possible

verification

phase,

[1988]).

problem was
an

additional

questions

Thomas [1982]).Strengthening
Polya [1957]calls this technique

(see Rosenberg [1973]). It

not
to repeat,
careful
in the
by being
asked during the elimination
and
Smithphase
(King
induction
is probably
a very
the
old trick.
hypothesis
the inventor's
paradox (because it is easier to invent,
first

suggested

|_l\302\260g2wJ questions

by Aanderaa

114

Designof

by Induction

Algorithms

It is also sometimescalled generalization.

formalized by Bellman
It has
and
[1957].
programming
Dynamic
a
For
detailed
of
variations.
and
description
many
dynamic
numerous applications,
Law
or Denardo [1982]. The
[1977],
see, for example, Dreyfus and
programming
to Exercise 5.24 was pointed out to us by Tom Trotter.
observation
leading

or prove,

is

that

something

stronger).
introduced

was

Drill Exercises

5.1

evaluation.
algorithm for polynomial
your algorithm require? Can you think

a divide-and-conquer

Design
and

does

multiplications

How

many additions
this

of an advantage

algorithm has over Homer's rule?

inductive
that were used in Section
5.3 to solve the
reasoning
a graph G=(V, E), we are looking
induced subgraph problem: Given
induced
G' such that all the degrees in G' are at most k (as
for the maximal
subgraph
more difficult
5.3). This version is much
opposed to \"at least\" in the problem in Section
and the approach taken
for the original version does not work
than the original
version,
11 for a discussion of this problem
for
here. Discuss why it does not work. (See Chapter
the simple caseof k = 0.)

5.2

Try to follow the

maximal
following

5.3

Consider algorithm
end of the algorithm?

5.4

5.5

that the set S will become

(Fig. 5.3). Is it possible
Show an example, or prove that it cannot happen.

are

loop

space to

all

T is called

while loop in

that

efficient

algorithm

should
problem. Given a tree T, the algorithm
AVL tree. The answer
should
be only yes or no.

decision

5.6

algorithm

Modify
actual

subsequence

Maximum

Consecutive

and not

the sum.

5.7

Write a program to

5.8

In algorithm

Knapsack,

PV-Uj]).

If

Redraw

5.9

Fig. 5.11

of

solution

we

first

have
the

determine whether or

Subsequence (Fig. 5.9) such

to a knapsack

problem

using the

that

the

enough
following

not

T is an

it finds

the

belong flag.

whether the

checked

is a solution

to reflect
problem

rest? How
of Exercise5.7?

the

with

i-\\

choice.

this

may

the solution

from all the

the policy

do not
to solve

is unnecessary (by checking

/th item
We can also
items, we take this solution.
choice, which is to take the solution with the /th item if it exists (i.e.,
will have a better performance?
Which
version do you
think
first).

A given knapsack
characteristics

the solution

recover

there

make the opposite

check P[i,j-kj]

only

also Section 4.3.4)if

the nodes

Assume

at the

(Fig. 5.4).

Celebrity

algorithm

tree (see

AVL

an

are 0, 1, or -1.
factor. Design an

its nodes

balance

the

store

for the first

invariant

tree T.

a binary

given

factors of

balance

empty

Mapping

the appropriate

Write

You

steps of

are the special

What
algorithm
Knapsack?
separates this
to
your answer change if the choice is made according

have

obtained
does

many
from

different

solutions.

What

115

Exercises

Creative

Exercises

Creative

5.10 Solve the

skyline problem. Suppose that the buildings in the skyline

with
a triangular
roof on top. (You can assume for
that all the roofs have
all the
angles with the buildings.) Again,
45-degree
simplicity
an algorithm to draw the skyline in this case.
Design
buildings have a common horizon.
Each

5.11 Supposethat
screen

there

efficient

an

words, compute the

5.12 Let x,, jc2,

is a rectangle

building

a blue

with

Design

extended

following

have roofs.

on a
are two different
(maybe
proposed)
skylines: One is projected
on the first one with
a red color.
color, and the other is superimposed
to compute the shape that will be colored purple.
In other
algorithm
of two skylines.
intersection

be a sequence
..., x\342\200\236

algorithm to

the

find

of

product of the numbers

the empty subsequence is defined
5.13

real

xh
subsequence
in it is maximum

numbers

over

all consecutive

an O

Design

elements)

such

(n)
that the

subsequences. The product

of

as 1.

Suppose that a given tree is not an AVL tree. We call a node

to mark the nodes in
an algorithm
factor is 0, 1, or -1. Design
are AVL nodes.
all of whose
descendents

5.14 LetG = (V,

positive).

(not necessarily

*,+,, ...,*, (of consecutive

node if
are not AVL

an AVL
T that

its balance
nodes,

but

binary tree with n vertices. We want to construct an n x n matrix whose

the tree is undirected,
the matrix
to
the distance between
v, and vjm (Since
//th entry
such a matrix for a tree that is
an O (n2) algorithm to construct
will be symmetric.) Design
E) be a

is equal

given

5.15

in

the adjacency-list

representation.

Let G =(V, E) be a binary

path

connecting

maximal

distance

diameter of a given

tree.

these two

over

all

The distance

vertices
pairs

between two
have

(neighbors

of vertices.

vertices

distance

Design a

linear-time

is the length of the

diameter of G is the

in G

1). The

algorithm

to rind the

tree.

5.16

in algorithm
Knapsack (Section 5.10). Is there
Improve the space utilization
complete n x K matrix? What is the space complexity of the improved algorithm?

5.17

Solve

a need

for a

to
are identical
of the knapsack problem: The assumptions
the following variation
those of Section 5.10,exceptthat there is an unlimited supply of each item. In other words,
the problem is to pack items of given sizes in a given-sized
knapsack, but each item may
appear many times.

5.18 Here is another

are the same as in

now
each item has an
but
knapsack),
supply,
associated value. Design an algorithm
to find how to pack the knapsack
fully, such that the
items in it have the maximal value
all possible ways to pack the knapsack.
among
Exercise

5.19 Hereis the

variation

5.17 (n items,

of the

unlimited

knapsack problem: The

assumptions

fixed-sized

common variation of the knapsack

problem: The assumptions are the same
5.17 (n items with sizes and values, unlimited
fixed-sized
supply,
knapsack,
the value), but now we are not restricted
and the goal of maximizing
to rilling the knapsack
We are interested
to capacity.
value,
only in maximizing the total
exactly
subject to the
constraint that there is enough room for the chosen items in the knapsack.
most

as in Exercise

116

Design of Algorithms

5.20

Let x i, x2,..., xn

by

be a

set of

Induction

let S =

and

integers,

to partition

an algorithm

\302\243a,.Design

the

/=i

set into two

subsets of equal
(nS).

sum,

or determine

that

it

to do

is impossible

so. The algorithm

run in time O

should

box \342\200\224
you cannot see how it is designed
you are given an algorithm as a black
If
and an
the
has
numbers,
you input any sequence of real
following properties:
is
there
a
subset
will
or
whether
of
answer
the
\"no,\"
indicating
\"yes\"
algorithm
integer k,
to find the subset
how to use this black box
sum is exactly k. Show
the numbers whose
n is the size of
sum is k, if it exists. You should use the black box O (n) times
whose
(where

5.21 Supposethat
\342\200\224
that

the

5.22

sequence).

The towers of

Hanoi

solution.

recursive

puzzle

There

example of a nontrivial
sizes arranged
5.12).
(see
pegs,
Fig.

is a standard

are

that has a

problem

of different

disks

on

a peg

in

simple

decreasing

There are two other empty

The purpose of the puzzle
move all the disks, one at a time, from the first peg to another peg in the following
way. Disks are moved from the top of one peg to the top of another. A disk can be moved
of
to a peg only if it is smaller than all other disks on that peg. In other words, the ordering
sizes must be preserved at all times. The goal is to move all the disks in
disks by decreasing
as few moves as possible.
order of sizes.
is to

a. Design

an algorithm

towers of

b.

Hanoi

(by induction) to
for n disks.

a minimal

find

How
moves are used in
many
number of moves,and solve it.

your

algorithm?

Write

5.24

The

a nonrecursive

following

assume

is a

movesthat

Construct a recurrence

c. Prove that the number of moves in part b is optimal;

any other algorithm that uses less moves.

5.23

sequence of

the

solves

problem

program for the

variation of the
disks are

towers
towers

that

is, prove

that there cannot

of Hanoi

problem (defined

of Hanoi

problem (see Exercise5.22). We

in Exercise

the

for

relation

exist

5.22).
no

peg. They may be arbitrarily distributed

among the three pegs, as long as they are ordered in decreasing sizes on each peg. The
purpose of the puzzle remains to move all disks to one specified peg, under the same
constraints as the original problem, with as few moves as possible. Design an algorithm
to
find a minimal sequence of moves that solves this version of the towers
of Hanoi problem
longer

for

that

all the

on one

initially

n disks.

Figure

5.12

The towers

of Hanoi

puzzle.

Creative

5.25

in Section

described

assignment

The problem
very

the

difficult

coloredwith
a. Prove
b. We
find

by

that

within

an

that trees

induction

a partition

connected

the

one subset.

bipartite.

to vertices
the

global

Consideragain

the

and

then

oldest possible color, and

old colors. Prove

of all the

connections,

we

that maximizes

the

may

need

is bipartite).
are no

want to

We

edges connecting

algorithm for determining

vertex, remove

wrong

5.11: We take
color the vertex in

in Section

with the

vertex

regardless of

are always

means
that the graph
the graph is a tree (which
of the vertices into the two subsets such that there

a graph is bipartite, given

the rest (by induction),

color

algorithm for determining whether a graph is bipartite,

that not only is the algorithm
sense, this exercise shows

colors.

two

assume

vertices

some

approach

simple

Given

coloring:

graph

wrong

cannot work. Considerthe more general problem of

undirected graph G=(V, \302\243),a valid coloring of G is an
such that no two adjacent
of colors to the vertices
vertices have the same color.
as few colors as possible. (In general,
is to find a valid coloring, using
this is a
if it can be
11.) Thus, a graph is bipartite
problem; it is discussed in Chapter

but also

wrong,

to the
5.11. In

is related

exercise

This

117

Exercises

an arbitrary
the
add

that,

best possible
a new
if we

That

way.

color only
color one

whether
color

it,

is, we

vertex is
time
You should

if the
vertex

colors.
up to 1 +log2/i
of colors for every
order

at a

of choosing
on
order
in
The
the
the
depend
following way.
algorithm
vertex
and starts checking the vertex's
edges. At that point, you
picks a vertex as a next
that the graph
are allowed to add edges incident to this vertex as you desire, provided
You
number of colors will be required.
remains a tree, such that, at the end, the maximal
which has
the algorithm,
cannot removean edge after it is put in (that would be cheating
this construction is by induction.
to achieve
seen
the edge). The best way
already
<k colors with few vertices, and build
Assume that you know a construction that requires
new vertices.
without adding too many
one that requires k + 1 colors

design

a construction

vertices. The

construction

can

number

CHAPTER

INVOLVING

ALGORITHMS

SETS

AND

SEQUENCES

is a

Order

on disarray
teaching

lovely thing;
it lays its wing,
simplicity to sing.

Anna

6.1
In

Branch

(1875-1937)

Introduction

this

we deal

chapter,

difference between
elements

not

Since

sequences.

which

sets

are either

is

in sets

finite

it

sequences
is not.
Also,

the order
in

sets. The

or finite

sequences

in

that

sets

in

we assume

once, whereas there is no such assumption

in some order, we can regard
usually
given
we may call an input a set when we are not interested

the

that an

than

more

so

the

which

for

inputs are

integers, reals),
in

that

and
whereas

this
given order. Throughout
the input is assumed to be an
The elements in the sequences

problems

inputs

important

appear

Nevertheless,

sequences.

with

sequences

are given is

element does

(e.g.,

Hempstead

unless

chapter,
and

array,

or sets

that

elements

they

are

specified otherwise,

we assume

are assumedto
can
all

that

be

the

taken

In
be compared.
of the same type.

size

the

We

of

is known.

array

ordered

totally

chapter,
study

in the

representation

of the

from a
this

as

them

set

we consider
issues

such

as

data
and similarities
of sequences.
compression,
maximality, order, specialsubsequences,
different
This chapter contains
with
a
of
algorithms
many
applications.
variety
Our purposeis to give more examples of the design methodology
introduced in Chapter
some important algorithms. We include
to describe
5, and, at the same time,
algorithms

are

that

and universally
are very

important

very

and Sets

Involving Sequences

Algorithms

120

and

search

(binary

applicable

for

sorting,

specific applications (file

important
algorithms
example),
that
are not very important but
and
and
algorithms
comparisons),
compression sequence
in
a set,
and the
two
elements
the
largest
techniques
(finding
illustrate
interesting
that

have

but

stuttering-subsequence problem).
first

The

in many forms

comes

that

two

compression,
We end this

and

examples of

several

with

and

\342\200\22

sorting

statistics,

data

algorithms.

probabilistic

elegant algorithms

algorithm

discuss

then

\342\200\224
order

problems

text manipulation,

involving

problems

chapter

We

situations.

algorithmic

and elegant

\342\200\224
a basic

search

binary

in many

appears

extensively studied

the most

of

one

in this chapter is

example

interesting

illustrating

design techniques.

6.2

is to

search

Binary

immensely
question.

algorithms

cut

search

the

this

In

a wheel

what

rediscovered

it is

and

important,

is to

search

and Variations

Search

Binary

in half

space

mechanics:

is

It

The

basic

(or approximately

so)

frequently.

several

we describe

section,

is to

of

variations

binary

elegant, and
idea behind binary

simple,

by

one

only

asking

and show

search

its

versatility.

Pure

Search

Binary

The Problem
that

<xn.

<jc2<

jc,

Let A-,,*2

* \342\200\242

in the sequence,

z appears

such
be a sequence
of real numbers
,
z , we want
to find whether
real number
and, if it does, to find an index / such that
\342\200\224
xn

Given

Xj=Z.

For

we look for

simplicity,

interested
idea is to

in

search

the

cut

simplicity

that n is

sequence;

otherwise,

nil,

which

can

comparing

z to

number

to the
the

of

z is
element.

we

version

advantage is

half

the

second

xnl2+\\,

that

that

and

This version

we may be

In general,

one,

middle number.
is clearly in the

so

and

first the
z

then

half. Finding z

We handle

algorithm

.v,=z.

the largest

one,

by checking

by induction.
The

/ such

smallest

comparisons

is given

in

in

either

half

the

base

case

Fig.

6.1.

on.

Assume,

first

half of

The
for
the

is a problem of size
of n = 1 by directly

therefore,
of size n with

half;

comparisons
binary search delays the equality
equality with z in each step. The disadvantage of
there
is no hope for stopping
the
the
search
present is that
early;
is made in every step (instead of one equality
one
comparison
only
one inequality comparison). This searchis thus usually
faster.
Although

end. The alternative

comparison

in

is 0(logA?).

search

binary

space

in

index
the

Each time a comparisonis made,

is cut by one
the range
required to find a given number in a sequence

Complexity

the

one

indices,

even. If z is less than

be solved
the

only

all such

finding

is to check

of

6.2 Binary

Input:

(X,

Binary_Search

Algorithm

X (a

sorted array

Output: Position

that

= z,

X[i]

search key).

z (the

to a?), and

range

/ such

index

(an

121

z);

n,

in the

Search and Variations

or 0

index exist).

such

no

if

begin

Position := Find(z,

1, n);

end

function Find (z, Left, Right): integer

begin

if Left

= Right

then

ifX [Left ] = z then

else Find := 0

:= Left

Find

else
Middle

f/2(Left+Right)]

:=\\

ifz<X[Middle]then
Find := Find
else
Find := Find

(z, Left,

Middle-1)

(z, Middle,

Right)

end

Figure 6.1

Binary_Search.

Algorithm

more convenient to write the program as a recursive program, we

is not as effective for small
search
nonrecursive program. Binary
for large ns. If n is small, then it is better simply to search the sequence

it is

it to a
is

in a

Search

Binary

some
sequence
in increasing order.
is a, for

minimal element

Given
in

as

it

linearly.

the

be cyclically sorted if the smallest

the sequence xhxi+\\,...,xn,x{,...,

/, and

unknown

The Problem

values

convert

of n

Cyclic Sequence
is said to

x\\,x2,...,xn

sequence

can easily

sorted
a cyclically
list (we assume, for

list,

find

the

simplicity,

position

that

this

number
a*,_j

in

the

is sorted

of the
position

is unique).

To find the
eliminate half
that

k <m.

minimal
the

If xk

whole sequence.

element jc, in
with one

sequence

then
<x\342\200\236n

(Noticethat

/ cannot
we

the

we use

sequence,

comparison. Take any

be

cannot

in

the

range

exclude

k <i<m,

xk.) On the

the

of

idea

numbers

two

since
other

xt

hand,

binary search to
xk and xm, such
is minimal
in the
if xk >xm,

then i

122

must

be

Thus,

one

find

i in

Cyclic_Binary_Search

Algorithm

somewhere

is switched

order

eliminate many elements.

n)
comparisons.The algorithm
(log

we can

comparison,

we can

appropriately,

since the

range k<i<m,

in the

with

and Sets

Sequences

Involving

Algorithms

By

is given

cyclically sortedarray in the range 1 to a? of distinct

Position (the index of the minimal element in I).

Output:

in

range.

and

6.2.

Fig.

n,z);

(X,

X (a

Input:

that

in

choosing

elements).

begin

:= CyclicJ

Position

n);

ind(\\y

end

Cyclic Jind

function

(Left, Right): integer

begin

if Left

then

Right

:= Left

Cyclic Jind

else
Middle := ['A (Left + Right)] ;
ifX [Middle ] < X [Right ] then
Cyclic Jind := Cyclic Jind
else

CyclicJind := Cyclic

(Left, Middle)
(Middle+1,

Jind

Right)

end

Figure 6.2 Algorithm

for a Special

Search

Binary

the following
search problem,
index that satisfies a specialproperty.

In

The

a i,

Problem

the

Given

a2>..., an, determine whether

Cyclic

Jearch.

Jinary

Index
key

is not

a sorted
there

given; instead, we are

sequence
exists

distinct

of

an index

such

looking

for

an

integers
that a{ -i.

applicable here, becausethe value of the searched element is

the property we seek is adaptableto the binary search principle.
However,
given.
Consider the value of anl2 (assume again that n is even). If this value is exactly nil, then
we are done.
the
Otherwise, if it is less than nil, then, since all numbers are distinct,
value of an/2-\\
so on. No number in the first half of the
is less than nil - 1,and
sequence
we can continue searching the
and
can satisfy the property,
The same
second
half.
is
holds
if
the
answer
The algorithm
is given in Fig 6.3.
\"greater than.\"
argument
Pure
not

binary

search

is not

Binary Search

6.2

Input: X (a sorted

in the

array

Output: Position (the

such index exists).

index

123

(A, n);

Special_Binary_Search

Algorithm

and Variations

range 1 to

of distinct

integers).

] = Position,

A [Position

satisfying

no

or 0 if

begin

Position:= Special_Find(\\,

n)

end

function SpecialJFind (Left, Right): integer

begin

if Left = Right
if A [Left]

then

= Left then

else Special_Find

SpecialJind

:= 0 { unsuccessful

:= Left
search

else

Middle :=\\'/2{Left+Right)\\
if A [Middle ] < Middle then

Special_Find

:= Special_Find

(Middle +

Special_Find

:= Special_Find

(Left, Middle )

1,Right)

else

end
Figure 6.3 Algorithm

in Sequences

Search

Binary

use a

we

Sometimes

Special_Binary_Search.

of Unknown Size

procedure much like binary

search

the search
but suppose that the

to double

space rather

size of the
problem,
the search range, since we do not know its
look
for an element jc, that is greater than or equal to z. If we
find
in the range 1 to i. We first
such
search
an element,
then we can perform
binary
<
z to jc i. If z x j, then z can only be equal to jc i. Assume,
compare
by induction, that we
; > 1. If we
know that z >
we
double
the
search
z to
then
for
some
space
j
compare
jc,jc2y,
with one comparison. If
and
we can find z with
then
we know
that
jc, <z<jc27
z<xlr
O
then it
(logy) additional comparisons. Overall, if i is the smallest index such that z<xh
takes 0(log/) comparisons to find an jc, such
that
and
another
0(log/)
z<jc,,
than

to halve

it. Consider

sequence is unknown.
boundaries. Instead, we

comparisons

to

find

the

We

regular

cannot

search

halve

i.

same algorithm can also be used when

the size of the sequence is known,
but
we suspect that / is very small. This algorithm
is an improvement
over regular
binary
time is O (log /) rather
search in such cases becauseits running
than
O (logn).
However,
there is an extra factor of 2 in the running time of this algorithm,
since we perform two
The

binary

search

like procedures.

Therefore, this

algorithm

is better

only

when

i =

0(yn).

Algorithms

124

and

Sequences

Involving

Problem

Stuttering-Subsequence

The

Sets

that do not seem to require any

search appears even in problems
and B be two
of characters
from a finite
alphabet
sequences
'' ' an and\302\243=/?j/?2 ' ' ' b\342\200\236n
We say that\302\243 is a subsequence
that m<n.
of
such
A = axa2
are indices /, < i2 < ''' <im such that, for all j, \\<j<m, we have bj=dj. In
A if there
B is a subsequence of A if we can embed B inside A in the same order but
words,
other
B is a subsequence
of A. We scan
holes. It is simple to determine whether
with
possible
from there until
we find b2,
we find the first occurrence
A until
(if any) of b\\, continue
and we leave it as
and so on. The proof that this algorithm is correct is easy by induction,
involves
one linear scan of A and B, its running time is
Since the algorithm
an exercise.
a
B with each
Given
we
define B' to be the sequence
#,
sequence
clearly 0(m+n).
The

of binary

principle

search.

Let

character appearing

times

For

consecutively.

example,

if

B=xyzzx,

then

B3 =xxxyyyzzzzzzxxx.

The

Problem

value of

This

problem

such

Given

that B'

is called

the

two

A and

sequences

B,

find

the

maximal

is a subsequence of A.

problem.

stuttering-subsequence

first, but it can be solved easily with

For each given value of /, we

binary

It may seem

difficult

at

search.

can construct

B' easily. Hence, we can

if
value of /. Furthermore,
Bj is a subsequence of ,4, then
B' is a subsequence
of A, for 1 </ <j. The maximal value
of / that needs to be considered cannot exceed nlm, since in that case the sequence
B
would be longer then A. So, we can use binary
search.
We first set i=\\nlm\\l2, and
check
whether
Bl is a subsequence of A.
We
then
continue
with
search,
binary
the
lower
if
the
is
answer
the
It will
and
otherwise.
eliminating
range
yes
upper range
take \\\\og2(n/m)\\
tests to determine the maximal
is thus
/. The overall running
time
determine

whether

B'

is a subsequence

of A

for

the

sequence

any

specific

= 0(n\\og(n/m)). Sequencecomparison
also
are
0{{n+m)\\og{nlm))
problems
discussed in Section
6.8.
This solution suggests a general technique.
we are looking for the
Whenever
maximal
/ that satisfies some property,
that
be sufficient to find an algorithm
it may
determines whether
a given
i satisfies that property.
We can do the rest by binary search
if we
have an upper bound
/ satisfies it,
is such that,
whenever
for /, and if the property
then j satisfiesit, for 1 <j </. If we do not know an upper bound for /, we can use the
scheme.
That is, we can start at / = 1 and double the value of / until we find the
doubling
will
This
take longer, but, unless the desired
search
/ is extremely large, it
right range.
will
still be efficient. The resulting
In many
not
be optimal.
algorithm,
however,
may
extra
the
it
is
to
eliminate
cases, such as the stuttering-subsequence
problem,
possible
O (log n) factor.

6.3
Equations

Solving

area doesnot

This subject
here.

mention

(i.e., a<x<b),

f (a)

that

and

is negative).

one

we

which

can

<0 (i.e., one

to

find a

of

the

f (a)
to

solution

that

x is

the

the

in

equation

/is

[a, b ]

range

is positive

and f(b)

short

(a) = 0, where

equation

given

it deserves a

but

chapter,

to

We are

compute.

want

this

a solution

to find

want

'f(b)

We

subject of

to the

conform

that we

Suppose

function

a continuous

other

125

Search

Interpolation

the

and

within

[a, b].

We

given

precision.

Since

of

precision),

required

or

as bisection
at

evaluated

function/is

then we have

solution.

in the range

exist

must

solution

known

search,

binary

follows. The

works as

is continuous,

function

the

a variation

use

Bolzano

the

=(a+b)/2.

a*j

= 0 (within
select one

If/(jtj)
we

Otherwise,

can

can

which

method,

the

of

the

one
half the size of the original,
in which a
or Uj,/?], each being
is guaranteed
to exist. The selectionis done
such
that the values of the function
solution
at the other. We continue
in this way until
the
are positive at one end and
negative
After
k
the
size
of
the
that
a
is
contains
solution
achieved.
desired
steps,
region
precision

[0, Jtj]

subranges

\\$(b-a)l2k.

6.3
In

Search

Interpolation
the search

search,

binary

However,

performance.
number

search

z,

if

space is always
the

during

we

search

find

which guarantees the logarithmic

a value
that is very close to

to

reasonable

more

seems

it

in half,

cut

in

search

the

continue

the

that

if z is very
In particular,
to the next half
\"neighborhood\" instead of blindly
point.
going
of the sequence instead of
start
the search somewhere in the beginning
small, we should
at

the

point.

halfway

Consider
number.

the

Say

200 is

thus

the

around

where to

we

way

about
page

is at about
200,

that

the

now

We
80

the

of

one-fifth

a book

which

be larger than z.
X14]. The algorithm,
in

Fig.

are

percent

way.

we can

We can

flip

search to a range
page 1 and

cut the

mark between
one

this

continue

page

interpolation search. Instead of cutting

amount that
seems
the most
likely
interpolation,

we

when

searching

for a certain

page number is 200 and the book looks like an 800-page

the one-fourth
mark, and we use this knowledge as an
We will probably not hit page 200 on the first try;
book.

open the
we get page 250 instead.
page

open

at a
the

process

of

250

250.

search

exactly

space by a fixed

half,

we

of

suppose
the

to

close
the

page
Page

indication

pages, and
We now try

until we get

time. This is

book.

that

desired

go back

enough

to

idea

behind

cut

it by an

is determined
to succeed. This amount
by
in Fig. 6.4. The first guess is at X [8], which turns out to
leads to X[5], and then another finally leads to
Another
interpolation
the precise expression used for the interpolation,
is given
including
is illustrated

6.5.

The performance of interpolation

search
not only on the size of
depends
on the input itself. There are inputs
for which
also
search
interpolation
checks every number
in the sequence
search
(see Exercise 6.4). However,
interpolation
of relatively
is very efficient for inputs
consisting
uniformly distributed elements (the
Complexity
the sequence,

but

and Sets

Sequences

Involving

Algorithms

r&

-r--&-

,OH

X[i]

2 3 4 5 6 7 8 9 10 11 12

13

14 15 16 17

6.4

Figure

(X, n,z);

Interpolation_Search

Algorithm

Input:

(a sorted

Output:

array

Position

in

the

to \302\253),
and

range

/ such

index

(an

search.

Interpolation

that

= z,

X[i]

z (the

or 0

if

search key).
no

such

begin

ifz < X

[ 1]

or z

>

{ unsuccessful
else Position

[n ] then

search

:=

:= 0

Position

}
1, n)

Int_Find(z,

end

function Int_Find(z, Left, Right): integer

begin

= z then
else if Left = Right
IntJFind := 0
ifX [Left ]

:= Left

IntJFind
or X

[Left ] =

[Right

] then

else
Next

Guess

:=

-(z-X[Left])(Right-Left)
Left +
X

ifz<X[Next_Guess]

IntJFind :=

[Right]-X

[Left]

then
IntJFind

(z, Left,

Next

Guess

\\)

else

Int_Find

end

:= Int_Find

(z, NextjGuess,Right)

Figure 6.5 Algorithm

Interpolation

JSearch.

index exist).

6.4

127

Sorting

a book
It can be shown
that the average
are, of course, uniformly
distributed).
where
the
is taken
number of comparisonsperformed
search,
by interpolation
average
this
seems
to
an
be
order
of
over all possible sequences,is 0(loglog\302\253).
Although
over
the
of
to
search
the
extra
(due
improvement
performance
binary
magnitude
search is not much better than binary search in practice
for two
interpolation
logarithm),
n
is
unless
the
value
of log2\302\253is small enough that the
main reasons. First,
very large,
of it is not much smaller. Second,interpolation
search
more elaborate
requires
logarithm
arithmetic.

pagesof

6.4
Sorting

Sorting

many

of the

is one

basis for

most extensively

algorithms,

many

of sorting

As

creasing

order.

l<ii,/2,

...,/*<> i, such

For simplicity,
unless
the methods described
algorithm

is called

that holds

the

6.4.1

Given
other

In

x ,*2.\"

n numbers

that

words, find a
< ''
<x

in

section

this

in-place

if

no

We

them

\302\273-*fl\302\273
arrange

of

on the

concentrate

problems.

will

into

go

in inindices

distinct

**i.

assume

are valid for

additional

we

usual,

sequence

jc^

specified otherwise,we

computing

sorting problem, and

even a small part of this

section

other

the

is

It

time for

the

of

subject.

The Problem

of

proportion

variations

algorithms. We cannot cover in this

We mention only several
common
techniques.
the algorithms
that can be useful
for
principles behind
more detail than usual in this section.

dozens

science.

in computer

problems

a large

are numerous

There

applications.

typical

studied

it consumes

and

that

the numbers

nondistinct

work space is

are

used besidesthe

All

distinct.

as well.

numbers

initial

sorting

array

elements.

Bucket

Sort

Perhaps

the

simplest

number

of

''boxes\"

and

sorting
\342\200\224
we

Radix Sort
technique is
call

them

the

\"mailroom\"

buckets

\342\200\224
and

sort:
put

a sufficient
element in
are letters

allocate
each

the

bucket. This method is calledbucket sort. If the elements

and
be sorted accordingto states, for example, then allocating one bucket per
state
is sufficient
and the resulting
is very efficient. On the other
hand, if the
algorithm
letters
need
to be sorted by zip codes (with 5 digits), then
this method
requires 100,000
boxes and a very large mailroom. Thus, bucket sort works very well only for elements
from a small, simple range that is known in advance. A more detailed description of
bucket sort follows.
all of which are integers in the range 1 to m.
We
assume
that there are n elements,
We allocate m buckets, and then, for each /, we put jc, in the bucket corresponding to its

corresponding
need to
they

128

At

the

complexity

of

value.

Sequences and

Involving

Algorithms

end,
this

we scan

the

buckets

algorithm
On the
algorithm.
simple

Sets
and collect all the elements.
0(m+n). If m=0(n), then we
if m is very large relative to n (as

in order

is obviously

other hand,
linear-time sorting
then
with
O(m) is too large. In addition,
be the case
zip codes),
is
an
even more serious problem for large m.
0{m) storage,which
extension of this idea is radix sort. Consider
A natural

the algorithm

the

zip-code

The
get a
may

requires
example

because
the range of zip codesis
again. Using bucket sort for zip codes is not effective
We use induction
to
reduce
the
on the
Can
we
do
handle.
to
too
range?
something
large
we
10
buckets
We
use
several
use
and
sort
in
the
First,
stages.
following
way.
range
of
the
code.
Each
bucket
now
to
the
first
covers
zip
only
digit
according
everything

digits). The running

10 buckets, each with
At
the end of the first
for
time
to a smaller range. We can now solve the problem for each
elements
corresponding
Since we reduce the range by a factor of 10 in each stage, and since
bucket
by induction.
Once the buckets are sorted,
have
5
5 stages will be required.
all
codes
it
digits,
only
zip
to the
into a sorted list. We leave the details of this algorithm
is easy to put them together
of the same idea. We note
reader (Exercise 6.5), since we want to show another variation
that
can
be
In
the
the range
divided in any convenient
way.
zip-code example,the range
is divided according to the zip codes'
decimal representation. If the keys are strings of
characters that need to be sorted in a lexicographic
order, we can considerone character
at a time, leading to a lexicographic sort. Both
are similar.
The version of
algorithms
radix
sort presented
here (namely, a left to right scan) is known
as radix-exchange
sort.
A straightforward
recursive
of
sort
implementation
radix-exchange
requires
buckets
50 buckets will be needed
in the zip-code example;see also
(about
temporary
Exercise
Another way to achieve radix sort is to apply
in the opposite
the induction
6.5).
order. That is, the sorting is done from right to left, starting with the least significant
are large integers
parts instead of the most significant
parts. We assume that the elements
represented by k digits, and each digit is in the range 0 to d - 1. The induction
hypothesis
is the straightforward
one.
10,000

zip codes
this stage is O (n).

different

Induction

hypothesis:

(correspondingto

We know

how

the

to sort

four
remaining
we have
stage,

elements

with

< k

digits.

The difference betweenthis method

and the previous radix-exchange sort is the way we
the hypothesis. (This idea of applying induction in the opposite order is similar to
the
one
for Horner's
rule in Section
we first ignore
k digits,
Given elements with
5.2.)
the
most
to the rest of the digits
by
significant
digit and sort the elements
according
induction.
We now have a list of elements sortedaccording
to their k - 1 least
significant
digits. We scan all the elements
digit,
significant
again and use bucket sort, on the most
with
d buckets.
is called
This algorithm
in order.
Then, we collect all the buckets
to k
sort. We want to argue that the elements are now sorted according
straight-radix
digits.
We claim that two elements that are put in different buckets in the last step are
arranged in the right order. We do not even need the induction hypothesis for this case,
since, by the lexicographic ordering, the most
significant
digit is the one that determines
the order regardlessof the other digits. On the other hand, if two elements have the same

extend

6.4

most significant
then, by the induction
digit,
we
have to make sure
Thus,
step.

are in the

they

hypothesis,

the last

that

129

Sorting

before

order

right

order.

This is

the
they stay
right
of
the
is
and
it
a
of
the
use
of
the
inductive
part
algorithm,
good example
only
is correct.
to make sure the algorithm
It is essential that elements
that are put in
approach
in
remain
the
This
can
same
bucket
same
order.
be
achieved
a queue for
the
by using
the d queues at the end of a stage to form
one
each bucket, and by appending
global
to the / least significant
The precise
digits).
queue of all elements (sorted according
algorithm is given in Fig. 6.6.

the

in

subtle

X (an

Input;

(X, n,

Straight_Radix

Algorithm

Output:

integers,

each

in sorted

order).

of

array

(the

array

k);
with

k digits,

in the range

in a

global queue

to n).

begin
{

that

assume

We

use

We

all elements

are

simplicity;

it can

GQfor

fori := 1 toddo

{d is

number

the

initialize

for

Q [i ]

queue

X }

through

numbers }

of decimal

case

do

empty

xfrom GQ ;

pop

d := the

ith

of x

digit

insert x into Q[d]

I tod do
fort:insert

for i :=

to be empty

GQ ;

I do

is not

while GQ

be implemented

digits; d = 10in

of possible

i := k downto

initially

GQ ;

into

Q[t]

1 to

n do

from GQ

pop X[i]

end
Figure6.6 Algorithm
It takes

Complexity
the

initialize

pops each element

the Q [i ]s together.
In

the

they

make

The

compared.

The

main

overall

pushes

it

of the
into

one

time of the

without

the whole

about

the elements

in

the

algorithm

the

types

regard to

the

is executed
It

is O

except

also

k times,
all

concatenates

use direct
keys. Each
more general since
two
can be
elements

techniques

that

of their

are
that

d steps to

(nk).

\"structure\"

key. These algorithms

of elements,

GC, and

queue

algorithm, which
of the Q [/]s.

section, we consider sorting

elements

involve

no assumptions

loop

running

of this

remainder

thus

to put all

steps

GC and

from

comparisons betweenthe

comparisonwill

Q[i].

queues

Radix.

Straight

130

Involving

Algorithms

and Sets

Sort and Selection

Insertion

6.4.2

Sequences

Sort

induction. Suppose that we know how to sort n -1 numbers

use a straightforward
\342\200\224
1 numbers
and then put the nth number
We can sort the \302\253
numbers.
n
are
we
and
given
until
the correct place to insert is
n
1
numbers
the
sorted
correct
in its
place by scanning
It is simple and effective
insertion
is
called
sort.
This
found.
procedure appropriately
for large n. In the worst
is
not
efficient
it
an
n.
of
However,
for small values
algorithm
1
to
all
n
numbers.
The total number of
the
the nth number is compared
previous
case,
=
as
numbers
l+2+-+\302\253-l
n
for sorting
may be as high
comparisons
the nth number in its correct
inserting
place
i/2(n- \\){n-2) = 0(n2). Furthermore,
in the nth
involves moving other elements. In the worst case, n - 1 elementsare moved
is also 0(n2).
We can improve
insertion
hence, the number of elementmovements
step;
in
an
search
on
the
-1
and
n
sorted
elements
the
sort by storing
array,
using
binary
to
insert.
The
search
takes
correct
the
find
numbers to
only
0(\\ogn)
comparisons
place
the number
of data
to 0(n \\ogn) comparisons overall. However,
leading
per insertion,
so this is still a quadratic-time
movements remains
algorithm.
unchanged,
the straightforward
induction by selecting a special nth number.
We can improve
is a
number
as the nth number.
The maximal
For example,
we can select the maximal
\342\200\224
to
the
end
we
know
where
it
it
at
of
choicebecause
the
The
put
belongs
array.
good
the maximal,
it in the right place (by
then putting
algorithm consists of first selecting
it with whatever is there), and then recursively
swapping
sorting the rest. This algorithm
is called
of selection
sort over insertion
is that only
selection sort. The advantage
sort
n - 1 data movements (swaps in this
case) are required versus O (n2) in the worst case for
insertion
On the other hand,
sort.
since
it takes n -1 comparisonsto find the maximal
element (finding
the
is discussed in Section
maximal
the total number
of
6.5),
is
whereas
insertion
sort
with
search
0{n2),
comparisons
always
binary
requires only
We

O (n

\\ogn) comparisons.
also

is

It

Chapter 4).
Scanning

an

by induction

to do is to

possible
AVL

Using

AVL
that

tree
we

to use

balancedtrees for

trees,

to get a list
how to

for example,
of its numbers

know

build

an

AVL

insertion

efficient

each

insertion

in order
tree for

or selection

requires

takes O (n) time.

n

- 1
numbers,

0(\\ogn)
If we

(see
time.

assume

then all we need

time to insert n

which takes O (\\ogn)

it takes O (n \\ogn)
time.
Overall,
an empty AVL
and O(n) time at the end to list them in sorted
order.
tree,
For large \302\253,
this
is a much better solution
sort or selection sort, but it
than
insertion
more space to hold the pointers.
It is also
It is clearly not an in-place
requires
algorithm.
and
it
is
not
as good as the algorithms
next. The programs
we present
quite
complicated,
insert,

into

numbers

for insertion
6.4.3

To

and

numbers

the efficiency of insertion

to find the correct

place for many

sets of numbers
merge

sorts are

simple and

are

left as

exercises.

Mergesort

improve

sorted

selection

involves

we notice that in the time it takes

to scan the
sort,
place to insert one number, we can find the correct
We have already used this idea in Section 5.6. If we have two
numbers.
that are already sorted, we can merge
them
with one scan. The
together
the numbers of the second
set in order and finding
the correct
considering

6.4

place in

set for

first

the

precisely, denote
that both

assume

b{ is

to insert

place to
The

insert

about data

What

It

the

and
b\342\200\236n
right

scan

is inefficient

to move

will be

moved many

elements

same

so

b2,...,

set until

first

the

elements each time

times.

More

on.

and

b\\,

from that place until

bs are sorted, we never have to
case, is the sum of the sizes of
the

worst

the

in

Scan

order.

continue

smallest,
set by

the second

\342\200\242..,
an, and

Sincethe

and so on.

movements?

performed, since the

right

go back.
sets.

the

is

insertion

an

since the

Instead,

place

the

merge

we copy them to a temporary

one by one in sorted
order,
array;
is copied exactly once. Overall,
two
sorted
n
of
sizes
merging
sequences
and
data
can be done with
movements
that
0{n+m)
comparisons
(provided

producesthe

elements

element

each

and

additional

storage

is available).

The merge procedure

sorting

and-conquer

First,

the

parts.

is divided

sequence

part

sorted

one

An example

6.7.

that

algorithm,

Second, each

mergedinto
Fig.

it; then

of comparisons,

number

total

is found,

a2,

in increasing

insert

and

the second

number,

set by a\\,

sets are sorted

found,

b2

smallest

the

first

the

131

Sorting

into

is sorted

two

can be used as a
The

(in case

or close-to-equal

equal

recursively.

separately

basisfor

the

Third,

worst case. Let's assume,

need to solvethe following

be
for

the

of comparisons

number

that n is

simplicity,

recurrence

required

a power of

2.

a divide-

as follows.

works

algorithm

of an

odd

size)

sorted parts are

is given in
algorithm
two

sequence, as described above. The precise

in Fig. 6.8 (the copying is not
of mergesortis shown

Let T(n)

Complexity

we just described
known
as mergesort.

by

shown).

in the

mergesort

T(n), we

To calculate

relation:

T{2n) = 2T(n) + 0{n), 7(2)=1.

The

of this

solution

asymptotically

sort.

data movements
Although

drawbacks.

relation

the 0(n2)

is
running

T(n)
time

= 0(n

\\ogn)

required

(see

Chapter

for insertion sort

3), which is

or selection

movements is also 0(n log\302\253), which is more than the O(n)

by selection sort.
required
is better than insertion sort for large \302\253,
it still has several
mergesort
the merging
is not as easy to implement.
Second,
step
mergesort
is not an in-place
to copy the merged set. Thus,
mergesort
storage
of data

number

The

recurrence
than

better

First,

requires additional
versions
of mergesort that use only
algorithm. (There are more
complicated
amounts of extra storage; see the bibliography
This copying must
section.)
are
which
makes
the
time
two
smaller
sets
every
merged,
procedure slower.

6.4.4

constant
be

done

Quicksort

demonstrate the efficiency

of divide and conquer. If we can
two
the problem
into
equal-sized subproblems,solve each subproblem
separately,
and combine the solutions,
we can get an O (n \\ogn) algorithm, provided that the division
was the need for
step and the combining
step take O(n). The problem with
mergesort
is arbitrary and we cannot predict where each element
extra storage, since the merging
will end up in the order. Can we somehow
a different divide and conquer
so that
perform
is to spend most of
the position of the elements
can be determined? The idea of quicksort
the
in the divide step and very little in the conquer step.
effort
Mergesort

divide

and

its analysis

(X, n);
in the range

Mergesort

Algorithm

Input:

array

(an

X (the

Output:

and Sets

Sequences

Involving

Algorithms

in sorted

array

to n).

order).

begin

M_Sort(l,n)

end

procedureM_Sort(Left,

Right)

begin

if

Right

{ checking

correctly

Left =

1 then
this
case is not
for
but
it makes
anyway,

ifX[Left] > X[Right]

else if Left * Right then
Middle :=

then

it

handled

be

will

more efficient}
(X[Left], X[right] )

the program
swap

+ Right)]

l/i (Left

\\

because

necessary,

M_Sort(Left, Middle-l) ;
;

M_Sort(Middle,

Right)

{ we

the two

now

merge

sorted sequences above into

sorted

one

sequence}

i:=Left;

j := Middle
k := 0 ;

(i <

while

k:=k

Middle -\\)and
+ J;

(j<

Right)

do

ifX[i]<X[j]then

TEMP[k]:=X[i] ;

i :=i

else

TEMP[k]:=X[j];

if j

> Right

then

the rest of the left side to the end of the

then
the right side is already in
Middle,
{ifi>
:=
t
0
to
-\\-ido
Middle
for
{ move

{ we

copy

TEMP

[Left+

back into

do

:=0tok-1

fort

-\\-t];

[Middle

[Right-t]:=X

now

t]:=

TEMP [t]

end

Figure 6.7 Algorithm

Mergesort.

array
the

}
right

place }

6.4

133

Sorting

10

12

15

13

11

16

14

(2)

(6)

10

12

15

13

11

16

14

(5)

(8)

10

12

15

13

11

16

14

(5)

(6)

<\302\253)

10

12

15

13

11

16

14

(9)

do)

12

15

13

11

16

14

15
(\342\226\2402)

13

11

16

14

(0

15
(\342\226\2402)

13

11

16

14

15

13

11

16

14

13

11

16

14

11

16

14

11

16

14

16

14

(2)

10 (0

(.0)

(9)

\302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251

10

12

10

12

10

12

10

12

13

15

10

12

13

15

10

12

13

15

10

12

\302\251 \302\251

15

\302\251 \302\251

\302\251 \302\251 \302\251 \302\251

\302\251 \302\251

11

\302\251 \302\251

\302\251 \302\251 \302\251 \302\251

\302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251

\302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251 \302\251

an exchange

either

illustrates

current

of mergesort. The first row is

operation or a merge. The
are circled.

An example

6.8

Figure

operation

Supposethat
or

than

equal

partition

can be

partition
divide

the
trivial

accomplished without

We can
step.
the two parts

now

sort

additional

each

space,

subsequence

already occupy
space is required.
Thus far, we have assumed that we
since

elements are

the

correct

as

will

be

in

the

greater

can compare all

to the answer. This
We

x.

- 1
comparisons. Since the two parts are equal in
half of the array
the other the second half.
and

first

the

of the

one-half

requires

occupy

the

partition

that

are involved

that

elements are smaller than

into two parts
according
sequence

row

Each

order.

initial

numbers

of the

one-half

and

elements to x and

a number x such

know

we

to x

in the

part can

one

size,

this

Furthermore,

shown

shortly.

recursively. The
in the array.
positions

This is

combine step is
no

Therefore,

additional
It is

easy to see, however,

that

used

for the

to

the array into two

less than or
numbers

with

partition.

We

partition

following algorithm.
left side of the array

call

the same

algorithm

value
will

of .v, which we usually

no matter which

work

the number used in the partition

one with numbers greater
parts,

equal to

We use two
and

the

know

R points

the

pointers

to

the

We

pivot.

to the
right

side

the pivot.
the

than

can achieve

array, L and
of the array.

R.

this

pivot

not.
is

Our purposeis
and the other
with the

partition

L points to
\"move\"
pointers

Initially,
The

do
number

the
in

134

Induction
that

At step

hypothesis:

and pivot < Xj

i <L,

of

the correctness

guarantees

invariant)

The hypothesis is trivially

Our goal is to move either

The

other.

each

toward

opposite directions

Sets

Sequences and

Involving

Algorithms

the

all j

for

of

induction

following

partition.

the

such

algorithm,

pivot >x^ for

all i such

j >R.

that

beginning (since no / or j satisfies

right or R to the left at step k + 1 without

true

at the

L to

the

(or loop

hypothesis

the hypothesis.

the

conditions).
invalidating

is almost completed except possiblyfor xL, with which

When L = /?, the partition
There are two cases.
If either xL<pivot or
L<R.
that
we deal later. Let's assume
can
move
and
the
the
then
>
corresponding pointer(s)
hypothesis is preserved.
pivot,
xR
<
>
In
we
and
this
can
we
have
case,
Otherwise,
xL
pivot
exchange
xL with xR
xR
pivot.
inward.
Both cases involve the movement
of at least one of the
and move both pointers
meet and the algorithm
will terminate.
hence, the pointers will eventually
pointers;
with the last
and
We are left with the problems of choosing a good pivot
dealing
in
the
two
meet.
the
which
of
algorithm
pointers
Divide-and-conquer
algorithms
step
when the parts have equal sizes, which
that the closer the pivot
is to
best
work
suggests
the middle, the faster the algorithm runs. It is possible to find the median of the sequence
in the next section), but it is not worth the effort.
As we shall
(we discussmedian
finding
see in the analysis, choosing a random element from the sequence is a good choice. If
the sequence
is in a random order, then we might
as well choose the first
element
as the
pivot. We make this choice, mainly for simplicity, in the algorithm
presented in Fig. 6.9.

Algorithm Partition (X, Left, Right)

X (an array), Left (the left
Input:
(the

and

Middle

and

of the

boundary

array), and

boundary).

right

Output:

X [j ]

>X

[Middle

such that X[i]<X [Middle] for

] for all j > Middle.

all i < Middle

begin

:=X[Left]

pivot

L ;= Left; R :=
whileL <Rdo

Right

X [L]

while

[R ] >

ifL <

then

exchange

Middle
exchange

:=

L < Right

< pivot and

while

pivot and
X [L]

>

X[R]

X [Middle

with

do

Left do R

:=L +

X [Left ]

with

end
Figure

6.9

Algorithm

1;

= R-1 ;

Partition.

Right

6.4 Sorting

the

When

step of the
We

the

from

policies

can be

sequence

exchanged

6.9 canbe used.

An

number

The circled

(6).

we can

exchange

of

are

numbers

those

is given in
that have

Fig.

the

with

the

6.10.

xL at

the last

as required.

partition

then

been

just

it

case,

any

and

first element,

the

with

Partition

of algorithm

example

pivot,

will put the pivot in the middle

in the complexity discussion. In

which

other

mention

is chosen as the

first element

partition,

135

any pivot

chosen

algorithm

in Fig.

The pivot is the first

After three

exchanged.

exchanges, ph points to X [6] = 1, and pt points to X [7] = 12. The last exchange involves
this exchange,
the middle point
(1) and the pivot (6). After
everything to the left of the
is
less
than
or
to
and
the
to
it. The two
it,
equal
right is greater than
everything
pivot
1
to
6
be
and
from
7
to
can
sorted
is thus
16)
Quicksort
subsequences (from
recursively.
an

of it

The

of the

selection

Fig.

of quicksort

time

running

pivot. If

the

depends on the particular

the sequence into

10

12

15

13

10

12

15

13

0
4

12

15

15

15

of an

array around

6
6

\302\251

Figure

Algorithm

and an example

12

\302\251

12

\302\251

6.10

Partition

14

13

16

14

10

13

16

14

10

13

16

14

the

pivot

Q_Sort(l,n)

Right);

begin
then

Left, Right);

Q_Sort(Left, Middle
Q_Sort(Middle

\\)

+ 1, Right)

end

Figure

6.11

parts,

16

end

< Right

and on

begin

Partition(X,

equal

14

Quicksort (X, n);

the range 1 to n).
in sorted order).

procedureQ_Sort(Left,

input
two

16

Input: X (an array in

Output: X (the array

if Left

6.11,

partitions

always

pivot

\302\251

in

6.12.

in Fig.

is presented

Complexity
the

The algorithm for quicksort is given

algorithm.

in-place

Algorithm Quicksort.

6.

136

Algorithms

Sequences

and Sets

10

12

15

13

11

16

14

(?)

12

15

10

13

11

16

14

CD

(?)

12

15

10

13

11

16

14

0)

(?)

12

15

10

13

11

16

14

12

15

10

13

11

16

14

11

10

13

15

16

14

11

10

13

15

16

14

10

13

15

16

14

13

15

16

14

15

16

14

16

\302\251

\302\251 \302\251

\302\251 \302\251

\302\251 \302\251

selected
two

each

in

it is

pivots

then

obviously

T(n) = 0(n \\ogn).

weaker
conditions.

the
the

then the

eliminate

comparing
(namely,

among
be 0(n2)

first

partition

pivot is

A new

input.

number

appears between

is

requires

sequence is already

- 1 comparisonsand

in

order,

increasing

places

and we

only

the pivot

always

select

in

the

T(n) =
-

case, because there is

analyze

that each

and

of

the

for the

comparisons

n-i.)

running

If each

time is

that

that the pivot is

this worst

the

smallest

case occur is

very

jcf-

has

same probability of being

ith smallest element is the

the

if the

selectedas the
pivot

is

+ 7(/i-i).

n-\\ +7(i-1)

a chance

still

likelihood

case.

this

running time 7(a?) of quicksort

The

takes

the running

then

pivot,

worst

assume

We

average

initial

a single

position.

sequence. However, the

We now

will

the

the

in

in the

element

sizes

is the

\302\251

time of the algorithm

is 0(n2).
We can
the
case
worst
for
that
are
sorted
or
almost sorted by
quadratic
sequences
the first, last, and middle elements,and then taking the median of these three
the second largest) as the pivot.
An even safer method is to choosepivots
from
the elements in the sequence
at random. The running
still
of quicksort
time
will
as

first element

pivot.

14

7(/i) = 27(/i/2) + 0(/i), 7(2)=1, which

implies
see that we get an 0(n \\ogn) running time even under
much
then
However, if the pivot is very close to one side of the sequence,
much higher. For example,if the pivot is the smallest elementin the

We

If the

place.

right

small.

is

time

running

sequence,

right

relation

recurrence

the

the

in

\302\251 \302\251

\302\251 \302\251 \302\251 \302\251

An example of quicksort. The first line

line.
The pivots are circled. When

6.12

Figure

\302\251

\302\251 \302\251 \302\251 \302\251

\302\251

\302\251

\302\251 \302\251 \302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251

\302\251 \302\251

\302\251

\302\251

!\302\251 \302\251

\302\251

\302\251

1\302\251 \302\251

\302\251

\302\251 \302\251

(It

Involving

partition,
element has the

and

we

same

to sort two smaller

sequences
of being selected, then
probability
need

of
the

137

6.4 Sorting

= \302\273-l +
-\302\243(T(i-l)
\302\273
/-i

r(\302\273)

\302\273-1+

+ T(/i-i))

n;-1)
\302\243

r(n
\302\243

-i)

\" ,=0

This is a recurrencerelation
Section 3.5.3, and its solution
is indeed quick on the average.

with

was

full history.
We
shown there to be

discussed

this

T(n) = 0 (n

in

relation

particular

log\302\253).

Hence,

quicksort

reason
its name. A major
for
In practice, quicksort is very fast, so it well deserves
and conquer, is that
elements
are
divide
its quickness, besides the elegant
many
the same element
(the pivot). The pivot is thus stored in a register and
compared against
In most computers, this saves
from memory.
there is no need for a data movement

time.

considerable

is to use a technique
we call
One way to improve the running time of quicksort
The idea is to start the induction
not always
choosing the base of the induction
wisely.
is called
until
the base case, which
from 1. Quicksort,as describedabove,
recursively
such as insertion
consists of sequencesof size 1. However, simple sorting
techniques,
well
for
small
whereas
of
sort or selectionsort, perform
the efficiency
sequences,
very
for
we
can
define
base
case
for
the
Therefore,
quicksort shows only
large sequences.
1 (it seems that 10 to 20 is a good size, but
that
than
quicksort to be of size larger
sort.
and handle
on the specific implementation),
the base case by insertion
(In
depends
- Threshold\"
other words, we replace the check \"if Left < Right\"
by \"if Left < Right
and add an \"else\"
This change leads to an improvement
sort.)
part which runs insertion
of the running time of quicksort
In Section
we will see how
a
small
constant.
6.11.3,
by
to use the principle
of selecting the base of the induction
to improve asymptotically the

of an algorithm.

running

time

6.4.5

Heapsort
is another

Heapsort

quicksort for large

performance

fast sorting algorithm. In practice,

it is not much slower. On the

but
\302\253,

is guaranteed.

Unlike
0(n \\ogn).
we emphasizeone

Like mergesort,
heapsort is an

mergesort,
of heapsort
part

\342\200\224

building

the

it is usually

not

other

unlike quicksort, its

time of heapsort is

worst-case

in-place

hand,
running

sorting

the heap.

The

algorithm.
algorithm

quite

as

In this
for

fast

as

section,

building

the

should
be interleaved.
heap
way design and analysis of algorithms
4.
in
were
discussed
We
assume
here
an
Chapter
Heaps
implicit
representation;
in an array A [\\..n], which
to a tree in
specifically, the elements are given
corresponds
the following
tree
is
the
of
root
stored
The
in
A
and
children
the
of
node
[1],
wayi
any
A [i ] (if there are any)
in A [2/ ] and A [2/ + 1 ]. Such an array satisfies the heap
are stored
is greater
than or equal to the values
if the value of each node
of its children.
property
illustrates

the

138

and Sets

Sequences

Involving

Algorithms

as follows.
in
The input is an array A [l../i].
First, the elements
Heapsort works
how
build
a
will
discuss
to
form
We
later.
If
a
to
A
are
heap
the array
heap.
rearranged
We
A
of
the
with
A
A
is
maximal
element
the
then
[1]
[n ]
[1]
array.
exchange
is a heap,
We then consider the array A[\\..n-\\].
so that A [n ] now contains the correct element.
to worry
to form a heap (we have
only about the new
Again, we rearrange the array
with A [l../i -2]. Overall,there
is one
A [n
with
1], and continue
A [1]), exchange A[\\]
and rearranging the
elements
a heap, and n - 1 stepsof exchanging
initial step of building

is basically the same

as algorithm
an
the heap after
exchange
Rearranging
heap.
a
is
an
4.3.2.
in
Section
Remove Maxjromjieap,
Building
heap
given
interesting
below.
time
and
it is described in detail
of
on its own,
Overall, the running
problem
of
the
time
the
plus
running
algorithm
per exchange),
(0(\\ogn)
heapsort is 0(n \\ogn)
The algorithm
the heap. Heapsort is clearly an in-place
for building
sorting
algorithm.
for

is given

heapsort

in

6.13.

Fig.

Heapsort (X, n);

the range 1 to n).
in sorted order).

Algorithm

Input: X (an array in

Output: X (the array
begin

BuildHeap (X); {seetext

for i :=

below

2 do

downto

swap(A[\\lA[i]);

Rearrange Heap (i-l)

the

basically

in

same

as RemoveJAaxJrom

procedure

Heap

4.7 }

Fig.

end

Figure 6.13

a Heap

Building

We now

Heapsort.

Algorithm

on the

concentrate

Given

Problem

The

order,

problem of building

rearrange

the

elements

an

array

a heap

of elements

A [\\..n]

so that

the

from an arbitrary

satisfies

array

in

an

array.

arbitrary

the heap

property-

There

ways to

are two natural

respectively

to

scanning

right to left. Figure

the use of induction.

build

the array

6.14
We

illustrates

then

a heap

\342\200\224

top

representing
both

and bottom up. They correspond

or from
either from left to right
with
We first describe both methods

down

the

methods.

show that there is a

heap

substantial

difference

in performance

6.4 Sorting

Figure 6.14 Top

betweenthe

and bottom

up heap construction.

left to

(corresponding

methods.

two

Consider

down

139

scanning

the

from

array

Induction hypothesis(top down):The

right

A[\\..i]

array

to top

is a

down).

heap.

The basecase is

since A [1] by itself

is always a heap. The main
of the
trivial,
part
is
A
A
is
to
into
the
same
as
this
the
But,
algorithm
incorporate [/+ 1]
exactly
heap [1../].
A
+
A
into
the
is
to
its
and
[/ 1]
1]
[/+
inserting
heap (see Chapter 4).
compared
parent,
are made until the new parent is larger. The number
of comparisons
in the
exchanges
case

worst

is

element A[i].
to a collection
by

],

A[\\..n

now

array from

the

scanning

to say

like

would

We

+ 1 )J.

|_ log2(/

Consider

the

that

to

right

array

A[i + \\..n]

left

is a

A[i+l..n]

does not

to bottom up).
adding the

(corresponding

heap

to

and

consider

to one heap; it corresponds

of heaps. (Note that we consider A [i + 1..n ] as part of the tree represented
and not as an array by itself.) Therefore, the induction
is slightly
hypothesis
the

But

array

correspond

more complicated.
Induction

A [i +

A[n]
however.
trees

the

array

itself

The

We need to

trivially.

that

is also

satisfy

trees representedby

condition.

the heap

is obviously
a heap, so the base case is satisfied. We can do better,
whole array A [[n/2\\ + \\..n] represents leaves
in the tree.
Hence, the
+
A
to
are
all
so
\\..n]
corresponding
singletons,
[[n/2\\
they satisfy the heap
by

property
hint

up): All the

(bottom

hypothesis
\\..n

start

the

induction

process

only

at

|_fl/2_|.

This

is a good

approach may be better. After all, half the work is trivial.

of selecting
another example for the importance
the base of the induction
the bottom-up

(This

with

care.)

Consider

now

serving as roots to
heap

It has

A[i].
valid

is straightforward.

heaps
A

[i ]

at most

(by the

two

induction

is compared

children
hypothesis).

(A[2/

to

the

maximal

+ l]

and

A [2/]),

both

A [i] into
Incorporating
of its children, and,

if

140

and Sets

Sequences

Involving

Algorithms

with the larger child. This is similar

necessary, it is exchanged
down the tree until
continue
The
exchanges
(see Chapter 4).

reaches

place

illustrated

the

hence,

comparisons;

of

overestimation

the

time,

> XLlog2/J

il\\og2i\\

,=1

height

is

A[i]

[\\og2(n/i)\\,

is

of

number

the

at most |_log2'J ^l^g2n\\

The /th step
requires
time is 0(n\\ogn). Moreover, 0(n\\ogn) is not an
shows.
as the following
argument

running

running

of

construction

2|_log2(\302\253//)J.

down)

(top

Complexity

>\302\253/2Llog2(\302\253/2)J

Q(/ilog/i).

i=n/2

most twice

the

comparedwith
complexity is at
evaluate

//(/),

look at complete

Let's
in

the

tree of height
i, and

of this

binary
complete
construction
heap

trees

each

in

//(0) = 0.

tree of

height

is

2/+l

-1,

it

is at

step

the

Since

of

the complexity

of

sum

the

a recurrence
i - 1 and a

(by induction) that

number
of nodes

verify

that

follows

trees of height

of two
can

We

recurrence is //(i) = 2/+l-(i+2).

We

/.

height

i consists

by //(/)
can derive

denote

and

first,

binary tree of

complete

noting that a

root. Hence,//(/) =2//(/- l)

solution

involved

comparisons

node
the corresponding node (since each
may have to be
the
the tree).
its two children,
Therefore,
exchanged, and so on down
We want to
of all nodes in the tree.
most
twice the sum of the heights

heights of all nodes

for

of

number

of

height

sum.

this

The

Up)

(bottom

Complexity

relation

case is

worst

the

in

the

of A[i]

value

old

the

A bottom-up

its children.

both

than

6.15. Since

in Fig.

comparisons

it is larger

where

in a heap

a deletion

to

the
in

bottom-up

2k -1
trees
is O(n) for complete binary
nodes).
(namely,
heaps with
The complexity for a heap with n nodes such that 2k <n < 2k+l - 1 is no more than that
for a heap with 2*+l - 1 nodes, which is still O(n). (A more careful analysis shows that
the bottom-up
the constant
is not increased; see Exercise6.32.)The reason
approach is

10

11

12

13

14

15

16

10

12

15

13

11

16

14

10

12

15

13

11

16

10

14

15

13

11

10

16

14

15

11

12

10

13

16

14

15

11

12

10

13

16

14

11

12

10

13

11

10

13

11

11

2
\302\251

Figure
indices.

\302\251

\302\251

15

6.15
The

16

\302\251

\302\251

15

\302\251

14

10

An example
numbers

circled

\302\251

\302\251

\302\251

\302\251

12

12

a
of building
are those that

\302\251

14
\302\251

heap
have

\302\251

bottom
been

\302\251

\302\251

up. The
exchanged

on

numbers

on

that

step.

\302\251

\302\251

top are

\302\251
1

the

6.4

faster
and

top. Thus,
minimize the work

This is another
The

algorithm.

Comments
that

it is

worthwhile

the

induction

also

have

to summarize

this translates

analogy,

that the analysis

seen

experience one learns

This

We

with an 0(n2)

proof that

prove a lower
approach. We
and

algorithm

that the

to prove

higher

called

a decision

to

the

or

with a

associated
We

the

problem

it

an 0(n

to

for

model

and

general
\342\200\224

lower

hence,

and are

with.

work

such

consist

of computation,
bounds using them

easier to

not just

one

arbitrary (unspecified)
fits the model must be

that

computations

log a?)

a particular

It is much harder to

better.

to an

corresponds

models

respects

and improved
A lower bound

of any algorithm that

In this section, we discuss one

bound.

tree. Decision trees

possible output.

The computation

that

time

running

trees are not

random-access
machines

each

with

address all possiblealgorithms

a model

lower

but
they are simpler in many
variations of decisiontrees, and
We define decision trees
and leaves. Each internal
node

than

rather

sorting

can solve

Decision

comparisons.

orders

in hand

further?

even

it

we have to

since

or equal

for

algorithm

algorithm

to define

need

not

to improve
no

bound,

than

possibilities,

intuition

started

problem is a

the conquer easier. In

of induction,
in
and,
to arbitrary elements. We
the
With some
design.

to make

order

different

about efficiency of algorithms

even
before
the
is helpful
in directing the search
for a better
not far removed from the intuition.
always!)

for Sorting

algorithm.

machines

go

The main techniques

paragraph.
and conquer. We have seen

intuition

Bound

it possible

in

hand

A Lower

Is

leads to a better
intuitive, but the

divide

of

trying

(but

have

one

in

subsets

is usually

truth

The

6.4.6

into

must

to develop

analysis is performed.
algorithm.

straightforward

sorting

to special

induction

the

and

superior.

section are variations

to spend time for the divide

applying

of induction

order

different

more

the

in this

described

particular,

to be

out

is hard

It

were

that

trying a

is

method

bottom
of the tree
bottom nodes rather

the

at

for the

work

the

nodes.

where

top-down

turns

to minimize

the top

for

many nodes

are

there

that

is better

it

example

method

bottom-up

is

approach

top-down

at the

to

than

the

than

few

141

Sorting

mainly

model
of

as are Turing
are weaker\342\200\22

There

are many

lower bound

them.
proofs utilizing
internal
nodes
as binary trees with two types of nodes\342\200\224
is associated
with
a query
whose outcome is one of two
is associated
with
one of the emanating branches. Each leaf

known

many

assume

that the

input

is a

sequence

of numbers x \\,

jc2,

..., xn.

to the
node,
query
is
the
left
or
the
branch
of the query, either
and, according
input
right
taken.
When a leaf is reached, the output
of the
associated
with the leaf is the output
time
with a tree T is the height
associated
of 7\\
computation. The worst-caserunning
which is the maximal
of queries required by an input.
number
A decision tree thus
decision
to an algorithm. Although
trees cannot model every algorithm
corresponds
(for
root
of a number with
we cannot compute a square
a decision
example,
tree), they are
A lower bound
reasonable models of comparison-based
obtained
for
algorithms.
better.
decision trees implies that no algorithm
We now use
of that form can perform
decision trees to prove a lower bound for sorting.
at

starts

to

the

the root

outcome

of

the

tree.

At each

the

is applied

142

and Sets

Sequences

Involving

Algorithms

6.1

Theorem
\342\226\241

Every decision tree algorithm

for

has height

sorting

Q,(n log a?).

The
Proof: The input for sorting is a sequenceX\\,x2,...,*\342\200\236.
sequencein the sorted order. Another way to look at the output is that

the input; namely, the output

become sorted. Every
permutation
is
A sorting algorithm
order.

is a
correct

permutation

(rearrangement)

of (1,

the decision

tree for

The

Since two

different

permutation.

that

total

The

the tree is

2,..., n)

the

tree,

height

to different

of

a permutation

same

the

such that they

input can be in any
Thus, every
inputs.

possible

as a possible output
in
represented
decision tree is associatedwith the leaves.

in a

output

it is

elements

the

since
all

is the

be

should

be

there must
Therefore,
number of permutations

binary

rearrange

possible output,
if it handles

correspond

permutations

leafs.

different

with

sorting.

how to

indicates

of

output

least

at

tree

the

one

n elements

on

they

outputs,

is at

leaf

is n!.

be associated

must

for every

possible

Since we

least log2(fl!).

By

assume
Stirling's

formula

n\\ =

^2nn(-)\"(\\+0(\\/n)).

=
!)

Hence,log2(fl

kind

This
because

defined

it does
the

kind

\302\243l(n
log

a?), which

completes

the

\342\226\2

proof.

is called
of a lower bound
an information-theoretic
not depend at all on the computation (notice that we
of queries we allow), but only on the amount of information

lower
have

bound,

not even

contained

in

the output. What

the
lower
bound says in this case is that every sorting
algorithm
to distinguish
between
Q,(n log a?) comparisons in the worst case, since it needs
requires
n! different
cases
and it can distinguish
between
two
at
a
time.
We
only
possibilities

could have defined

example, to

\"=,\"

\"<,\"

which is

a decision tree
and

\">\.")

as

a tree

In this

three

with

case,

the

height

children

(corresponding,
would
have been at

for
least

the Q,(n log a?) lower bound applies to

decision
tree with constant number
of branches
any
per node.
This lower bound
that
no
proof
implies
only
comparison-based sorting
algorithm
can be faster than
It may be possible to sort more quickly
Q(a? log a?).
special
by utilizing
For example,
on the keys.
properties of the keys and performing algebraic manipulations
if there are n elements,
all integers with
values
between
1 and
4a?, then bucket sort will
a
sorted
list
in
since
to the lower bound,
time.
This is not a contradiction
produce
0{n)
bucket
sort does not use comparisons.
can
It uses the fact that the values of the numbers
be used efficiently
as addresses
(buckets).
When discussing decisiontrees, we usually
only
ignore their sizes, and concentrate
on their heights. As a result,
to
even
linear-time
correspond
may
algorithms
simple
decisiontrees with an exponential
since we
number of nodes. The size is not important,
do not intend actually to construct the tree. We use the tree only as a tool for lower
bound
the size makes the proofs
since they may apply
more powerful,
proofs.
Ignoring
to programs of exponential size. On the other hand, the technique may be too powerful,
lower
it useless for deriving
bounds
for problems that cannot
be solved by
rendering
practical-sized programs, but can be solved with an exponential-sized
program (e.g., a
log3A?!,

still

Q(a?

log

a?).

In

other

words,

6.5 Order Statistics

with a table for all

program
models

build

the

we can

out that

This is not

of polynomial

trees

decision

build

of the

size

the
\302\253,

trees are

Decision

solutions).

possible

of computation.
The tree dependson
different
trees
for different values of n.

input.

\342\200\224
but

height

nonuniform

can potentially

We

a whimsical

just

143

worry.

exponential

turns

It

size

\342\200\224
for

so decision
trees are too
time,
probably require exponential running
sometimes.
That is, a decision tree lower
bound
may fall far below the actual
optimistic
of the problem. On the other hand, if the lower bound
is equal
to the upper
complexity
\342\200\224
of a particular
as is the case with
then
the lower bound
bound
algorithm \342\200\224
sorting
implies that even if we use a lot more space, we cannot improve the algorithm.
It is interesting to note that
the average
based
running time of any comparison
is
also
We
omit
the
which
is
much
more
\302\243l(n
sorting algorithm
log n).
proof,
for
and
Ullman
[1974]).
complicated(see example Aho, Hopcroft,

that

problems

6.5

Order

Given

a sequence

Statistics
S =x

sequence

by sorting

can be

answeredwithout

the

with

{,jc2,...,xn

the

elements.

problem

6.5.1

Maximum

In this

kih

minimum

such

in

S if Xj is the
a

in

about

ranks that

questions.

We start

elements,

then consider

the

element.

smallest

Elements

Minimum

and

with

of all elements

questions

many

section, we deal

rank

ranks

the

there are
and

Xj has

that

easily determine

the maximum

of finding the

general

we say

elements,

However,

sorting.

of finding

problem

of

We can

in S.

element

\302\243th-smallest

the maximum
or the minimum element of a sequenceis straightforward.
If we
this
the maximum of a sequence of size n-\\,
then we need only
to compare
the
maximum to the nth element to find the maximum
of a sequence of size n (finding
maximum of a sequence of size 1 is trivial).
This process takes one comparison
per
1.
of comparisons
is /i \342\200\224
element,
starting with the second element; hence, the number
now that we want
to find both the maximum and the minimum
elements.
Suppose
Finding

know

The

Problem

Find

the

maximum

and minimum

elements in

a given

sequence.

The

solve

The total number

and then a?-2 to find the
minimum
need not be considered).
the maximum
Can we do better? Consider
(because
Assume that we know how to solve the problem
an inductive
for n - 1
again
approach.
to find the solution for n elements
and that we want
elements,
(the base case is trivial).
element
We have to compare the additional
to the maximum and minimum
elements
found so far. This requires two comparisons,
which
of
number
implies that the total
no comparison
is required for the first element,
be 2\302\253-3, since
will
comparisons
again
of

straightforward

comparisons

solution

will be 2n

is to

-3: n-\\

to

both

find

problems

the

independently.

maximum

Involving Sequences

Algorithms

144

one

only

and

solution

by

elements

in

The

Let's try

and Sets

is required for
the elements in a
is irrelevant to the

scanning
the

sequence

We cannot

element.

second

the

comparison

different

because

order,

the

improve
of

position

the

the

problem.

one element
at a time.
the solution by more than
next attempt can be to extend
a
time.
That
assume
that
we know
at
two
elements
the
solution
is,
to extend
by

the problem for n -2 elements, and try to solve it for n. (For this approach
n = 1 and n =2, so that extending by two will
base
cases,
to be complete, we need two
be the maximum
{min)
cover all natural numbers.) Consider xn_x and xn, and let MAX
It is easy to see that finding
elements
first n-2
the
of
induction).
(known
by
(minimum)
the new maximum and minimum
only three comparisons. We first compare
xn_x
requires
two
values to MAX, then compare the smaller of
these
of
the
to xn, then compare
larger
with approximately
3\302\253/2comparisons
to min.
them
So, overall, we have an algorithm
elements
at a
of In comparisons! Can we do better by adding three (or four)
instead
of comparisons. It turns
the same
time? Following
approach leads to the same number
of comparisons
for this problem
out that we cannot reduce the number
by any method. It
to about
also
leads
3\302\253/2
a divide-and-conquer
to note that
is interesting
approach
to solve

how

6.5.2
We

6.14).

(Exercise

comparisons

the Ath-Smallest Element

Finding
consider

now

the general

The Problem
an

integer

k such

problem.

Given
that

a sequence

\\<k<n,

This problem is called order statistics

to \302\253,
then
we can find
the /:th-smallest
minimum (maximum) element k
comparisons.

Sorting

There

-OQogn).

for any value

The idea is to

element

would

or

times.

is, however,
of k.

another

If k

selection.

This

algorithm

is

by running
algorithm,

that

in S.

very

close

the

algorithm

requires

approach

than this naive

elements,and

element

\302\243th-smallest

element

be better

...,xn of

S=jtj,jt2,

find the

unless

to 1 or very

close
the

for finding

approximately
k is 0(\\ogn)

finds efficiently

the

kth

kn

or

smallest

use divide and conquer in the same way as it is done in quicksort,

is
the sequence
only one subproblem has to be solved. In quicksort,
sorted
The two subsequences are then
partitioned
by a pivot into two
subsequences.
which subsequence contains the kth
to determine
Here, we need only
recursively.
smallest
this
and then
to
continue
the algorithm recursively only
element,
for
in Fig.
The rest of the elements
can be ignored. The algorithm
is given
subsequence.
that

except

6.16.

Complexity
Since

this

only one
algorithm

in quicksort,
As
choosing poor pivots leads to a quadratic
algorithm.
in each
time
recursive call, the running
subproblem has to be solved
is lower
than that of quicksort. The average
number
of comparisons

of

is

6.6 Data Compression

k) ;
range 1 to

(X, n,

Selection

Algorithm

145

Input: X (an array

Output: S (the kih

in the
smallest

k (an

and
\302\253),

element;

the

integer).

X is

array

changed).

begin

or (k > n)

if(k<l)

then

\"error\"

print

else

S :=Select(l,n,k)

end
Select

procedure

(Left,

Right, k);

begin

if Left = Right

then

:= Left

Select

else
Partition

(X,

{ see Fig. 6.9}

Left, Right);

be the

Middle

Let

output of Partition

>k then
if Middle -Left+\\
Select
(Left, Middle, k )

else

Select(Middle

+ 1,

Right, k

(Middle

Left +

1))

end
Figure 6.16 Algorithm

but

(a?),

steps

in

we will
the

worst

not prove that here.

case. However, in

It is

Selection.

also possible to

practice,

the

find

the

kth smallest

presented

algorithm

in

Fig.

in O(n)
6.16 is

more efficient.

Comments
the

A?/2-smallest

Most

There is no simpler

algorithm

problem

median-finding

simpler! This is another

recursion requires arbitrary
Data

6.6

statistics require
Algorithm Selection is an excellent
for finding
only medians. In
of order

applications

element.

to

finding

example
values

any

\302\243th-smallest

of strengthening
of k.

the

finding

the

median,

median-finding
other

element
induction

words,
makes
hypothesis

that is,

algorithm.
extending the
the algorithm
since

the

Compression

Given a file, which

we
file, as small as possible,
such that the original
file can be reconstructed from the
information
in the compressed
for example,
when access to the file is infrequent, so the
file. Data compressionis useful,
is justified by the storage savings. It is
work
involved
in compressing and uncompressing
Data

compression

consider

as a string

is an

important

technique

of characters,we

want

for saving
storage.
to find a compressed

greater

of processing

cost

the

than

and Sets

it.

Data

developed field. In this section,

of data compression.
aspect
the file is assumed to be
For simplicity

particular

the

compression

a very

is

of sending
information is
cost
has many more applications, and
for one
one
we describe only
algorithm

problems where

in communication

important

also

it

Involving Sequences

Algorithms

146

a sequenceof

letters.
Each of the
English
of the
of
the
called
is represented by
bits,
encoding
unique
string
26 characters
standard
character. If the length of all encodings is the same (as is the case for most
of
on the number
file
the
of
bits
number
the
only
depends
representing
encodings),
to choose smaller bit
it is possible
characters in that file. On the other hand,
more often and larger
characters
for
(such as A) that
appear
representations
a

representations for characters (such

as

Z)

that appear rarely. For

all characters
Interchange),

example, in

ASCII

are represented by
Standard
(American
B
of size 7. A is represented
1000001,
bit strings
by 1000010, and so on.
by the bit string
lower-case
and special characters.) The
including
(There is room for 128 characters,
three letters) requires 21 bits. If we change the
with
\"AND\"
word
(and any other word
time A appears. However, not
of A to, say 1001, we save 3 bits every
representation
For example, we cannot
valid.
There
be
is
of
set
ambiguities.
may
encodings
every
of M as 1001101, because
for A and leave the encoding
choose 1001as an encoding
of M. We could use
it is A or is part
1001 we cannot determine whether
we read
when
In
to separate
characters, but that would
only add to the representation.
specialdelimiters
not be equal to a complete
of one character must
general, the prefixes of an encoding
Whenever
the prefix constraint.
We call this constraint
character.
of another
encoding
to lengthen
the encodings of
have
of one character, we may
we shorten
the encoding
of
the frequency
others.
The
is to find the best balance, assuming we know
problem
Code for

appearances of the

encoding

we have to

First,
this

value

tables

length

the

total

of

of times

number

the

find

character.

(In

an

and that

each character appearsin the

many cases, we can use standard

text;

we call

frequency

table for the

of computing the exact frequency
Denote the characters by C \\, C2,...,
by
text.)
Cn, and denote their frequencies
Given
an encoding E in which
a bit string S, of length
C,, the
\342\200\242\342\200\242\342\200\242>/\342\200\236\342\200\242
j, represents
of the file F compressed by using
E is
encoding

L(E,F)=

for

of characters),
(a sequence
satisfies the prefix
constraint
of bits needed to encodethe text.
text

that

number

compute the

the frequency

computed

particular

/i>/2>

Given a
characters

the

for

minimizes

characters.

different

Problem

The

Information

typical

texts,

instead

isrfi.
1=1

Our

goal

L(\302\243,F).

is to

find

an

encoding

E that satisfies

the

prefix

constraint

and minimizes

6.6

The prefix
a decoding

to an

is equal

each node has either

which

in this

the path from

root

constraint

file is

encoded

character.

scanned

a graphic

is

not

n -1

define

the

characters
induction

Consider

characters.

of

and

is based

to solve

problem

(and
of

reduction

and

in

the

that minimizes
It is

problem.

a problem
As

order

which

to

tree

binary

in

of 0s

and

Is on

(see Fig. 6.17).

leaves.
When the
corresponding

The

L(\302\243,F).

tree

useful, however, to

its constraints).

case is trivial).

(the base
hypothesis

in order

on a

correspond

0, or no emanating

can safely determine the

a tree

such

construct

the

must

is reached, we

a leaf

and

necessary

illustration

The algorithm
with

that all characters

says

Our problem is to

representation
have

the

so let's look at
unambiguous,
of bits one by one until we get a

labeled

two

147

Compression

emanating
edges
by
tree correspond to the characters.
The sequence
to a leaf corresponds to the character's
encoding

edges. The leaves

The prefix

decoding

scan the sequence

encoding of one of the

We need to

procedure.

sequencethat

to make the

is needed

constraint

Data

with

usual,

n characters

the main

to eliminate

difficulty

to a

problem

is how

to

characters. The reduction

from the ones we have seen so far. Instead of simply

one
eliminating
from consideration, we introduce
a new \"artificially
made\" character in place
is a little more complicated, but
it serves
the
of two existing characters. This technique
is reduced.
the size of the input
Let C, and Cj be two characters with
same purpose \342\200\224
two
such characters,
then ties are broken
minimal
(if there are more than
frequency
We claim that there exists a tree
that
minimizes
these
L{E, F) in which
arbitrarily).
if
distance from the root.
characters correspond to leaves with the maximal
Otherwise,
lower in the tree, it can be exchangedwith C,
there is a characterwith higher
frequency
or Cj decreasing L(\302\243,F).
(If its frequency is equal, it can still be exchanged without
or no children (or
L
each
node in the tree has either two children
Since
(\302\243,
F).)
changing
else we can shorten the tree), we can assume that C, and Cj are together. We now
is the sum/)
C, and Cj with a new character, called C,7, whosefrequency
replace
+/y.
The problem now has n - 1 characters (n-2 old and one new), and as such can be
of the original
solved
the solution
problem
by
by the induction
hypothesis. We obtain
in
to
an
internal
node
the
reduced
with
two
leaves
C,
problem
corresponding
substituting
and
in
of
as
an
of
the
leaf
to
We
leave
the
proof
optimality
place
corresponding
Cj
C,y.
here

is different

character

exercise.

010

Figure 6.17

The tree

representation

Oil
of encoding.

Algorithms

148

Sets

and

Sequences

Involving

for Huffman's
The operationsrequired
encoding are (1) insertions
Impl6ni6ntati0n
with
of
minimal
from the
deletions
the
two
characters
data
structure,
a
(2)
into
frequency
a
the
tree.
is
data
structure
for
the
first two
A
and
(3) building
data structure,
heap
good
in
the
worst
case.
of
which
The
is
each
requires
steps
0(\\ogn)
algorithm
operations,
known
as
This
is
Huffman's
6.18.
encoding after
compression
technique
given in Fig.
D. Huffman
[1952], who proposed this algorithm.

(S, f);

Huffman_Encoding

Algorithm

Input; S (a string

Output;

of characters),

begin
insert all characters
while H is

not

make X

one

only

root

the

H according to their

frequencies

do

empty

ifH contains

frequencies).

S).

a heap

into

array of

and/(an

tree for

Huffman

(the

X then

character

ofT

else

pick two
replace

lowest

with

frequencies

H ;

from

new

Z whose

character

of the frequencies

ZtoH

make X

them

Y with

and

sum

the

insert

X and

characters

and delete

ofX and Y

frequency is

;
Y children

and

{Z has no

ofZ inT

yet}

parent

end

Figure 6.18

Example

that the data contains six

Suppose
in

1, respectively

The Huffman

6.19.

The internal

nodes are numbered

O (log

6.7
Let

the tree takes constant

each.
Overall,the running
steps

n)

String
A -a

assume

\\

time

characters

although

sequence of

and

an

the

it

is

characters

a i a2

time

time

to the

per node.

of the

frequencies

characters

created.

were

they

5, 2,
is given

\342\226\2

Insertions and deletions

is O

algorithm

(n log

n).

Matching
\342\200\242
\342\200\242
\342\200\242

a2

that

characters,
substring

according

Building

Complexity
take

characters 4, \302\243,
C, D, E, and F, with
tree correspondingto these

10,and

Fig.

Encoding.

Huffman

6.1

\342\226\241

3, 4,

Algorithm

' '

' a{ (h\\

\342\200\242
\342\200\224

B=bxb2
come from a
not

b2

'

m<n,
set.

be two strings
(It is convenient

A substring of
\342\200\242\342\200\242\342\200\242
from
A We denote
a}
' '
bt).

necessary.)

atai+\\

bm,

finite

a
by

string
A(i)

of

characters.

to

think

A is a
(B(i))

of

We
English

consecutive
the special

6.7 StringMatching

6.19

Figure

The

Huffman

for example

tree

149

6.1.

Given
A and \302\243,
find
the first occurrence
two
The Problem
strings
(if any) of B in A. In other words, find the smallest k such that, for all /,
=
1 < / < w, we have ak+i
bt.

or pattern in a
word
obvious exampleof this problem is a search for a certain
to find patterns.
The problem also has
editor must contain commands
molecular
applications to other areas \342\200\224
biology, where it is useful to find
including

The most
text

certain

file.1 Any text

patterns

inside large

RNA

or

DNA

molecules.

This problem seems simple

We can try to match B inside A by starting at
at first.
the first character
of A that matches bx and continuing
(comparing to b2 and so on) until
we either complete the match or find a mismatch. In the latter case, however, we must go
back to the place from
in Fig.
we started and start
This process is illustrated
which
again.
6.20

an example

will

use

In

this

example,
at a 4
mismatch
leads
to
a
which
a2,
right
we start at a3, which is a match, but a4*b2.
The next attempt is more
Next,
away.
We
have a match from a4 to a-,, only
to have
a mismatch at a%. Now, we
promising:
need to backtrack severalstepsand to compare
bx to a5 (mismatch), then
b{ to a6, and
so on. Eventually,
find a match starting
we
at a ,3. We may have to backtrack and
of times,
of comparisons
compare
again a substantial number
leading to O(mn) number

by

we

that

AyxxyxyxyyAyxyxyyxyxyxx,
since b4*aA. We now must

'At least, that

is the most obvious

throughout

and B=xyxyyxyxyxx.
start comparing b{ to

one to me, as I am currently

this

The

editing a text

section.

first

file.

mismatch

occurs

150

Involving Sequences

Algorithms

and Sets

B = xyxyyxyxyxx.

xyxxyxyxyyxyxyxyyxyxyxx.

10 1112

123456789

13 1415 1617

18

19 20

2122 23

xyxxyxyxyyxyxyxyyxyxyxx
1:

xyxy-

2:

3:

4:

\342\200\242

\342\200\242
'
\342\200\242

y
x

5:

\342\200\242
\342\200\242

6:
7:

y
y
\342\200\242
\342\200\242

xyxyyxyxyxx

x - x y

8:

9:

10:

11:
12:

13:

y
-

\342\200\242
\342\200\242
\342\200\242

xyxyyxyxyxx

Figure 6.20
in the

\342\200\242
\342\200\242
\342\200\242

worst case.

An

Noticethat

example

a lot

of a

of the

straightforward

work

is redundant.

string

matching.

For example, we

find

twice

of
the subpattern xyxy fits inside A starting at au (lines 6 and 11). In the example
a
in
a
will
since
word
text
of
be
the
number
file,
finding
very small,
backtracking
steps
most of the time the mismatch will occur early on. This simple algorithm
is fairly
good
that

for such applications. In other cases, where the alphabet is small and the patterns have
above
of backtracking
many repetitions, the number
steps may be large. The algorithm
to the same place in the text many times. We would
may compare the same subpattern
like
to find an algorithm that
avoids
such worst cases. The problem is to arrange the
on
information
we learn throughout the algorithm
later
such that it can be used efficiently
when the same matches occur in other places.
To improve the straightforward
the reasons for
we must first understand
algorithm
its
A
The bad case we discussed was caused by the need to backtrack.
inefficiency.
We
bad case will occur
and the text is yyyyyyyyyyyyx.
if the pattern is yyyyyx
particular
will compare the five ys in the pattern
to the text, find the mismatch with the jc, move one
step to the right, and make four redundant
again and again. (This simple
comparisons
case is easy to handle, but it illustrates the general problem.)
consider
On the other hand,
the pattern xyyyyy.
this pattern in the text, we look for occurrences of x
To
match
followed
there is no need to backtrack.
sufficient,
by five ys. If the number of ys is not
We
will need to find
the next x, and all the matched
will
The straightforward
not help.
ys
is
to
the
runs
in
since
no backtracking
linear
time
xyyyyy,
pattern
algorithm,
adapted
needed.

6.7

String

151

Matching

Let's return now to the original pattern B =xyxyyxyxyxx.

Suppose that a mismatch
occurs when the fifth character of B is scanned(as it is when a% is compared to it in line
in
A must
two characters
have been xy (since
4 of Fig. 6.20). The
preceding
they
are
also
the
first
characters
of
B.
We
now
want to \"slide\" B to the
But, xy
matched).
in A to some characterin the middle
and compare the current
character
of B (taking
right
account
the previous matches). We would
like to slide B as far to the right as
into
without
bypassing
potential matches. In this case, we can
possible (to save comparisons)
slide B two

steps

A that caused

the

right. We continue

to the
mismatch

(a

in the

the

match

by comparing

example) to b3, since we

what we did later on, in

the

already

same

character

know

that b

in
{

and

line
6 of Fig. 6.20, except
(In fact, that is exactly
in
three more redundant
\342\200\224jc
line
and
of
5,
comparisons
xy at the beginning
Notice that this whole discussion is completely independent
of A!
6 \342\200\224
to get there.)
line
We know the last few characters in A since they have matched B so far.
In the following
discussion, we will not assume that there are only two characters
will contain only two
even though, for simplicity, the examples
in the text (and pattern),
is the subject of Exercise 6.45) to make
It is possible (and that
the algorithm
characters.
even more efficient in this case.
The mismatch at line 6 of
Let's look at another
example
by continuing the match.
b
We
can
now
a
lot
more sliding. Consider
is
at
the
last
character
of
do
\302\243,{{.
Fig. 6.20
'''
=
the
B(\\0)
b{ b2
bw. We know that \302\243(10) is exactly the same as the
subpattern

b2 matched.
it took us
that

in A; that
We want
10 characters
is, B(\\0)=A [6..15], becausethey matched.
preceding
to the right until
there
is some
to determine exactly how many steps B can be shifted
for
a
maximum
of
of
We
determine
this
number
another
match.
by
looking
suffix
hope
In
that
suffix
is
of
as
is
that
is
to
a
of
B.
this
3
case,
#(10)
length
(xyx),
prefix
equal
is
at
a
is
in Fig.
one
and
illustrated
6.21. In the figure,
time,
shifted,
\302\243(10)
step
last character,
(The
bn, is ignored
compared to itself, until a prefix matches a suffix.
since it is the cause of the mismatch.)
Since we know that
\302\243[1..3]=\302\243[8..10], we can
We save
occurs.
continue by comparing
a{6 to b4, and so on, until the complete match
all
the comparisons
difference
on lines 7 to 12 and half those on line 13. The only
on B.
between
in Fig. 6.21 depends only
Fig. 6.21 and Fig. 6.20 is that the information
This is important
because
we can preprocess B once, and find all the relevant information
about
it regardless
of the text A. We now can take advantage
of all the matches done in

line

6 of

Fig. 6.20; none of them

B=

will

be repeated.

xyxyyxyxyxx
x
x y

Figure

6.21

Matching

the

pattern

against

itself.

Algorithms

152

Sets

and

Sequences

Involving

of the improved algorithm. We will study

all
The preprocessing of B is the essence
B
when
to
handle
mismatches
of
and
devise
a
occur
they
way
the repeating patterns
The string A is always
Our scheme is the following.
scanned
without
backtracking.
in
the
same
of
A
character
is
no
be
there
A, although
may
backtracking
forward;
B
are
When
a
of
there
mismatch
characters
several
is
(when
mismatches).
compared to
in
B
how
far
we
There
is
an
must
backtrack.
encountered, we consult a table to find
entry
to the amount
of backtracking
table for each character in B corresponding
(or the
in the
this character.
when
there is a mismatch
In a
number of shifts)
involving
required
We first define the table
moment, we will show how to construct this table efficiently.
and show how we use it for the string-matching
problem.
precisely
the table
should be clear now. For each b{ we want to find the
The idea behind
of B(i-\\).
If the length of this suffix
is
suffix of B (/ - 1)that is equal to a prefix
largest
character in A can be matched against
without going
the mismatched
directly,
j\\ then
bj+l
know
that the most recent j
the other redundant matches. We
all
already
through
of B. Furthermore,
since this suffix is the largest
in A match the beginning
characters
B
to
a
we
know
that
cannot
fit into
A any farther to the
are
that
those
prefix,
equal
among
and here is a precise definition
of the values of its entries:
The table is called next,
left.

= the

next(i)

B (J),

0 if

and

maximum j (0 < 7
no such j exists.

< i \342\200\224
1) such

that

fe/__/-fe/_y+1

\342\200\242\342\200\242\342\200\242
=

bj_{

this case. It is clear that next(2) is

next(\\) = -\\ to distinguish
always equal to 0 (sincethere is no j satisfying 0 < j <2- 1).The values of the next table
B in Fig. 6.21 are given
for the pattern
in Fig. 6.22.
These values can be computed
in a
brute force way,
as was done in Fig. 6.22.
to
However, there is an elegant
way
compute
all these values in time 0(m).
Let's first assume that the values of next are given
to us,
and see how
to perform
the matching. Afterwards,
we will describe
how to compute
For

we define

convenience

next.

The matching
until

there is

character

in

match). If this
so on.

The

At

is compared
is a

mismatch

exception

only

1=
B

next=

The characters

as follows.

proceeds

a mismatch.

that

point, say

at

bh

the next

A are compared
to those in B
table is consulted and the same

in

the first next(i) characters already

(since
against bnext{j)+l
too, then the next comparison is against
^W
(\302\253\302\253\342\200\242.\302
(/hi hi > anc*
to this rule is when
the mismatch
is against b {; in this case,

10

11

-1001201234
Figure

6.22

The values

of next.

6.7 String Matching

to proceed in A. This case can be determined
The program for string
is given in
matching

we want

is -1.

which

Algorithm

Input:
{

We

Output:

at A

value of next(\\),

the

by

Fig.

153

special
6.23.

String_Match (A, n, B, m) ;
of size n), and B (a string of size m).
assume
that next has been computed;seeFig.6.25)
Start (the first index such that B is a substring of A starting
(a string

[Start]).

begin
Start :=

0;
and i<ndo

= 0

Start

while

ifB[j]=A[i]then
i := i +

else

j :=next[j]

ifj = 0

/:=/

ifj = m

+ 1;

then

+ 1

/;

then Start

:= i - m

end

6.23

Figure

Algorithm StringMatch.

use
of the next table. We
algorithm to compute the values
=
We
assume
which
takes
of
the
base
case.
next
care
0,
mentioned,
(2)
that
the values
of next for 1, 2,...,/- 1 have been computed, and we consider next (i). At
In other
will happen if bj_\\ = bnext{i_])+\\.
best, next(i) can be next(i- \\) + \\, which
suffix that is equal to a prefix
is extended
words, the largest
by bj_{. This is the easy
case.
The difficult case is when
We need to find a new suffix that is
b(_{ *bnext{i_i)+i.
suffix of B(i-2): It
know how to fit the largest
equal to a prefix. However,we already
\342\200\242\342\200\242\342\200\242
fits in b{b2
But
(see Fig. 6.24).
bi_x *bnexni-\\)+\\ is exactly the
having
JVvf(/-i)
same
as having a regular
what
to do
mismatch
at bnext{i.\\)+\\\\
And we already know
about that.
If there
is a mismatch at index j\\ we go to next (J). So, we have a mismatch
at index next(i- 1)+ 1,and we go to next (next (/- 1)+ 1). That
is, we try to match bj_\\
to
= next (next (i - 1)+ 1) + 1. Otherwise,
bnext(nextu-\\)+\\)+\\ - ^ tnev match, we set next (/)
we continue
we either
in the same fashion until
to the beginning.
get a match or we return
It

induction.

to find an

remains

As we

6.2

\342\226\241

Example

Let B = xyxyyxyxyxx
next(\\Q),

which

(the

is 4,

and

same
compare

as

in

Fig.

bw

6.21),

to b5. If

and consider
they

had

been

next(\\

1).

the same,

We
then

first look
the

largest

at

next(i-l)+l

next(j)+l

at b5y

mismatch

happen to be the
The

to a

is equal

that

and we

look at

next(5)

same.

Hence,

next(\\ 1) = 3, which

for computing
The program

algorithm

to implement.

been

have

would

suffix

Computing next(i).

6.24

Figure

prefix

and Sets

Involving Sequences

Algorithms

154

the

which

table

next

is given

in Fig.

5,

but

can

is difficult

are

not.

compare

they

We now

is 2.

10 to b3y

be verified

easily

to

So, we have

but

understand,

by

hand.

it is

not

and they

\342\226\

difficult

6.25.

Algorithm Compute_Next (B, m) ;

B (a string
of size m ).
Input:
Output: next (an array of size m).

begin
next(\\)

:=-l

next(2):=0;

for i := 3 to m

do

j:=next{i-\\)+\\

whilebi_x

*bj

j:=next(j)+\\

and

j>0do

next(i):=j

end
Figure 6.25 Algorithm

A character

Complexity

there is a
pointed

mismatch,

then

the same

by the next table.

the same character
of
to

of A

may

Compute

be compared

character of

If there is another
.4 until there is

Next.

against

many

is compared
mismatch,

then

characters

of B.

If

against the character of B

we continue comparing

of
either a match or we reach the beginning
How
we claim that the running time of this algorithm
is still O(n).
times
can we backtrack for one characterfrom
Let's assume that the
many
A> say aft
first mismatch involved bk. Since each backtrack
leads
us to a smaller index in By we can
to reach bk we must have gone forward k times
backtrack only k times.
However,
without
moves,
to the forward
any backtracking! If we assign the costs of backtracking
then we at most double the cost of the forward
moves.
But there are exactly n forward
\302\27
so the number of comparisonsis O (n).
moves,
against

B. Nevertheless,

6.8

155

Comparisons

Sequence

This algorithm
was developed
by Knuth, Morris, and Pratt [1977J, and it is known
for this problem was developed by
KMP algorithm. Another
fast
algorithm
We
The
is
and
Moore
sketch
it
difference between the algorithms
[1977].
briefly.
Boyer
B
from
the
end
rather
than
scans
from
the
the
that
Boyer-Moore algorithm
beginning.
be of bm against
then
the next
That
is, the first comparison will
am. If there is a match,
we use the
am_x, and so on. If there is a mismatch,
comparison will be of bm_\\ against
information, much as we did in the previous
algorithm, to shift the whole pattern to the
- \"Z,\" and Z does not appear
if am
For example,
at all in B, then the whole
right.
as the

to the right by m steps, and the next comparison will be of a2m

If
Z
at bh then we can shift
does
by m-i
say
steps. The
appear in \302\243,
against bm.
becomes
more
when
there
how
much
to
shift
are
several
decision
complicated
partial
the matches
matches. On the one hand, we want to utilize
already found. On the other
the whole
even if the same
hand, it is more efficient to shift
pattern as far as possible,
comparisonsmay have to be performed twice. We omit the details. The interesting
it is likely to make fewer than
n comparisons
of this algorithm is that
characteristic
(in
one
allows
us
This
is
bad
mismatch
to
without
more
because
shift,
any
regular text)!
be shifted

can

pattern

comparisons,
6.8
The

by

m.

Comparisons

Sequence

that

attention

concentrate here on
required

section

assume
would

comparisons has received a lot of

is the applications to problems

of sequence

subject

reason for
to

that

one string

change

is dynamic
Let A=a{a2

one

only

programming (discussed
\"'
'\" an and B-b\\b2
come

change

character

steps

throughout

this

of characters.

We

5.10).
two

be

bm

number of edit

strings

characters, for example). We

it becomes
equal to B. We allow
a cost of 1 to each: (1) insert \342\200\22

set (English

a finite

from

Section

We

biology.

technique used

main

The

another.

into

The main

lately.

molecular

minimum

the

finding

in

the characters

like to

\342\200\224

problem

attention
in

such

by character

that

types of changes (or edit steps),and we assign

insert
a character
from the string, and (3)
delete
a character
into the string, (2) delete \342\200\224
with
For
a
different
character.
one
character
replace \342\200\224
example, to change the
replace
three

string

abbe

insert an a

into the

between

two

bs (babe),

However, we can
changes.
and then delete the last c, for a total
three

of single-character

also insert
of

first

the

replace

b at

a new

a, forming
the

last c

the

beginning

the
with

bbc\\

string

a b

for a

of

babbc),

(forming

Our goal is to minimize

two changes.

then

total

the

number

changes.

The string-edit
maintenance. We
modification of the

delete
and then

we can

babb,

string
the

problem

has

also applications

to

file

comparisons

and

revisions

is a
It is convenient to extract the differences
between
the two
nles. There may be several versions of the same program, and, if the versions are similar
and
need to be archived, it is more efficient to store only
the differences
instead of
they
In such cases, we may allow only insertions and deletions.
the whole
storing
programs.
to each of the edit steps.
costs
In other
cases, we may assign different
may

first

have

one.

a text

file

(or

program)

and

another

file

that

156

Involving Sequences

Algorithms

and Sets

possible changes,and it seems difficult to find the best one.

We
denote by A (/) (B (/)) the prefix substrings a { a2 ' '' a{
As usual,
''' hi)- Our problem is to change A (n) to B (m) with a minimum number of edit
(b\\b2
steps. Supposethat we know the best way to change A (n- 1) to B(m) by induction.
we assume only
that
we know one of
best
solutions;
may be several different
(There
to change
A (n) to B (m). But
one more deletion, that of an, we have a way
With
them.)
be that it is better to replace an with bm,
this may not be the best way of doing it. It could
or better yet, an may even be equal to bm.
all the different possibilities of constructing
the
minimum
We need to consider
of
smaller
best
A and
A to B with the aid of the
from
changes
sequences involving
change
B
the
minimum
cost
of
A
to
Let's
assume
for
now
C
B. Denote by
(/)
(/, j)
changing
(J).
in finding
A to B and not
in the change
the cost of changing
that we are interested only
in finding
a relation between C(n, m) and C(/, y)s for some
itself. We are interested
is and js. It is not hard to see that there are four possibilities,
of smaller
combination
are

There

quite

a few

try induction.

we

corresponding to the

delete: if an is deleted
scenario above holds.

B(m)

insert:if

nothing:

change

from

be the

would

deletion.

more

one

minimum

the

the

induction)

from

change

bm, then we

match

to

then
\302\243,

one from

(n

other

In

the

- 1) to
words,

have C(n, m)

minimum

change

to B

insertion of

involves

1)+ 1. That
from A(n) to B(m-\\)
= C(n,

m-

a character

is, we find

(by

insert

and

equal to bm.

character

replace: if
change from
match:

in the minimum

to doing

= C(fl-l,m)+l.

C(/i, m)

to

and

The best change

then

and

edit steps

different

three

if

an

an

is

replacing

- 1) to B (m

A (n

to

is equal

bm,

bm,
-

then

we

then
1) and
C(n,

then

first

to add

m) = C(n

to

need
1

find the minimum

if a\342\200\236
* bm.

- 1,m- 1).

Denote
f

'<''\342\200\242\302\273

=|l

We can now

combine

if ai

bj

ifai*bj.
these

four cases

into

the

(deleting

C(A?-l,m)+l

C(rt,

m)

mini

C(a?,

m-

C(n - 1,m
withC(/,

0) = /

for all/,

recurrence

following

l) + c(rt,

0</<a?,

and

a\342\200\236)

(inserting for bm)

1)+ 1
-

relation.

m)

C(0J)=j

( replacing

for

all

V matching

a\342\200\236),

y, 0<j<m.

It is not difficult
to prove that these possibilities are the only ones.
Consider an. It
be
handled
must
somehow. It is either deleted, which is handled by the first case, or it is
into some character in B. In the latter case, either an is mapped
is
into bnn which
mapped

6.8

handled

the third

by

or

fourth

case something must

in which

or it is mapped
case,
be inserted after an.

a character

into

157

Sequence Comparisons

appearing

before

bm,

with
this
is that we used induction
too
The problem
We
approach
many times!
of size (a?, m) to three problems of only
smaller
If
sizes.
we
reduced a problem
slightly
for
each
smaller
we
the
work
recursion
time
we
use
problem,
separately
every
triple
in
reduce the size by a constant. That leads to an exponential algorithm. Fortunately,
The key to this
this case there is no need to solve each
subproblem
separately.
too
is that there are not
different
observation
subproblems altogether. Each
many
< / <n,
involves
C
for
some/
and
(/,
j)
j in the ranges 0
computing
possiblesubproblem

and

There are

0<j<m.

for more

nm

than

of such

combinations

nm

is

This

subproblems.

/s

and

same

the

ys,

should

so there

not

be

we observed

phenomenon

a need
in

the

it, we use strong induction. Instead of

knapsack
problem (Section 5.11). To overcome
of size n, we extend all subproblems
a problem of size n-\\ to a problem
extending
just
so we have to
of size < n to the problem of size n. This is a two-dimensional
problem,
extend all subproblems of sizes <(n,m) to the problem of size (n, m). The notation
at least one of these values is less
combination
of (/, j) such that
<(a?, m) means \"any
the

than

bound and

corresponding
will
be able

to use

We

to us.

available
To

compute

the figure.

already
order

version

We

create

of C(/,

the value

We

visited

traversal

of the

want

the

(i.e.,

to

j),

the

with

need

we

other entries
an

example

its bound.\"
the

subproblems

are

of all subproblems. Consider Fig. 6.26.

in
the three other values indicated by shading

results

so that, whenever
necessary for its

row by row from

approach is

greater than

solutions of all

the matrix

scan

three

if the

induction

strong

a table

is no

one

other

the

is sufficient.

to

left

we arrive
computation.

right)
of dynamic
programming.

at an entry, we
In this case, a

This

have

row-

two-dimensional

of the
Each
C[\\..n, \\..m\\.
entry C[i,j]
move
that
leads
to
matrix
holds the value of C (/, j). Let M [/, j ] denote the last
(change)
the
we can
we need only the last change is that
minimum
value of C[/, j]. The reason
of delete(i),
This move is any
one
backtrack and find all the changes from the matrix.

Implementation

We

maintain

a matrix

'

\\','

?/fPYZ:

fK%-T<

Figure

C(iJ)

6.26 The dependenciesof

C (/,

j).

Algorithms

158

insertij), or

[/

the

1 ],

j ], we need to know
last change can be determined
to the minimum value for C[/, j]. The

replaced, j). To compute

and C [/ - 1,j - 1 ]. The
leads

possibilities

Sets

and

Sequences

Involving

of C [i -

the values

C [/,

is given

algorithm

1,7],
of

to which

according

in

Fig.

6.27.

Algorithm Minimum_Edit_Distance
of size n), and B
A (a string
Input:

C (the

Output:

cost

(A, n,

B, m);

(a string

of size

m).

matrix).

Begin

:= /;

fori:=OtondoC[i,0]

forj:=

:=j;

tomdoC[0J]

for i := 1 to
forj :=

n do
1

do

torn

x:=C[i-\\J]+\\;

y:=C[U-l]+l;

ifdj=bj

then

z:=C[i-\\J-\\]

else

z:=C[i-l,y-l]+l;

C[/,y] :=min (*,>>, z)

{ A/ [/, 7 ] can be set appropriately

end

Figure

Complexity

is clear

It

One major drawback

Comments
problem

is the

Dynamic

on many

depends

6.27

Algorithm Minimum_Edit_Distance.

from the

program in

need for

an

solutions of

previous results is common

in some order (usually
row
order),
dynamic

conquer

6.9
The

programming

is

programming

store

the

O (nm)

in

approach

slightly

useful

usually

at

less

where

cases

in

smaller

which leads to

is O

time

running

(nm).

well.

The table is

least

quadratic
efficient
than,

of a table

usually

running

say,

solution of

the

The use

problems.

programming.

dynamic

is

that the

6.27

Fig.

space as

the

to

scanned

times. Thus
divide-and-

approach.

Probabilistic
algorithms

If we use

we discussed

a deterministic

Algorithms
so

far

were

deterministic

\342\200\224

every

step

was predetermined.

same input, we will get two identical

algorithm
execution patterns
and results.
Probabilistic
include
steps
algorithms are different.
They
that depend not only on the input but also on results of some random
There are
events.
We will discuss two of them.
We start with
many variations of probabilisticalgorithms.
twice

for the

6.9 Probabilistic

a simple

example

Suppose

with

continue

and

that

we

have

a set

belongs to the \"upper

we
may want to select
example,
is
to
select
the
maximum
option
them

that

already seen that

to

start

the

finding

a more formal treatment.

of

half\"

x {,

numbers

and we want to select

jc2, ..., *\342\200\236,
than or equal to the median).

one of

For
(i.e.,
a \"good\" student
to
her
or
his
One
according
grades.
in the upper half). We have
is, of course, always
(which
the maximum
Another
requires n 1 comparisons.
possibility is
it

maximum-finding
algorithm and
one-half
number
that is greater than

reached. A
half. This algorithm
to prove that it

159

Algorithms

is greater

to

stop
of the

just after
numbers is

the

halfway

definitely

point

is

in the

upper
is not difficult

nil comparisons. Can we do better?

It
to guarantee that a number belongs to the upper half by
So, it may seem that we found an optimal algorithm.
making less than nil comparisons.
is
if we insist on a guarantee.
an
This algorithm,
however,
optimal algorithm
only
a good likelihood that the solution
is correct is
In many
cases, a guarantee is not required;
we could not guarantee that
no collisions
would occur,
enough. For example,in hashing
if they did. (Hashing can also be considereda
them
we
were
able to handle
but
will
become
If we do not
insist
as
on a
apparent
shortly.)
algorithm,
probabilistic
exists
for
an
element
in
the
then
a
better
half.
Let's
finding
algorithm
upper
guarantee,
two random
numbers from the set, xt and jcy, such that / *j. Assume that jc, >Xj. The
take
probability that a random number from the set belongs to the upper half is at least 1/2 (it
are equal to the median).
will
be more than 1/2 if many
numbers
So, the probability that
not
to
the
half
is
at
most
since
both
do
1/4.
But,
Xj and
belong
upper
jc,>jc7, this
Xj
the
half.
is
as
the
that
does
not
to
the
same
Thus,
probability
jc,
belong
upper
probability
the probability
half is at least 3/4.
that xt belongs to the upper
not good
of 3/4 is usually
However, the
enough.
Being correct with a probability
same
We
can
select
k
numbers
at
random
and pick the
can
be
extended.
easily
principle
maximal among
the probability
that the maximal of the k
them.
By the same argument,
half is at least 1 -Tk.
elements
For example, if k = 10, we have a
belongs to the upper
= 20, we have a success
If
success probability of 0.999. If \302\243
of 0.999999.
probability
k = 100, the probability
of error is, for all practical purposes,negligible. The
probability
of a programming
or of an earthquake for that matter,
exceeds
error, of a hardware
error,
that. We now
in
that selects a number
the upper half, with
have
an algorithm
overwhelming
probability,
using at most 100 comparisonsregardlessof the size of the
we
assume
that choosing an element
at random can be done in one operation;
input.
(We
discuss random-number generation in Section
6.9.1.)
This type of algorithm
is sometimes
It may give
called a Monte Carlo algorithm.
a wrong result
with
time may be better than that of
very small probability, but its running
the best deterministic algorithm. Another
of a probabilistic algorithm
is one that
type
never gives a wrong
but its running time is not
It
result,
guaranteed.
may terminate
or
an
run
for
it
time.
This
of
which is
arbitrarily
long
quickly
may
type
algorithm,
is useful if its expected
sometimes calleda Las Vegas
time
is
low. In
algorithm,
running
that solves a certain coloring problem. In
Section
6.9.2, we show a Las Vegas
algorithm
Section
for transforming some Las Vegas
we describe an elegant
6.9.3,
technique
the technique
to obtain an efficient
into deterministic
algorithms
algorithms. We apply
of
6.9.2.
the
Section
This technique,
for
deterministic
problem
coloring
algorithm
requires

is impossible

about

Involving

Sequences and

transform

every

Algorithms

160

cannot

however,

efficient

Sets
Las Vegas

efficient

an

into

algorithm

deterministic algorithm.

The idea of

has direct analogies to mathematical

proof
algorithms
is a powerful technique.
to
combinatorial
properties
prove
Using probability
techniques.
a set of objects, the probability
that an
the idea is to prove that,
a nutshell,
In
among
object has certain properties is greater than 0, which is an indirect proof that there exists
to algorithms
in the following
This method translates
these
an object with
properties.
with
certain
we know
for
an object
and
we
are
that
searching
properties,
Suppose
way.
will
the
it
desired
with
nonzero
that if we generate a random
satisfy
object
properties
that
the
desired
We
to
follow
is
a
exists).
(this
object
try
probabilistic proof
probability
then
random
events
when
the
the probabilistic proof by generating
appropriate,
finding
this
We
can
times
until
we
some
repeat
process
many
positive
probability.
object with
in our favor, we end up with
an effective
Las Vegas
If the probabilities work
succeed.
probabilistic

algorithm.

Numbers

Random

6.9.1

Probabilistic

for doing

methods

numbers

according

we select numbers

that

require

algorithms

efficient

We

random.

at

deterministic

must

find

procedure
generate
of the procedure.
If the
scheme is completely deterministic,
then
the numbers generated cannot be random
in the
true sense of the word.
in a specific way. Fortunately, this
They will relate to one another
is not a major practical problem. In practice,
it is sufficient to use pseudorandom
numbers.These numbers are generated by a deterministic procedure\342\200\224
and thus are not
\342\200\224
random
but
the
makes
the numbers
truly
procedure
any
among
relationship
that.

However,

any

to some fixed scheme,depending

on

will

the steps

by most applications.
It is beyond the scope
of this book to discuss this
in depth.
issue
We restrict the
discussion
to one very effective method, called the linear
for
method,
congruential
numbers.
The first step is to choose an integer
seed r(l),
generating
pseudorandom
unnoticable

which is a

r(/) =
where

can

t,

and

record

- 1)-ft+1)mod

t are

and

its decimal representation

same

numbers.

range

0 to

t- 1.

avoid

The numbers
We

appropriate factor (t

can
should

end

should

hitting

by

the

t must

and

be quite

with

some bad

generated

linear

be done

large,

in

be

about

should

a*21,

with

cases that

a different range
chosen
to be a multiple

achieve
be

are

numbers

f,

constants.

guidelines are designedto

in

time

current
the

rule:

following

The selectionof ft
the following guidelines: t should
suggests
be a power of 2 (or 10)if that is convenient;

ft

[1981]
and

(r(i

by some external means (e.g., the

of one's favorite team). The rest of

at random

selected

number

microseconds, the current

computed according to the

cause

range).

at least,

one digit less than

These (strange)
repetitions

many

multiplying

of that

millions

the

even.

congruential

by

carefully. Knuth

method are
the

numbers

of

the

in

the

by the

6.9 Probabilistic

Let S be
each

a set

with n elements, let

coloring

the

that

this

elementof

S, contains

subset

each

of

a collection

be

Sk

subsets

distinct

of S,

k<2'~2.

that

one of

S with

one

least

at

two

red

colors,

red and

at

least

one

is called
a valid coloring. It turns out that, under
there
is
a valid coloring.
We present a simple
conditions
subsets,
always
a probabilistic
that is adapted from
a
algorithm
proof of existence of such
The algorithm is almost as simple
as possible:

that

satisfies

condition

on the

given

probabilistic

coloring.

element of S and

Take every

1/2) independently

probability

This

Color each

Problem

or blue, such
blue element.

S\\,S2,...,
such

r elements,

exactly

containing

The

Problem

A Coloring

6.9.2

161

Algorithms

of

probability

of the

coloring

The probability

failure.

that at
probability
< 1 /4 (because
kl~r

it either

not always

does

obviously

algorithm

color

least one of
of the bound

the

on

k).

lead to a

valid

subsets

is colored

Hence, the probability

also a probability of at

Let's

coloring.

of S,

elements

all

that

red or blue at random

of the other elements.

(with

calculate

are coloredred is 2\"r.

red

only

that

is

no more

a random

the
The
than

coloring is

most
most 1/2 (sincethere is
1/4 of a subset entirely
This is a proof that
a valid
always exists (otherwise the
coloring
that the random algorithm
is
of
failure
must
be
It
also
probability
exactly 1).
implies
We
check
the
We
can
test
the
of
a
validity
coloring.
simply
very good.
easily
particular
colors.
We have a 50-50 chanceof
elements
of each subset until
we find two of different
we need to run
success.
If we fail, we simply try again.
The expected number of times
the algorithm
is clearly a Las Vegas algorithm,
to get a valid coloring is 2. The
algorithm
because
we check each coloring and terminate
only when we find a valid one. This is a

is at

valid

not

colored

blue).

simple application of probabilistic

not

often

a valid

6.9.3

so simple.

methods.

Next, we show

that

probabilistic
algorithms are
can be modified such that it finds

Unfortunately,
this

algorithm

coloring deterministically.
A

for

Technique

Transforming

Probabilistic

Algorithms

into Deterministic Algorithms2

We

now

show

a deterministic
algorithm.
algorithm

We
into

to transform
use induction
the probabilistic coloring algorithm
into
algorithm. The technique we presentdoesnot work for every Las Vegas
it is possible
to transform
do not believe that
efficiently
every Las Vegas
a deterministic
is interesting,
however, because
algorithm. This technique

how to

This section can be skipped

at

first

reading.

it

Involving Sequences

Algorithms

162

idea of

the

employs

algorithm

resulting

problem,

general

Let S be

a set

of n elements,

The probabilistic

of S.

subsets

be efficient and
some of the restrictions

not only

removing
again

induction

the

strengthening

will

and Sets

in
a powerful
but will also

on the

imposed

on the fact

based

way. The

solve a more

problem.

original

of

a collection

...,\302\243*be

S 1,^2,

and

was

algorithm

hypothesis
deterministic,

the

that

distinct

that

probability

is at least 1/2. Supposethat

valid
coloring
by coloring each element at random
that
the probability
that we get a valid
color an element either blue or red such
that this
of the rest of the elements
by a random coloring is nonzero. We claim

we get a
can

we

coloring

to an

lead

will

algorithm

by

probability of successremains
we are

Since
induction

hypothesis

The most

important

denote the

size

2~s,

The

l.

F(/i)

such

that

condition

of subset

probability

is not

elements

to

trying

+
\302\2432-5'

We are on

require

that

of

probability

S, is

that

probability

(i.e., the

of

we must

time,
all

with

Let s(
color

one

only

size.

same

the

nonzero.

remain

colored

the

strengthen

be of

subsets

success

that

probability

of

the sizes

solid grounds

Induction hypothesis: We
of

the

is that

at

coloring of

a random

is
all

as

the

long

sets,

but we

as F(n)

< 1.

write

it

function

as a

Let'stry

the

of

following

hypothesis.

provided
If one

one element

no longer

that

I.

F(n) is a function

induction

such
can color one element
color all elementsby induction.

If we

n.

we can

a valid coloring) is no more than

convenience.

for

then

handle
we

Sh The
of failure

This probability
n

on

induction
nonzero,

the

that

subsets

F(n)

has

know

how

to color

a set S with

<

n elements,

< 1.

only one

element,

then

this

element

contributes

to

F(n),

so

to be distinct,
F(n) cannot be less than 1. If a? =2, then, since the subsets are assumed
there can be only one subset with the two elements,
blue
and we can color one element
and one element red. Hence, the base
case is established.
We now try to reduce the
to one for n - 1 elements.
coloring problem for n elements
\342\200\224
Let a be an arbitrary
blue
element
of S. There are two possible
ways to color a*
or red. Suppose that x is colored blue. What
is the probability
that a random coloring of
the
other
n - 1 elements
is valid?
A subset S, that does not include x has the same
\342\200\224
of failure
a has one fewer
2~s,+ . A subset Sj that
includes
probability
namely
element, and it only needs to have at least one red-colored element (it already has a
blue-coloredelement).
subset Sj is 2 s'
to color
the probability
of failing
Thus,
Noticethat this probability is the same as it was before we colored a! Therefore,F(n)
remains
less than
Does that mean that
1, and we now have to color only n - 1 elements.
we
now have an algorithm?
No. It means only that the first choice
can be made
After
the first choice is made, the problem
is different.
arbitrarily.
We can no longer use the same induction
because, after we color the
hypothesis,
need to be colored with
first element, some of the subsets
two colors,
and some of them
one color. We have to strengthen
need
to be colored with
the induction
only
hypothesis
further to reflect this change.
that some elements are already
Suppose
A subset
colored.

6.9 Probabilistic

of four

163

Algorithms

subset has red and blue elements,

in which case we
the
one
subset
has
at
least
element but no
red
further;
(2)
any
blue elements, in which case at least one of the uncolored elementsmust
be colored
blue;
(3) the subset has at least one blue elementbut no red elements, in which case at least one
be colored
of the uncolored elements must
has no colored
red; and (4) the subset
a subset
in state (3) a blue
elements. We call a subset in state (2) a red subset,
subset,
and a subset in state (4) a neutral subset. Let ut be the number of uncolored elementsof
subset
5,-. If St is in state (1), then it is already colored successfully. If 5/ is red or blue,
the probability
of failure in coloring
it randomly
is 2~u>. If 5/ is neutral, then the
then
of failure in coloring
it randomly
is 2~M,+ . Let/
denote the probability
of
probability
We have to maintain the property
failure in coloring subset S, randomly.
that
be in one

may

do not

(1) the

states:

it

to consider

have

F(n)

(6.1)

2fi<l.
/=i

induction

The

to include

problem

must reflect the

red,
blue, and neutral
arbitrary
hypothesis

collection of subsets,each
is
(6.1) satisfied.
now a

The Problem

at

The induction
extension

is the

hypothesis

of the

each

red

the

The base

We

each leading

We extend the
words, the input is

all subsets.
In other

assume

We

neutral.

that condition

element

subset

hypothesis for

this

(nonstraightforward)

problem.
of the

conditions

case is similar

that (6.1) is

blue, or

straightforward

Induction hypothesis:We
satisfy

red,

of

subsets.

of 5 with one of two colors, red

contains at least one blue element,
at least one red element, and
each
neutral
least one red elementand at least one blue element.
Color

or blue, such that each

each blue subsetcontains
subset contains

labeled

status

to

know
how to color a set S with
problem, providedthat (6.1) is

< n

elements

to

satisfied.

n elements
the previous base case. Given a set 5 with
one element of 5 and to leave (6.1) satisfied.
color

such

satisfied, we need to
pick

again

to

an arbitrary

different

element x e

S. There

are

two

statuses of the subsets.

If we color
(but with one less uncoloredelement),

possible

x red, then

ways to color jc,

all

red

subsets

red
all blue subsets containing
containing x remain
x become successfullycolored(and can be removed), and all neutral
x
subsets
containing
become red subsets. Subsetsthat do not contain x are not changed.
leads
x
blue
Coloring
to similar
We can now compute the value
of F(nwe denote
1), which
changes.
by
x
red.
We
color
we
that
also
to
indicate
denote
the
1),
FR(n
by FB(n 1)
corresponding
value of F(n1) in the case when x is colored blue. The key to the algorithm is the
following

lemma.

Involving Sequences

Algorithms

164

and Sets

6.2

Lemma
\342\226\241

be

Let F(n)

the

of failure

probability

initially,

x red,

coloring

after

failure

of

probability

of failure

probability

blue.

coloring

after

and

let

FR(n-\\)

be

the

let

FB(n-\\)

be

the

Then,

FR(n-\\)

FB(n-\\)<2F(n).
Proof:A subset that

Its contribution
x remains
to
does not contain
unchanged.
with
is
the
claim.
which
consistent
is
the
We
and
same,
F(n)
FB(n-\\),
/^(fl-i),
consider now the subsets that contain x. There are three possibilities,
according to the
it
is now
because
to
subset
contributes
A
red
status.
subset
(1)
FB(n-\\),
nothing
to
is twice as much
as that
to FR(n-\\)
colored; its contribution
F(n),
successfully
with
the claim.
this is consistent
because it has one fewer element. Again,
(2) The case
of a blue subset is the same as that of a red subset. (3) A neutral subset with w, elements
+'
one less element.
red or blue with
This subset becomeseither
to F(n).
2~\"'
contributes
In
either case, its
2~(M_1)
and
to
both
contributes
Thus, it
FR(n-\\)
FB(n-\\).
is
the
claim.
to F(n), FR(n
\342\226\
contribution
1) the same, establishing
1),and FB(n
6.2 leads

Lemma

directly to

because, for (6.1) to be satisfied,

the element. If there is only

the

can

there
one

one red or one

can color the

be only
then we

subset,

of one element

base case

The

algorithm.

is simple,

subset

blue
element

containing
the other

with

color. If FR(n-\\) + FB(n-\\)<2F(n\\then either FR(n- \\)<F(n) or FB(n- \\)<F(n)

We
can compute
these values and color x blue if FB(n-\\) is less than
(or both).
and
color
x
otherwise.
Lemma
red
6.2, condition
(6.1) in the induction
1),
FR(nBy
is
of this
follows.
We
leave
the
and
the
satisfied,
hypothesis
implementation
algorithm
algorithm to
6.10
Let

the

reader.

a Majority

Finding
\302\243
be

a sequence

in E.

x appears

times

Given a sequenceof
or determine that none exists.

Problem

The

sequence

For example,
whether

of integers x {, jc2, ..., xn. The

A number
z is a majority in

an

someone

integer

can

won the

used effectively to solve the

possible candidates is very large (the

can be
We

assume

here

a vote

represent

election. If the

that there is no

limit

are represented as arbitrary

integers.
example, to achieve consistencyof

problem

multiplicity
E if

find the

numbers,

in

number

an

of x

election,

of candidates

in O(n)

in

its multiplicity

E is the number of
is greater than nil.

majority

and the

in

the

problem is to

is small, then

bucket

find

sort

time. However, if the number of

then bucket sort cannot be used.

sign of the times),

the number of possible candidates,and that they
for
is also performed in computer
Voting
systems,

on

decisions.

6.10

This

is an excellent example of a straightforward

solutions are not as efficient
as an elegant solution
more
efficient and simpler to implement.
We

a Majority

problem

straightforward

thinking

Finding

but

is

whose

problem
that

first

165

requires

some

several

discuss

problem, and then present the elegant algorithm.

is to use sorting. Once the
The most straightforward
way to solve this
problem
it
is easy
to count how
votes
are sorted,
each candidate got. Sorting,
votes
many
in the worst
We will see that
it is
case.
however, requires O (n logn) comparisons
to
do
better.
We
can
also
use
a
If
there
is
a
median-finding
algorithm.
majority,
possible
it must be equal to the median
smallest
then
element, and the
(the median is the (n/2)th
is found, we can
more than nil times). Therefore,once the median
appears
majority
it
if
is
the
number
of
times
and
the
median
not
a
then there is no
count
appears,
majority,
is easier than
this is a better approach.
the
median
sorting,
majority. Since finding
is to use a probabilistic algorithm.
We
can pick a small random
Another
approach
of the votes, take the majority
of the sample, and count
the number of times this
sample
in
it
is easy to verify
the
whole
list.
that
a
However,
although
majority
sample
appears
with
this
it
is
to
there
is
no
vote
is
a
that
impossible
algorithm
prove
majority,
given
majority. The outcomeof such an algorithm may be \"undecided.\" (This is the method
are indeed \"too close to call.\
used for public-opinion polls; someelection
predictions
size of the sample.
It is also not easy to determine the
appropriate
to find a majority that
can
handle
We
now
a linear-time algorithm
any
present
is faster and simpler than
the
number of candidates. The algorithm
median-finding
for finding a celebrity (Section5.5),we first try to
As we did in the algorithm
algorithm.
It turns
out
eliminate as many
elements
as we can from being candidates for majority.
is
this
one
that we can eliminate
candidate
all but one element.
Finding
helped by the
which allows us to reduce the problem to a smaller one:
observation,
following
If

to the

approaches

straightforward

Xi * Xj

majority

in

and we
the

eliminate

original

both

of

list remains

these

a majority

elements
in the

from

the

list,

then

the

new list.

but if we remove 1
the opposite
is not true: the list 1,2,5,5,3 has no majority,
5 becomes a new majority.)
in the
smaller
find the majority
So, if we find two unequal
votes, we eliminate both,
it
is
in
not
check
whether
the
list.
if
we
do
find
and
a
What
list,
majority
original
unequal
votes? If we scan the votes and they are all equal, then we have to keep track of only one
possible
candidate; once we find a vote that is not equal to this one candidate, we can use
the
If all the remaining
observation
above.
votes
are equal, then we have to check only
one candidate.
This is the seed of the idea; we now show how to implement it.
The
votes
are scanned in the order
C
variables,
they appear. We use two
When
and M (multiplicity).
we consider
is
for
the
candidate
C
(candidate)
xh
only
so far
C appeared
majority among x,, x2,...,*/_i, and M is the number of times
In other
eliminated.
the
times
was
C
the
votes
can
be
words,
excluding
x{,x2, ...,*/-i
into two groups of sizes 2k and M, such that 2\302\243+M = /-l,
divided
the first group
can be eliminated by the observation),
votes
k pairs of unequal
contains
(which
and the
If there is a majority
M times.
x
second group contains C appearing
\342\200\242\342\200
among
i, x2,
*,_! ,
elimination
this
then it must \"survive\"
scheme, and so it must be equal to C. (Notice

(Notice that

and

2, then

166

the

that

again

majority.)

is

opposite

not

survive
it to C,

C may

true;

consider

we

When

and Sets

Sequences

Involving

Algorithms

jc,- we

compare

the

elimination

and

either

being the
or decrement

without

increment

or not Xj is equal to C. We also have to take care

will
for example, at x3 if jc2 ^-^1)no
candidate
(which
of the case
happen,
we
set C = jc, and M = 1. At the end,
This case occurs when M is equal to 0, and
simply
the number
of times C appears in the
one candidate
C, and we can count
we have only
is no majority.
The
there
it is the majority or whether
list and determine whether
the

on whether

depending

multiplicity

of having

in Fig.

is given

algorithm

6.28.

(X, n) ;
of size n

Majority

Algorithm

Input:

array

(an

Majority (the

Output:

of positive

numbers).
if it exists, or -1

in X

majority

otherwise).

begin

C:=X[1];

M:=l
{

;
all

eliminate

scan;

first

but

candidate

one

C}

for i := 2 to n do
ifM = 0 then
C:=X[i];

M :=

else

ifC = X[i]thenM:=M+1
M

:-

else

{ second

scan; check whether

ifM-0

then

C is

a majority

:= -1

Majority

else
Count :=

for i :=I

0;
= C then

ifX[i]

if Count

n do

to

Count

:= Count

then Majority

> nil

elseMajority

:=

+ 1;

:= C

-1

end

Figure 6.28

We use

Complexity

the worst
at

most

2az-2

3n/2+ 1, and
constant

- 1
comparisons

case, to determine

number

whether

this

Algorithm

Majority.

to

a candidate

find

candidate

and n

is a majority. Thus,

comparisons,

in

overall, there are

It is possible to reduce the number

to
of comparisons
comparisons.
is optimal (Fischer and Salzberg
there
are
In
since
case,
[1982]).
any
of other operations per comparison,the overall
time is O (n).
running

that

6.11 ThreeProblems

167

Techniques

Exhibiting Interesting Proof

Three Problems

6.11

Proof

Interesting

Exhibiting

Techniques3
this

In

we present three

section,

Each algorithm
the

is an

of

principle

strengthening
times during

hypothesis

four

algorithm.

The second

6.11.1

not

a sequence

have Xi<xir

The

we

xit,

obvious

an

strengthen

efficient

\342\200\224

technique

improving

that this

shows
to

improve

an

wisely.

xx,x2,...,xn.

subsequence (IS)

An increasing

''' < /*,

such

(LIS)

subsequence

for all

that,

of 5 is

longest increasingsubsequenceof

< k,

<j

an

we

increasing

a given

integers.

this section is

excellent

an

We

hypothesis.

of problems

a result

will

encounteredin

the

principle of

of the

example
the

strengthen

hypothesis
attempt.

previous

several
Consider

induction.

straightforward

Induction

induction

Subsequence

increasing

longest

in
develop
the induction
as

induction

integers

Find a

of

know

of

an example

with ix<i2<
jc,2 ,..., xik,

of distinct

algorithm

the

distinct

Problem

times, each time

first

of

length.

strengthening

the

Increasing

of maximum

sequence

The

of

the base

choosing

subsequence

algorithm

always

a subsequence

5 is

of this

development
is

the
an
to
solution, leading
We

hypothesis.

unnecessary assumptions. The example

This third example shows how
straightforward.

Longest

5 be

Let
of

the

technique.

proof

induction

sequences and multisets.

The first algorithm utilizes

involving

problems

different

by eliminating all

the \"theorem\"

principle is
algorithm by

unrelated

of a
the

example

hypothesis (first attempt):

how to find

a longest

Given

sequence

of size

<

m,

we

subsequence of it.

increasing

Given
a
which the problem is trivial.
- 1 elements, and consider
xm. If xm is
then
greater than the last element in the LIS, given by the induction,
xm can be appended
to the LIS, creating a new longer LIS, and we are done. Otherwise, however, it is not
clear how to proceed. For example, there
be several different LISs and
xm may
may
extend one of them,
the one found
but not necessarily
The next step
by the induction.
seems to be a strengthening
of the induction
hypothesis as follows:
The

base

sequence

case consists

of size

Induction

we
*

know

m,

we

of sequencesof
find an LIS of

(second

hypothesis
how

to find

This section can be skipped

at

all
first

the

first

attempt):

longest

reading.

1 for

size

its

Given a

increasing

sequenceof

subsequences

size

of

it.

< m,

The

check

is still trivial.

case

base

all of

against

xm

and Sets

Involving Sequences

Algorithms

168

We use induction

the LISs

in the

same way,

a longer

whether

find

and

have to

\342\200\224
we now

another

we can

except that now

IS exists. This attempt

solves

LISs.
problem
be
IS
1
less
still
an
of
than
the
LIS,
may
length
any
new
seems
we
will
a
LIS.
It
that
have
extend
create
can
which
it,
gotten
longest, and xm
have
to find all ISs of largest and second
into a hole, because we now
ourselves
largest
ISs,
largest
length. But to find all the second largest ISs, we will need to find all the third
where
the
then all fourth largest, and so on. This is a good example
strengthening

the previous
If xm cannot

induction

problem,

it introduces

but

extend

We

there

then

is overdone.

hypothesis

look back at
only to know

Let's
need

all

find

the

induction

stronger

extend

xm can

whether

hypothesis.
one of them.

Do we really needall LISs?

we somehow

Can

find

the

is positive.
The best LIS is
of extension? The answer
one
\"best\"
the smallest
If we can extend any LIS, we can surely
number!
that ends with
one
the
LISs
the same
that end with
extend this one. (There may be several different
number,
we talk about
and they are all equivalent in terms of extension potential. For simplicity,
best one.\")
Let's try another
induction
\"the best one\" instead of \"an
arbitrary
a
this
one
little
weaker
than
the
last
one:
hypothesis,
terms

in

of potential

Induction hypothesis (third

how to find

know

increasing

longest increasingsubsequenceof

The basecaseis
the
then

found

LIS

have

we

so

still

Given

trivial.

by the induction.
a new LIS, which

it is

Given

attempt):

a longest

we

jcm,

is

can

longer

no other

whether

determine

it

can

be appended

s. If
of length
than the previous one; thus,
and we are done. Otherwise,
we are still not done. It may

that

Assume

we

that

subsequence of
a smaller last number.
such

it,

it has

of size < m,

a sequence

the LIS is

xm can be

LIS is

new

this

know

to

added,
that no

one,
definitely
be the case that
longer increasing subsequence exists. But
be added to the best LIS (since it is smaller than the last number in that LIS),
xm cannot
but it can be added to an IS of length s - 1, making
the latter
an LIS with a smaller
last
number. To accountfor this possibility,
we need to know
the best IS of length
s - 1. But
then
if the induction
the best IS of length
states that we know
5-1,
again,
hypothesis
then xm may extend an IS of length
We
s-2
new
IS
of
s1.
it
the
best
length
making
will
have
to be able to determine whether
in
with
IS
order
to
extends
such
an
xm
proceed
the induction.
s - 2, s - 3, and so on down
So, we will need to know the best IS of length
to the best IS of length 1, which
so far.
is simply
the smallest number in the sequence
(Even without
ISs cannot
be discarded
shorter
one can see that
induction,
using
\342\200\224
there
is always the possibility
that
one of these ISs is the start of the final
arbitrarily
unique,

the

\"best\"

we

LIS.)

Yet again

we

try

to

increasing subsequence of
number (if there is more than

denote

by

B\\S(k).last

Induction
know

how

the induction

strengthen

best

the last

one

hypothesis.

k \342\200\224
namely,

length

such

number

(fourth
hypothesis
to find BIS(k)for all

in

the

the

subsequence
sequence

denote

We

one that ends

we take

an

BIS(\302\243) the

the

arbitrary

smallest

one).

BIS(\302\243).

attempt):

Given a sequenceof

if

<m-\\,

by

with

they

exist.

size

< m,

we

We

The base case remains

so

> B\\S(k).last,

xm

xm at the end is better than

\342\200\242
\342\200\242
\342\200\242
< BIS(s)./as/,
where s is

with

< B\\S(J-

B\\S(J).last

better than
values of

\\).last

1).

for
B\\S(i).last,

the
and

induction.

(2) xm <

BIS(\302\243+

1).

that BIS(l)./atf

of

the LIS.

This claim is

y, then the

first

j-

so

\\).last,

BIS(\302\243)

< B\\S(2).last<
true

would be

BIS(s)

and

it

unchanged.)
is quite

xm,

creating

we have

BIS(/>m.

simple
binary

by

performed

with

Otherwise,

replace BIS(/ + 1)with

We then

).last.

extend

we

once we use the right

because we are

search,

searchinga sorted

each xm adds at most O (logm) comparisons,and

set.
Hence,
time is 0(n\\ogn). We leave it to the reader to complete the details
is not a straightforward task.
algorithm, which

the

Finding the Two Largest Elementsina Set

6.11.2
common

which is

technique,

for

thoroughly

proof

\"One should
assumptions;
should

boundaries

simple,but

and second

are

assume

or steps

that

almost
not

are

any

is to

theorem,

essential.

search the
such

Removing

from

that

a?

the
is a

better
the

theorem.

proof

set S of

n numbers

\\,

jc2,

..., xn,

the

number

find

the

first

largest of them.

for

looking

elements

that

Given a

Problem

The

We

in

an indication

sometimes

in proving

important

assumptions

is
also
Having inessential assumptions
be
and
[1927]:
Polya
wrong.
Quoting
Szego
may
in fact
scrutinize
each
made use of all the
proof to see if one has
fewer
. .. and
one should try to get the same consequence from
assumptions
at
the
not
be satisfied
until
show
that
one
has
arrived
counterexamples
is true for algorithms. This principle sounds
of the possibilities.\" The same
times it is not, as seen in the next
example.
many

assumptions results

one

total

of this

running

if

because,

of BIS(/)

numbers

can

algorithm proceeds as follows. Given

jtm, we look at the
= s9 s- 1, s-2, and
so on, until we find
one,
say B\\S(J).last,
If no such j exists, then
xm is the smallest number in the

is basically the whole

algorithm,
Notice that the search can be

This

following

it

occur: (1)

conditions

claim

which is smaller than xm.

so far, and it becomes
If j = s, then
BIS(l).
sequence
remains
a new BIS(s+l).
(The previous
BIS(^)

B\\S(j).last <xm< BIS(/+

two

the BISs

of

which

find

169

Techniques

We

size

the

have to

BIS(\302\243),

BIS(\302\243+

some

for

\342\200\224The

BIS(y

to

be added

can

xm

xnn we

and only if

BIS(\302\243) if

Proof

Interesting

Exhibiting

Given

trivial.

a certain

xm extends

change.

Problems

Three

6.11

an

set.

algorithm

that

minimizes

We ignore other

power of

only

operations.

Furthermore,

of comparisons
for

simplicity,

of
we

2.

the set S of size n into

divide-and-conquer technique, by dividing
two subsets P and Q of size nil. If we use straightforward
we assume that we
induction,
know
the first and second largest elements of P and
them by p ,, p2, and q\\,q2
Q, denote
and we try to find the first and second largest elementsof S. It is easy to see
respectively,
that
to find the first and second
two more comparisons
are necessary and sufficient
largest
is between the two
maximals
elements of S. One comparison
p x and q x, and the other
We

try

the usual

170

Involving

Algorithms

one is betweenthe

approachleads to

\"loser\"
the

and Sets

Sequences

and

recurrence

the second largest of the \"winner\"

relation T(2n) = 2T(n) + 2,

This

Fig. 6.29).

(see

7(2)=1,

is

solution

whose

2n -3 comparisons,and it is very
than
the straightforward
T(n) = 3n/2-2. Thisis better
minimal
the
maximal
and
elements
of
the
to
similar
finding
presented in Section
problem
to
do
even
better.
want
We
6.5.1.
are necessary for the inductive
can
we
If the two comparisons
step, then how
at the comparisons
of comparisons?
in Fig.
Looking carefully
improve the total number
in the algorithm.
Therefore, the computation
6.29, we see that q2 will not be used further
If we can avoid this computation,
was
then
we will
its
to
unnecessary.
discovery
leading
of comparisons.
to
we
do not
However, until we compare
save significant number
p {
q {,
If
or
can
be
we
had
known
which
subset
was
whether
know
p2
q2
ignored.
going to
have
the
we
could
used
for
this
then
subset,
\"lose,\"
regular
maximum-finding
algorithm

So, we suspectthat
saving many comparisons.
which
ones they are.
we do not know

quite

a few

comparisons can

be avoided,

but

The trick is to delay the computation of the second

We keep only a list of candidates for second largest,and
the second largest element in the induction
hypothesis:
Induction hypothesis: Given a set of
maximum element and a 'lsmall' set of

size

< n,

candidates

largest
we

we

do not

know

element

assume

how

until

the

that

we

to find

the

end.
know

for the second maximum

element.

We have not defined a value

in the hypothesis. We will discover
for
\"small\"
the
appropriate value when we develop the algorithm.
The algorithm
as follows.
Given a set 5 of size az, we divide it into two
proceeds
P and Q of size n/2. By the induction
subsets
the largest
elements
hypothesis, we know
of the two sets, p { and q i, plus a set of candidates
for the second largest, CP and Cq.
We
compare p, and q, and take the largest, say p x, to be the maximum of 5. We then

pi

p2

\302\2532

Figure
correspond

6.29

the largest
Finding
comparisons).

to the

and

second

largest

elements

(the dashed

lines

Three Problems Exhibiting

6.11

discard Cq, since all

and add

are less than

end, we get the

of Cq

elements

q x to

only

At the

CP.

Proof

Interesting

most

is at

q { which
largest

second

the

largest,

set of candidates

and a

element

171

Techniques

we choose the second

of comparisons
largest element directly. The number
=
the maximum satisfiesthe recurrence
relation
T(n)
2T(n/2) + 1, 7(2)=1,
=
that
It
n1.
is
to
see
that
is
a
sufficient size for the
T(n)
implies
easy
log2Ai

which

from

for finding
which

additional

-1

\\og2n

n-

consider. Therefore, finding

The total
comparisons.

the set we

of

size

we add one more elementto

set, because

candidate

Induction hypothesis:Given a set

element and a set of at

maximum

for

eliminated.
by

of size
most

be a multiset
of (not
(jc!, jc2, ...,*\342\200\236)
ordered set. A mode of a multiset is defined

is called

its

(there may be more

will

minimize the

be required is that
mode is

try

the

mode of a multiset
This is not easy
element

may

not

finding

then

high,

Let's

is

candidates

for

find

the

second

the

there

the

necessary.
the majority (Section

may be

can scan

a fast

way

suspect
it

multiplicity.

that,

without

to

way

sorted

mode
sequence and

find the

sorted sequence). We
that

thinking

6.12) can be done

of finding

induction
straightforward
with
n - 1 elements, and try

the

in the

consecutive

This leads us to

occurs

element

an

highest

One possible

The reason for

always

in

frequently

S.

multiset

are sorted, we
be

of times

with the

element

of comparisons.

will

occurs most

that

a totally

from

elements

distinct)

The number

mode of a given

elements

log a?) time.

0(n

requires

sorting

how to

know

we

n,

necessarily
as an element

mode).

thus

(equal elements

the

multiplicities
see that sorting

one

mode is

number

the

Once

sorting.

than

Find a

Problem

Our goal is to
count

The

multiplicity.

The

is to use

<

\\og2n

The

Modeofa Multiset

5 =

the multiset

of

number

more efficient.

which are

6.11.3 Computing the

Let

double the

element requires
is thus
comparisons

is constructed,
is a good
Once an algorithm
it
idea to examine it
to the final
result.
these parts can be
Often,
parts that do not contribute
if the redundant operations cannot be eliminated,
Even
they
may be replaced

operations,

simpler

set when we
largest

element.

maximum

Comments

second

is the best possible

(see [Knuth 1973b]).
incidently,
for the case when
n is equal to a powerof 2, is thus as follows.

hypothesis,

carefully

candidate

which,

+log2Ai-1,

induction

the
the

if the

may not
whereas
time,

sorting

in linear

of

multiplicity

the

sorting.

approach. We assume that we

to find the mode of an n element

know

the

multiset.

since there may be several elementswith

the highest
the nth
multiplicity;
break the tie. Supposethat the induction
hypothesis states that we know all

the elements with

element breaks the

the
tie,

Then, we can
multiplicity.
highest
but it may also increase the multiplicity

determine
of another

whether

number,

the

nih

which

Sets

Sequences and

Involving

Algorithms

172

track
already seen (Section6.13.1)that keeping
will
be
the
cost
too
On
but
is
high.
probably
possible,
of all
\342\200\224
be
we
can
element
choose
a
Arth
is
not
that
it
the
arbitrary
necessary
the other hand,
maximum
element.
We
still
is
the
have
the
Arth
element
that
special one. Suppose
We
can
but
now we are closer to a solution.
basically the same problemsas before,
but
all
occurrences
of the maximal
reduce the size of the problem
by removing not one
the multiplicity
of the mode
element. We then solve the reduced problem, and compare
element.
of the maximal
with
the multiplicity
of the reduced multiset
it is still
but unfortunately,
too slow. Finding
the
We now have an algorithm,
- 1
n
If
the
multiset
contains
of n elements
of a multiset
maximum
comparisons.
requires
to be

has

now

added to

distinct

a few

quite

In

performed.
this

then

so

disjoint,
two

is basically

the

same

of

of using
the multiset into
lead

they

We

have

multiset

the

seen

already

6.5). It is
although we have not
(Section
in

our

splitting,

multiset

can be

smaller

multisets

linear

we get

time,

the

T(n)<2T(n/2)
which

is basically

algorithm
We

at the
that

now

come

base of the

we can

parts

start

to

the

find

same

divide-and-conquer

The two

size.

of

subproblems of size no
the

disjoint.

Since

familiar

recurrence

+ 0(n)9

7(2)= 1,

the same
to the

heart

induction

more

be

into

multiset

and

the

the median.

of
in

comparisons

the

worst

algorithm

median-finding
find

should

a multiset

parts

than

and

than

induction, we

in the

elements

greater
equal to,
in O(n) expected number
in O(n) time
median

We use the
size n, we first

result.

multiset

induction.
the

the

than,

a median

find

this

\342\200\224
less

T(n) = 0(n \\ogn).

is chosen
at random

element

splitter

parts

of about

that

implies

small set of

or a

easily determined from

are

algorithm,

subproblems. How do we divide

We
can first find the median of
parts?

possible

proved

Given
algorithm.
then solve two

computations will have to be

(i.e., all the elements are distinct),

0(n2) selection sort.

we resort to the

as the
the

a set

fact

to independent

how to

also

in

element

two

equal disjoint
into three

approximately

then split

one

maximum

many

multiset is

the performance

Instead

that

too

then

if the

particular,

improve

technique.
to divide
try

solutions

elements,

algorithm

To

We have

list.

the

\"best\"

different

case,
as a step

median and perform the

The mode of the original
smaller
since the
multisets,
can
be
done in
splitting

the

nil.

of the two
the median and

modes

finding
relation

In fact, if the
this is no better than
sorting.
instead of being the exact
then this
median,

But

as quicksort.
of this

Suppose

from

To
algorithm.
that the multiplicity

improve the
of

submultisets of size M.

In

we look

performance,

the mode
other

is M.
words,

We

claim

we do

not

M. Since
all parts are
into parts smaller than
splitting the multiset
one of the parts of size M must
contain
disjoint,
only the mode. At this point, the mode
will
be discovered
because the multiplicity
of all other elements cannot exceed M.
there
is no need to divide the multiset
Therefore,
any further.
The implementation of this algorithm
We cannot use
is not straightforward.
we do not know
because
The
beforehand
how far to carry out the recursion.
recursion,
when the size of the multiset
recursion should
be terminated
becomes
at most M, but the
the execution
value of M is found
of the algorithm
the smaller
all
during
by checking
In each step, all the submultisets
multisets.
are checked,
and, if none of them contains
have

to continue

Bibliographic Notesand Further

one distinct

only

element, then

only one distinct

to the reader.

contains
details

Complexity

The

which

of

this

number

the

this is

c, then

constant

6.12

and

linear-time

check all subproblems.

If M is
algorithm.

often

try to

therefore

we employed

in

of

Many

be obtained

the size

of

are

this

a major

plays

the

subjects
basic

abstract

are quite

chapter

input

also better

Unfortunately,

discussed

[1973b].

combinatorics,

binary-search
subsequence

paradigm

problem

are

problems

practice,

in this chapter. One

presented
that
problem. The techniques

similar to those introduced

5. Induction

in Chapter

about

presented
can

and

Further

in

searching,
involving

and

Stanton

be found

of Section

in

6.2 is

Manna
from

too

are used

If

only good enough,

the size

Whenever

algorithms

and

algorithms

not

often

most

that

examples.

of the

input

to attempt to find better

solutions.
is quite
for example,
algorithms,

inefficient

sorting

solutions

have straightforward
chapter
search
and selection sort are two

this

in

solutions.

sophisticated

Additional
are

problems

and others.

algorithms,
In

order

sorting,

searching,

for

often

but

is not

The use

of

common.

inputs.4

large

Reading
including
sequences

White
and

their
and

[1986].

history,

Waldinger

Mirzaian

be found

can

sets, as well

[1987],

formal

of a

derivation

[1987]. The
where

in

as topics in
stuttering-

a linear-time

example highlighted this issue unexpectantly when the virus (or worm) that attacked over
the United States on November
2, 1988, slowed down those computers considerably,
all the search algorithms in it used linear search (see Spafford
[1988]).

recent

computers across
because

high

fairly

overhead for

\342\200\224

subject.

of a given

these solutions are

is small,
than

Notes

of material

wealth

parts

\342\200\224
linear

effort

little

these and other

Bibliographic

4A

is

if M

is an

this

role.

the problems
with

as are the

main

the

over 100),it is important

linear-search and quadratic-sorting

Knuth

= cn for

if M

a significant

probabilistic

in each

problems

small (e.g.,

only

a linear

takes

then

constant,

of

a multiset

particular,

sorting

is

chapter

manipulation,

string

two

this

the

in

and simple to define

as clean

not

should

one or

only

presented

they

In

intuitive

An

Summary

We touched on quite a few

statistics, data compression,

can

base:

subproblems).

remembering

again

implementation

of comparisons is 0(n \\og(n/M)).

the recursion is carried out only until
total of \\og(n/M)
times.
Each time it

is thus superior
to
0(n log a?) algorithm. This algorithm
if the cost of comparisons is also high
and
(there

We

in its

submultisets

the

is that

expression

is
size M is encountered,
which
number of comparisonsto divide

some

is modified only

relation

of

We leave the

T(M)= 0(M),

+ 0(n\\

that

implies

explanation

If any

divided.

further

we can terminate.

then

recurrence

resulting

T(n)<2T(nl2)

are

of them

all

element,

173

Reading

6(XX)
partly

Algorithms

174

and Sets

The average performance of interpolation

[1978], and some empirical results are given

is presented.

algorithm

Perl,

Involving Sequences

Itai,

and

Avni

Mergesort was probably

search

was

studied

by

by van der Nat

[1979].
in 1945, and it was one
version
of mergesort was
[1988], and Dvorak and

first developed
by von Neumann
to
be
of the
implemented. An in-place
programs
and Langston
first
by Kronrod [1969]; see also Huang
developed
Durian
Quicksort is due to Hoare [1962]. A detailed
study of quicksort appears
[1988].
A wonderful film
[1964].
in Sedgewick
[1978].
Heapsort was developed by Williams
nine
all
shown
with
beautiful
of
major
sorting
techniques
descriptions
containing
at
the
of
Toronto
was produced by the computer
animation
graphics group
University
has
been
studied
for
there
are still
Even
extensively
many years,
sorting
though
[1981].
exact
number
of
The
for
n
numbers
comparisons
required
sorting
open problems.
many
outlined
in Exercise 6.30 is by Ford and Johnson [1959].
is still unknown. The algorithm
for some time in terms of number of comparisons,but it was
the \"champion\"
It was
used
to
be
not
[1979]. Another widely
by Manacher
sorting
algorithm is
optimal
proved
is
Shellsort
and
to
Shell
shellsort invented
[1959].
very
by
simple
easy
implement.
is still unknown; see Incerpi and
for recent
However, its complexity
[1987]
Sedgewick
Decision trees have been used successfully
observations.
to prove
results and empirical
several
Moret
a
bounds
for
basic
of
their
uses.
lower
[1982] presents
problems;
survey
selection
An analysis of the probabilistic
and Rivest
algorithm was given by Floyd
deterministic
[1975]. A linear-time
algorithm for order statistics was first developed
by

stored

first

Blum,

Floyd,

Pratt,

and

Rivest,

the constant
CD(n)
a
median
presents
finding
bound
(on the number
since

is

very

algorithm
of

Tarjan
high.
with

comparisons)

time
However, the running
and
Schbnhage, Paterson,
Pippenger
at most 3n comparisons. The best-known
for finding the median is 2n (Bent

[1972].

is

in

fact

[1976]
lower
and

John

order
statistic
paper contains results for the general
problem; the
for the general lower bounds are more
expressions
complicated.
Data
has been studied widely
due
to its great importance. The
compression
in
Section
6.6
is
due
to
Huffman
also
Knuth [1973a]). Variations
[1952]
algorithm
(see
of Huffman's
that
use
one
are
described
Knuth
and Vitter
[1985]
algorithm
only
pass
by
Another
More
effective
and
is
due
to
Ziv
and
[1985].
[1978].
popular algorithm
Lempel
on data compression in general
can be found in Lynch
[1985].
The string-matching algorithms
in Section 6.7 are due to Knuth,
Morris,
presented
and Pratt [1977], and to Boyer and Moore [1977]. Galil [1979]improved
the worst-case
time of the
of the
More on the
running
Boyer-Moore
algorithm.
complexity
be found in Guibas
can
and Odlyzko
Boyer-Moore algorithm
[1980] and in Schaback
[1988]. Empirical comparisons betweenvarious
algorithms can be found
matching
string
in Smit
A
was
[1982].
probabilistic string
developed by Karp and
algorithm
matching
Rabin
short
to
make
This
of
idea
uses
the
[1987].
fingerprinting
algorithm
be
also
of
It
can
so that they can be comparedefficiently.
representations
large strings
to
used with two-dimensional
be
extended
can
The string matching
patterns.
problem
look for patterns
are
than
\"wild cards\"
more
For
complicated
just strings.
example,
we may want to search for all occurrences
of strings of the form
useful;
B*C, where B
* denotes
and
C are given strings and
any string. A more general problem is to look for
more
For
on
sets
of
these problems
see Aho and Corasick [1975].
any
strings.
regular
[1985]).

This

175

Drill Exercises

Another

Suffix trees

preprocessed.
and Myers
(Manber

Sankoff

and

in

strings

text
and

fixed

[1976])

McCreight

[1973],

(Weiner

[1990]) allow fast

comparisons

Sequence

search for

is to

problem

important

been

has

that
Suffix

Arrays

search.

their

many

are covered

applications

Various

a book

in

[1983].

subsets is NP-complete(Lovasz

for arbitrary-sized

present many

edited by

Salzberg [1982]
steps) can be reduced
excellent

which

Szekeres [1935] proved,

sequence of distinct

that

by

tournaments,

mode

in

by Lewis

is given in Dobkin
The solution to

set

a very
of
length

was

elegant use of the

length n2 + \\ must
n + 1. The problem

first

Carroll (see
and

one presented

Section

in

6.10,

Using

Fischer

number

the

elements

the

than

structure

decreasingsubsequenceof
largest elements

[1982].

and
of comparisons (but not the number of other
1 in the worst case, and
that this bound is optimal.
to 3/7/2+
of a solution to the longest
subsequence
increasing
description
Erdos and
we
is given by Gries
borrowed
[1981].
heavily)
data

showed

An

[1974]

properties.

and Gries

Misra

in

and Spencer

combinatorial

proving

problem was studied, for example,by

sophisticated

problem (from

Erdos

[1973]).

techniques for

of probabilistic

examples

The majority
a more

are

included

a book
problems involving
strings
in
is
The algorithm given
6.8
due to
Section
edited by Apostolico and Galil
[1985].
in many
This algorithm can be improved
[1974].
ways, including
Wagner and Fischer
times
when the alphabet is
running
savings of storage (Hirschberg[1975]),improved
and
when
the
are close (Ukkonen
and
(Hunt
[1977]),
sequences
Szymanski
large
very
in
A
relevant
results
of
and
[1983].
survey
appears Hirschberg
[ 1985]
Myers [1986]).
finds an element in the upper half is due to Yao
that
The probabilistic algorithm
in Knuth
is covered in detail
number
[1981]. The
generation
[1977]. Random
in Section 6.9.2 is based on a probabilistic proofof
given
probabilistic coloringalgorithm
for
in Bollobas [1986]. The technique
existence
probabilistic
converting
given
in
is due to
which
was
illustrated
Section
deterministic
to
6.9.3,
algorithms,
algorithms
use
this
to
solve
the
of
Section
The
of
[1986].
technique
coloring problem
Raghavan
6.9.2was pointed out to us by K. Pruhs. The general problem of finding a valid coloring
Kruskal

and

Munro

suggested,

[Knuth

1973b]).

in

principle,

pigeonhole

have

either

of finding
the

context

Another

[1980] (see also Gonn

the

that every

an increasing or a
and second
largest
of arranging

algorithm

for

finding

tennis

the

't [ 1984]).

Exercise 6.27 is discussedin Aho, Hopcroft, and Ullman [1974].

to Exercise 6.39 is
Karp, Saks, and Wigderson [1986]. A solution
in
is
6.42
discussed
in
The
Exercise
Choueka,
Rodeh
of
Fraenkel,
given
[1982].
subject
was
The
of
and
Perl
realizable
notion
Klein,
6.64)
[1985].
sequences(Exercise
introduced by Ryser [ 1957].

Exercise6.34is

from

Drill Exercises

6.1

well-known
strategy for the following
game: One player thinks of a number
other
The
player attempts to find the number by asking questions of the
range
than) a?\" The object is to ask as few questions as
form \"is the number less than (greater
that nobody cheats.)
possible. (Assume

Design

in the

a good

to //.

176

6.2

Algorithms

choiceis unknown

6.1 when the range of

to the guessing game in Exercise
the chosen number
may be any positive number.

a strategy

Find

\342\200\224
that is,

6.3

Sets

and

Sequences

Involving

handles
large texts, for example, a word
using a program that
a
as
The
takes
text,
represented as a sequence of
input
program
processing
an error
some output. Once in a while, the program encounters
and produces
characters,
what error it is, or
recover.
Not only that, but it cannot even indicate
from which it cannot
is to halt and to output
action the program takes
it is. In other words, the only
where
Assume that the error is local; in other words, it results only from a particular
4'Error.\"
string in the text which the program, for some unknown reason, does not like. The error is
a strategy
of the context in which
the offending string appears. Suggest
to
independent
locate the source of the error.

that

Suppose

are

you

program.

6.4
6.5

Construct

an

example

searching

in a

table of

6.7

6.11

this

6.12

for

if

to the skyline
to
use one solution
possible

invariant

loop

of the

Construct

an

chosen by

taking

cases,

and

for

the main

statement

\"if

Left

* Right.\"

in Chapter 5. Try to formalize

as a \"black box\" to solve the

problem
almost

loop in

Mergesort
(Fig. 6.7)
will not change if we

(Fig. 6.9),

Partition

Algorithm

and

quicksort

of the

first,

use

will

last, and

Q(n2)

comparisons

middle elements of the

the

when

pivot

is

sequence.

almost
is already
sorted, which means that
elements is small. Describehow the different sorting algorithms
6.4 perform for almost sorted inputs. Which algorithm would you use?

for a sorting algorithm

input

in Section

suggested

The

the median
the

search)

binary

of out-of-order

the number

Design

and

algorithm.

for which

example

the if

with

the solution

the correctness

6.14

number

it be

with
Will

the appropriate

Construct

search

linear

the input in Fig. 6.8 (by mergesort), and in

of comparisons for the same input for

sort

to

and start the algorithm

statement,

prove

6.13

(with

sort.

Write

(You are

sort

insertion

used
Compute the

selection

Comparemergesort

In some

for

programs

range

a loop invariant,
that the first if statement in algorithm
In other words, prove
that the result of the algorithm

the similarities.
other problem?

6.11

is in the

digit

(by quicksort).

Prove, by using
is not necessary.
remove

6.10

Each

of comparisons

number

the

Count

insertion sort and

6.9

comparisons

Q(n)

for radix-exchange

program

digits.

Write the complete

selection sort.

Fig.

6.8

use

will

sort. The input is a sequence of n integers,

to m. You can assume that O(m)
space is
the amount of extra
available. First, write the program as a recursive procedure. Determine
a nonrecursive program and try to
by the recursive procedure. Then,
design
space required
minimize the amount of extra space.
the complete

Write

each with k

6.6

which interpolation search

for

size n.

encouraged to
a table

design

similar to that

a divide-and-conquer
algorithm

should

your own.)
in Fig.

algorithm to

use at most

reason this algorithm requires

6.15 for building

3n/2

less than

find

the

a heap

minimal

comparisons

the straightforward

(for

top down.

and
In

elements

maximal

n=2k).

Can

- 3 comparisons

you

in a set.

pinpoint
algorithm?

the

6.15

the Huffman
tree for the set of characters in this question.
many bits are saved in the storage of this question using
storage based on a fixed-length encoding?

Include

Build

How

6.16

Construct

6.17

Construct

the matrices

baacbabaccaba

6.18

using algorithm

the appropriate

loop

invariant

the correctness

of the

first

Unless

are

that

times are worst

Given

an

Prove

6.22

not

is a

problem

algorithm

that

your

sets are assumed to be

are said to run in linear

the

of size

n,

if they

time

consist

to

and

run

time

in

of

O (n).

6.19 is optimal
you developed for Exercise
until you can prove that it is optimal).

algorithm

algorithm

n real

on

time algorithm to find

the number of steps

a number

required to

that

numbers, and a real number x.

to determine

algorithm

in time O

should

the set

whether

there

run in time O (n

S is given

in a

are two

elements of S

sorted order.

Design

an algorithm

to determine whether
two sets are disjoint. State
of the sizes m and n of the given sets. Make
is substantially smaller than n.

algorithm

case where

in

sum

is

to solve

this

(n).

two sets S, and S2, and a real number jc, find whether
there exists
and an element
from S2 whose sum
is exactly
x. The algorithm
O (n log n\\ where n is the total number elements in both sets.
an

whose

log n).

Given

your algorithm

6.25

and

for all /, \\<i<n,

we
have
that,
integers j4[1..h], such
A [ 1 ] = x and A [n ] =y, such that x < v. Design an efficient
search
such that A \\j ] = z for a given value
is the maximal
z, x <z < v. What
to z that your algorithm
makes?

set S containing

an

Design

Design

(Fig. 6.28),

Majority

Algorithm

set S with n real numbers. Design an 0(n)

set. Prove that Q(n) is a lower bound

b. Supposehow

6.24

in

in the

exactly x. The

6.23

loop

this problem.

The input

a.

for the first

phase of the algorithm.

1. Let

decision trees
case (or improve

is a

input

that is
solve

aabccbbaabca to

by comparing the sequences

Distance of Fig. 6.27.

case.

using

by

the worst

The

aabbaabababbaabbaabb.

of

array

|A [i ] -A [i + 1 ] | <
algorithm to find j
number of comparisons

6.21

versus a

Edit

Minimum

otherwise,
sequences and
real numbers. Algorithms

specified

in

characters.

all

trees

Huffman

Exercises

the running

6.20

the string

M obtained

and

prove

elements

6.19

Write

Creative

All

table (Section 6.7) for

the next

177

Exercises

Creative

terms

element

an

should

the
sure

run

from S,

in time

complexity

of

to consider

the

the union of two given sets, both

of size O (n). The sets are
Design an algorithm to compute
should
be an array
as arrays of elements. The output
of distinct elements that form
given
more
the union of the sets. No element should appear
than once. The worst-case running
time of the algorithm should be O (n log n).

178

6.26

Involving Sequences

Algorithms

The

to

compute

the

partition

of its

sum

the

find the

should

of

is a sequence

input

algorithm

real

and Sets

numbers

partition

the

minimizes

the following

in

pairs

numbers. Denote by
that

is even.

such
that n
...\302\273\342\226\240*\342\200\236\302\273

Jt,,*2,

into nil

input

s2,

5,,

n^

tnese
\342\200\224^n/2

an

Design

each

For

way.

sums. The

we

pair,
algorithm

sum.

maximum

*6.27

Modify lexicographic sort to work for variable-length strings. In other words, you can no
k digits.
Some numbers may be long and some
that all numbers have exactly
assume
longer
of
all
numbers
is
course
to
It
short.
by adding
\"dummy\" (0) digits to make
\"pad\"
possible
Find
an algorithm that avoids
all of the same length.
them
doing that and achieves a
in
of
linear
the
total
number
time
digits.
running

6.28

The input

..., xn of integers in an arbitrary

order, and another sequence
integers from 1 to n (namely ax,a2,...,a\342\200\236 is a permutation of
an 0(n logn) algorithm
Both sequences
are given as arrays.
to order
Design
order
the
In
other
to
the
for
the first sequence
words,
imposed by
permutation.
according
in ar For example, if x - 17, 5, 1,9, and a =
each z, jc, should
appear in the position given
should be in-place, so
3, 2, 4, 1,then the outcome should be x = 9, 5, 17,1. The algorithm
use an additional array.
cannot
you

is a sequencex,, jc2,
of distinct
tf\342\200\236

a,,02,...,
1, 2,..., n).

6.29

of n

total

sorted

6.30

is d sequencesof elements
elements. Design an

The input

Johnson

sequences

is

into one

1. Arbitrarily form

This

nil

description of a sorting

incomplete

distinct

known

algorithm

as the

Ford

of elements

pairs

in each pair

elements

the nil larger elements

order the nil remaining
elements

sort

in some

Insert

and

sorting.

2. Comparethe
3. Recursively

4.

logd)

and there

sorted,

sequence.

The following is a brief

and

sequence is already
algorithm to merge all the

that each

such
0(n

into the

sorted list

of larger

elements

almost
that the
any other algorithm, provided
order. Consider the cases of n = 5, 6, and 8. Find
a good order in which
to insert in step 4. You should end up with an optimal sorting
values of n (in fact, you will
(in terms of the number of comparisons) for these
algorithm
get an optimal algorithm for any n < 12 with this algorithm).

6-31

The

uses

algorithm

insertions

is a

input

distinct

integers

a. Design

comparisons

b.

6.32

is the

Why

n -1.

6.33 The sum

directly

n) is at

than

a \"good\"

sorting
algorithm
in the worst case.

to

such

sort

lower bound of Q(n logn) not

(A
representation using
most

comparisons

done in

with many duplications,

sequence of n integers
in the sequence is 0(\\ogn).

the sum

that

Prove

fewer

step 4 are

in

of

balanced

the

heights
binary

an array

of all

n.)

satisfied

nodes

tree with

of size

sequences

Show

in

in this

a balanced

nodes
a tree

is one

using

such

that

at

most

the number

O (n log

of

log n)

case?
binary tree
that

corresponds

whose sum of heights

with

n nodes

is

at

to an implicit
is exactly n - 1.

the heights of all nodes

in a heap (see Section 6.4.5) can also be computed
of
the
the
that
node corresponding
to position / in the array (of size
height
noting
by
- / + 1)1\342\200\242
Find the sum of
most [ log2(\302\253
heights
by using this method.
of

Creative Exercises

6.34

6.35

179

input is a heap of size n (in which the largest element is on top), given as an array, and a
real number x. Design
an algorithm
to determine whether
the fan largest element in the
is
less
than
or
to
x.
The
worst-case
time
of your algorithm should be
heap
equal
running
of the size of the heap.
You can use 0(k) space.(Notice
that you do
0(k\\ independent
not have to find the fan largest element; you need only determine
its relationship to x.)

The

The

The input is a sequence of distinct

problem is the following.
such that each number jc, has a positive weight
with
w(x() associated
the sum of all weights.
The problem is to find, given a value X, 0 < X < W, the
selection

weighted

numbers

jc,,

it. Let W be
number Xj such

that

<

2>(jcf-)

..., xn

jc2,

X,

Xj > Xt

and

>

5>(jc,)

w(xj)+

X.

Xj > Xt

6.36

Let

are 1, this

A be an

Prove that
fan

algorithm
A collects

that

finds

enough

the

fan

information

selection problem. (Notice

weighted
the regular selection problem.)

largest

that

when

of n elements

of comparisons.
by a sequence
which elements are greater
than the
can partition
it. (In other words, you
the set
more comparisons.)

to determine

elements are less than

largest element without

the fan

Considerthe

becomes

problem

and which

largest

around

6.37

to solve the

an efficient algorithm

Design

all weights

making

fan
element, and suppose that we are interested
largest
is a sequence of
fills
Each
element
one
memory cell. The input
only
space.
elements, given one at a time, inserted into a fixed cell C. That is, in the ith input step x-, is
put into C (and C's previous content is erased). You can perform any computation between
of C to a temporary
the content
two input steps (including, of course, moving
location).
Give an
The purpose is to minimize
the extra number of cells required
by the algorithm.
find
of
cells
needed
to
the fan
and
a
lower
on
the
number
bound
bound
memory
upper
element.
largest

of finding the

problem

in minimizing

6.38 The goal

problem is to find the fan smallest element, as in Exercise 6.37, but this time
not
as well as to use very little space (although
the running
time
minimal space). The input is again a sequence of elements xx, x2,..., x,n given
necessarily
element
an O (n) expected time
to compute the fan smallest
one at a time. Design
algorithm
ahead
of
time
that
sufficient
cells.
The
value
of
k
is
known
0(k)
(so
memory
using only
until the last element
can be allocated), but the value
of n is not known
amount of memory
we

want

of this
to

minimize

is seen.

6.39

resides in computer P and B in Q.

and
can perform
by sending
they
any kind of local
messages,
an algorithm to find the nth smallest element of the union of A and B
computation.
Design
You can assume, for simplicity,
that all the elements are distinct.
Your
(i.e., the median).
the number of messages, where a messagecan contain
one element or
goal is to minimize
one integer.
What is the number of messagesin the worst case?

Let

A and

B be two

P and Q can

6.40

Given

a set of

sets,

both with

elements,

such that A

communicate

integers S =

find a nonempty subset /?c5,

x\342\200\236),
[xx,x2,...\302\273

such

that

180

Jt,
\302\243
x,&R

6.41

Involving Sequences

Algorithms

= 0

and Sets

(modulo n).

Use the idea of the

comparisons for the

or determining
jt,,Jt2, ...,-*\342\200\236,
6.2 and 6.4.6.)

to prove a

bound

information-theoretic

problem

of finding the value

that no such / exists.

of

such

lower

of Q(logrt)

bound

that x, = i

in

the

(This problem is discussed

sequence
Sections

in

but that you do not use a programming

use Huffman's
encoding
access bits. You can read the sequence of bits as a sequence of bytes
on the machine). Each byte (block) corresponds to
(or any other blocks of size k depending
less than 28).
thus
to a sequence of integers
the
and
an integer,
(each
corresponds
encoding
of
such
that
can
use
the
to
translate
the
Huffman
method
a
integers
you
sequence
Design
tree and decode the corresponding
sequence of bits. Do it by building a table of size k x 2*,
on the tree
where k is the size of the block (8 in the case of bytes). The table depends
and subtraction of
can use only multiplication,
is given to you). You
addition,
(which
The table should allow
you to access any bit in a
you cannot use bit operations.
integers;
Now solve the problem again,
but this time
number z taken from the sequence of integers.
use a table of size 2x2*.

6.42 Supposethat
language

6.43

you

that

want to

lets you

tree has
that a Huffman's encoding has been applied to a certain text. The Huffman
been constructed and it is available to you. The frequencies of all characters in the text are
now that the text has been changed
also known.
Assume
such that the frequency of
slightly
X has been increased by 1. You want to update
one (existing)
character
the tree so that
it
remains optimal for the modified
text. A friend
makes
the following suggestion for an
the tree.
algorithm to modify

Assume

First, he notes

is that the frequencies

tree
property of a Huffman
are nondecreasing as the nodes are closer to the root. (In other
in the tree than
a node with higher
words, a node with lower frequency cannot be higher
The
of
an
internal
v
is
defined
as
of
all the frequencies
node
the
sum
frequency.)
frequency
of the characters associatedwith external
nodes that are descendants
of v. Consequently, he
whether
the increased frequency still satisfies
that property by checking
suggests checking
the next higher level. If there is no node in the next higher level with a frequency
smaller
than
the frequency
of X, then leave X in its place. Otherwise,
X with the character at
replace
the higher
level whose frequency
is now smaller than
of X. This algorithm
that
may
sometimes
work, but it is generally incorrect. Describe why it is incorrect and how it can be
corrected. You should mention
in the algorithm but, more
not only what is missing
discuss
as is, in general.
That is, either
important,
why the algorithm does not work,
construct
a counterexample
an optimal
under which
does not construct
tree,
this algorithm
or show that, had the algorithm been correct, it would have led to a contradiction
(or to
some highly
It is not enough to point out that the algorithm does
suspicious
implications).
not deal with some cases. It could be that those cases can be ignored.
You need to show
that
the algorithm
is definitely
wrong.

associatedwith

that

the

an

important

nodes

is two strings of characters

A =ax a2
and B=blb2
input
bn.
Design an
a\342\200\236
whether
B is a cyclic shift of A. In other words, the
O(n) time algorithm to determine
should
determine whether there exists
1 < k < n
such that
an
index
algorithm
\302\243,
=
<i<n.
1
for
all
/,
fli
fcot+owm

6.44

The

6.45

The KMP

\342\200\242
\342\200\242
\342\200\242

\342\200\242
\342\200\242
\342\200\242

string

matching

algorithm

can be

improved

for

binary

strings

in

the

following

Creative

181

Exercises

the next table, in addition

to looking at the suffix
of the string seen
constructing
can add the mismatched character.
That
of
is, we look for the longest suffix
a prefix of B. (fr, is the complement character of b(.) That
that matches
B(i\\)bj
way,
in B exactly once.
every character in A is compared to a character
When

way:

so far,

we

a. Give a precisedefinition
b.

6.46

the string

Modify

on-line

KMP

the

Modify

Let T and
Design an
all

y,

Design

of this

advantage

change.

string

words,

you

match (but

the largest

to find the

algorithm

matching

it

do
has

to

not need
to start

with

largest
all

match

of B

inside A;

for Exercise

algorithm

6.48

such

that,

if there

are

subsequences

many

have

and

to

f\\

-pr

\302\243c(/y)
y=i

that

.A

1 \302\243./\302\243*,
we

is maximized.

\302\243/,

Find the

sequence of

in T

is maximum.

then

Consider Exercise6.48;assume

and

you

two sequences f,, t2,..., t\342\200\236

and P\\,p2,
such that k<n.
...,p* of characters,
to determine whether
P is a subsequence
of T. (P is a
algorithm
\342\200\242
\342\200\242
\342\200\242
< i* \302\243fl
of T if there exist a sequence of indices
1 </1
such that for
<i2 <
< k, we have t(
=pr)

That is, find

it.

instead,

b x).

O(n)

an

tj

that matches a

prefix

of B

whose sum of indices

the algorithm finds the subsequence
\342\200\242
\342\200\242
of indices 1 < i, < i2 < \342\200\242
sucn tnat f\302\260r
all
the sequence
<ik^n

with

indices

</,

that

subsequence

matching

< i2 <

the ith

character of T has

that maximizes the

''' <ik^n sucn

that

cost c (/) associated

That
is, find the
have
we
f, =py,
j, \\<j<k,

a positive

sum of costs.

f\302\260r
all

is maximized.

common subsequence(LCS)of two sequences T and P is the largest sequence

L is a subsequence of both
T and P. The smallest common
supersequence
that both T and P are
(SCS) of two sequences T and P is the smallest sequenceL such
subsequences of L.

The largest
L such that

a. Design
b.

Let

efficient

d(T,

allowed

d(T,

algorithms

to

find

the LCS

and SCS of

two

given

the smallest edit distance

between T and P such
other
we have to insert
and
words,
| SCS(7\\ P)\\-\\ LCS(T,P) |, where
| SCS(7\\ P)
SCS (LCS) of T and P.
smallest

P)be

P)=

sequences.
that

no replacements

Prove
LCS(7\\ P) \\)

delete).

(in

size of the
6.52

the

P be

<j

are equal

6.51

matching algorithm to take

In other

of A.

subsequence

6.50

for

string

want to find

6.49

values

its new

show

and

a relatively

at

substring

6.48

table,

matching algorithm: Suppose that the pattern is input one character at a

slow pace (e.g., by typing), but the text is already
We would like
given.
to proceedwith the matching as much as we can, without waiting until all the pattern is
In other words, when
the fan character is input,
we would like to be at the first
known.
in
k
-1
in
the
text
that
matches
the
first
characters
the pattern.
place
Modify the KMP
to
achieve
that
goal.
algorithm

An

time

6.47

the modified next

of

in Fig. 6.21.

example

\\

(|

are
that

is

the

in Section 6.8 to the case where

Generalize the minimal-edit-distance problem
presented
are not counted. In other
insertions at the beginning or the end of one of the sequences
B; we count
words, if B fits inside A, then we do not count the insertions needed to enlarge

182

edit distance

the

only

insert at the

of B to

the

of A

subsequence

without cost,

of B

beginning

and Sets

Sequences

Involving

Algorithms

you

to

it fits.

which

(Notice

the insertions at

count

must

that,

the end of

if

you

A, and

vice versa.)

6.53

to three (or more) sequencesin the

comparison
problem can be generalized
sequence
from any
to insert, delete, or replacecharacters
following way. In each step, we are allowed
in all sequences are
of the sequences. The costof a step is 0 if the corresponding characters
and only one insertion or deletion
and 1 otherwise
is
(even if two sequences match
equal,

The

For example, suppose that the sequences are aabb, bbb, and ebb. One possible
a b in bbb and
is inserting a in front of bbb and ebb (which costs 1),replacing
an 0(n3)
an a, and then
the rest matches; the total cost is 2. Design
a e in ebb with
three given sequences.
algorithm to find the minimal edit distance between
necessary).

edit

6.54

sequence

Let A

= a,

'''

a2

string 0,-0/+i
between B and
all i,
6.55

A [i].

The

input

that

selects

is a

Design

sequence of

6.58

an

be two

bm

of

suffix

Denote
strings of characters.
by A [i ] the
Let d( be the minimal
edit distance
to find the minimum value of d, (among

A).

algorithm

0(n2)

numbers

a number from the

least

\\j/2 n\\

is in

terms

deterministic algorithm
to
greater than or equal

that any

Prove
...,*\342\200\236.

jc,,jc2,

set which

the upper half

(i.e.,

number
of steps required
of both k and r.

by

the

coloring algorithm of

probabilistic

that you have a procedure for generating

random numbers in the range 1
a random permutation of n numbers.
every k < n. Design an algorithm to generate
possible permutation should be selectedwith equal probability.

Assume

What

is the

6.59 The results

percent?

Discuss
not precise. What

percent

margin

of error.

absolute

bounds

is

6.60 The purpose

Each

examples

of public-opinion
with an \"error\"
polls are usually
given
indicate that candidate X has x percent of the vote, and add

may

they

probability

to k, for

of probabilistic algorithms. Suppose that there are two

A common algorithm
is to ask k random voters
and take the
favor each of the candidates.
that exactly one-half of the voters
that the results of the survey
(with k voters) are in the range of 45
be an expression with
(The result should
n, k, and the percentages as

Public-opinion polls are

candidates and n voters.
average response. Assume
percent to 55
parameters.)

the

comparisons.

Determine the expected

Section 6.9.2,in

6.57

the zth

<i<n).

median) must make at

6.56

'''

= b, b2

an and B
\342\200\242
\342\200\242
\342\200\242
a\342\200\236
(namely,

why

stating

would

exercise is to compare
Monte Carlo algorithms

the bounds

be the

on

precise way

the

range. For example,

the

that

percentage

to define

poll has a
of error

\302\2613

as

the error?

Carlo algorithms to Las Vegas algorithms.

the running time, but cannot
guarantee
but cannot
correctness; Las Vegas
on the other hand,
correctness,
guarantee
algorithms,
so
the running time. Suppose that the problem we consider is a decision problem,
guarantee
the answer is either yes or no. Assume
in the Monte Carlo
that the error probability
is at most 1/4. (This is enough
times
run the algorithm many
since we can simply
algorithm
and take the majority
as the answer, thereby reducing the probability
of error significantly.)
Which
to convert one
In other words, is it possible
type of algorithm is more powerful?
of algorithm
to the other?
type
In

6.61

a nutshell,

Design

of this

an algorithm

that, given a

list

Monte

guarantee

of n

elements,

finds

all the

elements

that

appear

more

Exercises

Creative

than

nlA

times

in the list. The

should

algorithm

use O(n)

comparisons.

(Hint:

183

Modify

the

majority algorithm.)

6.62

6.63

players.

to design a schedule
Each player must play every
n - 1
Denote the

Design

an algorithm

tennis tournament. There are n =2k

each player must play one match per
for
the schedule for each player.
day
days.
players by P\\ ,/>2,..,/>\342\200\236.
Output
Use divide and conquer in the following way. First, divide the players into two equal
(Hint:
the
groups and let them play within the groups for the first n/2-\\
days. Then, design
for
other
nil
the
the
between
games
groups
days.)

You

are

number

asked

of players.

to arrange a
If

the

number

for

a round-robin

other player, and

tennis

round-robin

of players

is odd,

tournament
then

in each

(see Exercise 6.62) for any

does not
round one player

participate.

6.64 Let r!, r2,...,rn

n

and

cx,c2,...,

cn be

two sequencesof integers

whose

sum is

equal; namely,

are called realizable if there is an n x n matrix all of whose elements are

sequences
for all /, the sum of the ith row is exactly r, and the sum of the ith
0 or 1, such that,
is exactly c,. Not all sequences are realizable. For example,the two sequences 0,2
column
Such

either

and 0,2 are not realizable

but it cannot be more

sequences are

realizable,

since only the

than

and

second element of

the

second

row can

be nonzero,

two given
whether
Design an algorithm to determine
a matrix with
row and column
construct
the corresponding
1.

the induction hypothesis to extend

the problem to
are. (Hint: First, strengthen
on n (the number of rows). Try to place Is in the first
n xm
matrices.
Then, use induction
row so that the problem for the other n -1 rows can be solved if and only if the original
sums

problem

if they

can be solved.)

CHAPTER

ALGORITHMS

GRAPH

is the

shortcut

longest distance

two points.

between

Anon

7.1

Introduction

of objects.
The
were limited to ordering, multiplicities, overlappings, and so
relationships
on.
In this chapter, we discuss more
involved
relationships
among objects. We use
a large variety of situations,
and
to model
these relationships. Graphs can model
graphs
social
We
to
have
in
fields
from
been
used
diverse
they
archaeology
psychology.
ranging
to
and
certain
to
several
basic
present
manipulate
compute
graphs
important
algorithms
In

the

graph

previous

let's see

Finding
streets
the

examples of modeling

a good

intersections

sets

involving

and sequences

by

graphs.

a city is a graph-theoretical problem. The

edges (directededgesin the case of one-way streets), and
and each edge (street segment) can be
the vertices. Each vertex
to

a restaurant

in

to the
to
an

expected

time delay,

path between two vertices.

Some programs can be partitioned
have several possibilities to

undesirable.The
3.

route

correspond

associatedwith
2.

discussed algorithms

properties.

First
1.

chapter, we
we studied

problem

into

the

states.

is to

problem

which states

the

states

can lead

to

the

find

From each state

proceed. Some of

of finding

is a graph-theoretical problem
edge indicates a possiblemove

and

the

\"quickest\"

program

may

may be considered
an undesirable
state

and an
correspond to the vertices
from
one state to another.
The problem
of scheduling
classes in a university
can be viewed as a graphThe vertices correspond to the classes,
theoretical
and two classes are
problem.
connectedif there is a student who wishes to take them both or they are both taught
is to schedule
the classes such that the
by the same professor. The problem
in

which

the states

Graph

186

Algorithms

conflicts are minimized.

hard

This

is a difficult

problem

and

good

solutions

to

it

are

to find.

with several user accounts. Each user has a security

Consider a computer
system
his
or
her
Users may want to cooperate and to give
to
access
account.
permission
if A has permission to use
to
their
account.
use
another
one
However,
permission
then
A may be able to use
and B has permission to use C's account,
\302\243's
account,
The problem of identifying
which
as well.
users can access which
C's account
is a graph-theoretical
to the vertices
in
accounts
problem. The users correspond
A to user B if A gives
user
and there is a directed edge from
B
this
case,
his
or
her
account.
to
use
permission
are quite a few textbookson graph
and
There
(see the Bibliography
section),
theory
other
numerous
applications.
Representationsof graphs were discussed in Section 4.6. For the most part, we
which is more efficient for sparse graphs
will use the adjacency list representation,
(i.e.,
few
We
with
standard
A
by
relatively
begin
edges).
introducing
terminology.
graphs
consists
of a set V of vertices (also called nodes),and
a
set
E
of
G
(V,
E)
edges.
graph
Each edge correspondsto a pair of distinct vertices. (Sometimesself-loops,which
are
to
are
we
will
from
a
vertex
assume
that
are
not
A
itself,
allowed;
allowed.)
they
edges
in
a directed graph
are ordered
graph can be directed or undirected.The edges
pairs;
the order betweenthe two vertices the edge connects is important.
In this case, we draw

4.

arrow

from
one vertex (the tail) to another
The edges
head).
(the
are unordered pairs; we draw
them
as
line
simply
segments. A
is
a
with
several
between
the
same
of
vertices (i.e.,
multigraph
graph
possibly
edges
pair
\302\243
is a multiset).
that
are
not
are
sometimes
called
Graphs
multigraphs
simplegraphs.
Unless
The
that the graphs we deal with
are simple.
otherwise, we will assume
specified
we
v is the number of edges incident
to v. In a directed graph,
degree d(v) of a vertex
also distinguish
between the indegree,
is the number of edges for which
v is the
which
which is the number
v is the tail.
of edges for which
head, and the outdegree,
A path
from V! to v* is a sequence of vertices v1( v2,..., v* that are connected
by
the edges (v1( v2), (v2, v3),..., (vk_{, vk) (these edges are also usually
to be
considered
A path is called simple if each
vertex
part of the path).
appears in it at most once.
Vertex u is said to be reachable from vertex v if there is a path (directed
or undirected,
on
the
from v to u. A circuit
is a path whose first and last vertices are
depending
graph)
the
same.
A circuit is called simple if, except
for the first and last vertices, no vertex
more than once. A simple
are sometimes
is also called a cycle. (Circuits
circuit
appears
called cycles even if they are not simple; we will assume
that cycles are always simple.)
The undirected form of a directed graph G=(V, E) is the the same graph without
directions on the edges.
A graph is called connected if (in its undirected
form) there is a
from
vertex
vertex.
to
other
its
undirected
form)
A
that
is
a
forest
path
(in
any
any
graph
does not contain a cycle. A tree is a connected
as an
A rooted tree (also known
forest.
is a directedtree with one distinguished vertex called the root, such that
arborescence)
from the root.
all the edges are pointing
away
A
of a graph G = (V, E) is a graph // = (\302\243/,
such that UQV and
subgraph
F)
F \302\243E. A spanning
tree of an undirected
is
G
a
of
G that is a tree and
graph
subgraph
an
in

edge as an
undirected

an

pointing

graph

7.2

Eulerian

187

Graphs

vertices of G. A spanning
forest
of an undirected graph
G is a
forest and that contains
all the vertices of G. A vertex-induced
H = (U, F) such that
of a graph
G = (V, E) is a subgraph
(/cV
and F consists
subgraph
of
whose
the
in
E
both
vertices
to
U.
A
all
is
vertex-induced
of
edges
belong
subgraph
=
If
an
induced
a
called
G
is
not connected, then
(V,
E)
usually
simply
subgraph.
graph
be partitioned in a unique
called the
it can
way into a set of connected
subgraphs
that

all the

contains

subgraph of G that

is a

connected components of

G.

G such

subgraph of
connected subgraph.

other

no

that

a maximal

is

component

vertices

can be

vertices

in

with

associated

divided

other

the

into

set. A

two

sets

component of

G contains
A

it.

connect verticesfrom

with

of

subgraph

words, a connected
is a graph
whose
graph

In other

bipartite

such that all edges

weighted graph is a graph

G is a connected

weights

(or

one

costs, or

set to

lengths)

the edges.
for

definitions

Many

A connected

connected

obvious differences. For

directed
example,

and undirected
directed

paths

graphs

are

similar,

and undirected

paths

except
are

for some
defined

in

of the edges in directed

exactly the same way, but, of course, the directions
paths are
will
one
of
we
not
When
we
discuss
a
use
different
type
graph
specifically
specified.
when
we
talk
in
for
about
the
context
of
we
directed
notation.
So,
paths
example,
graphs
will mean directed paths.
to be the first problem
in graph
We start with a simple example that is considered
\342\200\224
how to traverse a graph,
the bridges of Kbnigsberg. We then discuss
theory
walking
to find shortest paths
in a graph, how to partition
the graph
how
to order a graph,
how
a
and
other
10
into blocks satisfying
certain
includes
properties,
problems. Chapter
matrix
more
on the relationships of graph
and
Several
discussion
algorithms
algorithms.
are presented there.
graph
algorithms

Eulerian

7.2

Graphs

graphs is involved in what is considered to be the first solved

The Swiss mathematician Leonhard Euler encounteredthe
The
town
of Kbnigsberg
in
1736.
(now Kaliningrad) lay on the banks
puzzle
following
and on two islands of the Pregel river, as is shown
in Fig. 7.1. The city
was connected
by
seven bridges. The question
to
solve) was whether
(which
attempted
many townspeople
it was
in town
from anywhere
and return to the starting
possible to start walking
point by
is obtained by abstracting the problem.
crossing all bridges exactly once. The solution
The graph in Fig. 7.2 is equivalent, for the purpose
of Fig.
of the problem, to the layout
7.1. The question
becomes
the graph-theoretical
it is possible
to find
problem of whether
a circuit
in the graph
contains
each edge exactly once. Another
that
to
way
pose the
to draw the graph
is to ask whether it is possible
in Fig. 7.2 \342\200\224
and end at the
question
same place from which we started \342\200\224
without
the pencil. Euler solved this
lifting
is
a
traversal
if
that
and
problem
such
possible
by proving
only if the graph is connected
and
all its vertices
are called
have even degrees. Such graphs
Eulerian
graphs. Since
odd
of
the
vertices
i
t
follows
that
the
in
contains
7.2
degrees,
graph
Kbnigsberg
Fig.
A proof of this
theorem
which
by induction,
bridges problem is impossibleto solve.
is
closed
for
the
next.
an
to
efficient
path,
given
constructing
algorithm
corresponds
The

of Eulerian

notion

problem

of

graph

theory.

188

Graph Algorithms

Figure

7.1 The Kbnigsberg

bridges problem.

Figure

7.2 The

graph

The Problem
that all the
each

It is
When

edge

Since each

be even.

Given

connected

vertices have even degrees,find

appears in P exactly once.

all

a closed

vertices

by

must have

path, we enter

edge is used exactly

To prove

undirected

an

Kbnigsberg bridges problem.

graph G = (V, E) such

closed
path P such that

of E

easy to prove that

traversing

to the

corresponding

induction

once,

and

even degreefor
leave

the number

that the

condition

each

of edges
is

such

vertex the

a closed

same number

adjacent

sufficient,

to

each

path to exist:
of times.

vertex

we first have to

must

decide

7.3 Graph Traversals

189

The first consideration is to be able to reduce

the
remove
a
or
an
vertex
the
changing
edge,
resulting
graph
may
problem
We should remove a set of edges
S such that, for
not satisfy the even-degree
property.
each vertex v in the graph, the number of edges from S adjacent to v is even (possibly 0).
this requirement, so the question
circuit
satisfies
is whether
an Eulerian graph
Any
contains
a circuit. Suppose that
we
start traversing
the graph,
without
always
going
v in an arbitrary order. We
vertex
through any edge more than once, from an arbitrary
that the traversal will
to v because,
whenever
return
we enter another
claim
eventually
we reduce the degree of that
vertex
it
and
therefore
we can
vertex,
1,
odd,
by
making
it.
this
leave
that
circuit
not
all
the
include
(Note
may
always
edges.)
We are now ready to state the induction
and prove the theorem.
hypothesis
which

to

parameter

induction.

the

apply

without

it.

If we

Induction hypothesis:A
vertices

exactly once, and we

(It is

easier to state the induction

of closed paths,
even

resulting
be

must

though

Let

(V, E) with m edges.

from removals of all the
of
even, since the number

cannot simply

G\\,G\\,...,G\\ be the

edges

that

edges, all
includes

that

of

whose

each edge

path.

of

the

of edges

number

rather

than

performed on paths.) Consider

P be a closedpath in G. Let G' be the graph
in G'
of P from G. The degreesof all vertices
the induction is

edges adjacent to

removed

induction

the

apply

with < m

in terms

hypothesis

G =

graph

graph

to find

how

know

number

the
a

connected

degrees, containsa closedpath

have even

yet, since G'

of G'. In each

hypothesis

any

may

vertex
not

is even.

But we
Let

be connected.

the degrees of
component,
components
of edges in each
Furthermore, the number
(indeed, in
component
we
can
now
the
induction
all of them together) is < m.
Hence,
hypothesis to each
apply
a closedpath that
the
each
has
That
induction
is, by
hypothesis,
component
component.
these k closed
includes
every edge exactly once, and we know how to find it. Denote
We now need to merge all these paths to one closed path covering
paths
by P! ,/>2,...,/>athe
we meet the first
with
whole
any vertex in P and traverse P until
graph G. We start
the path P}
vertex vy belonging
this
we
traverse
At
to one of the components
point,
G'y.
the
the
the
of
to
We
can
this
continue
components
paths
returning
way, traversing
vy.
first
vertex.
At this point, all edges will
time
we meet them, until
we return
to the starting
have
been traversed
exactly once. This closedpath is called an Eulerian circuit. The
the
is not yet complete.
We still need to find an efficient method to identify
algorithm
Both of those
connected
method
to traverse the graph.
and an efficient
components,
issues are discussednext. The implementation
is left as
of the Eulerian
circuit
algorithm
all

vertices

an

exercise.

7.3

Traversals

was

This

of

dimensionality

Scanning a
traversal

we encounter

problem

input.

connected

even.

Graph

The first
the

are

graph,

algorithms

the

or

trivial

input

trying

problem

\342\200\224

traversing
\342\200\224

when

depth-first

sequences

it, as we
search

a graph algorithm is how

to look at
the previous chapter because of the oneand sets can be easily scanned
in linear
order.
to design

in

call

it,

is not

(DFS), and

We present two
straightforward.
breadth-first search (BFS).Most

of

Algorithms

Graph

190

the

this chapter

in

algorithms

depend,

one

in

of

on one

or another,

way

these

techniques.

7.3.1

Search

Depth-First

The

for directed
algorithms
we also want to

search

depth-first

identical. However,
different

since

directed

in

and

graphs

undirected

in

undirected

and

graphs

graphs

explore several graph

we divide

graphs,

the

are almost

properties

that

are

discussion

into

two

parts.

Undirected

Graphs

to an art gallery
of
undirected
consisting
graph G = (V, E) corresponds
The
are
of
G
corridors
where
the
to
of
an arrangement
correspond
paintings
hung.
edges
the
intersections
of
the
corridors.
We
the
vertices
to
and
want
the corridors,
correspond
We
assume
that we can see both
and
see all the paintings.
to walk
through the gallery

Suppose

the

that

when we walk
possible to walk throughout

of a corridor

sides

it is

then

assume here that

it in

through

If the

direction.

any

graph is

corridorexactly

visiting each

the gallery

Eulerian,

We

once.

we allow
each edge to be traversed
graph is Eulerian, and
more than once (as it turns out, each edge will be traversed exactly twice). The idea
We walk through the gallery
to enter
behind
depth-first search is the following.
trying
new
corridors
whenever
we can. The first time we visit an intersection, we leave a

do not

pebble there,
from

we

which

return

through

intersection
essential
the

part

new edges
The

(going deeper

adjacency

list

the

see

now

representation.

of the DFS.

the root

vertex

rx, connected to r, is

(recursively). The recursion

v are

adjacent to r are
unmarked

already
marked,

vertex r2

an

root

The
then

stops

the

the

from

this

it is not an
only to clean the gallery;
to explore new corridors;
we return from
intersection,
only if we tried all other

that
search (DFS)to indicate
The
main
reasons
for
the
gallery).
its adaptability
to recursive algorithms.

of DFS

marked.
then

first

in

was

terms

DFS is implemented
The traversal is started

called

connectedto

try

and

graph

how

always
entered

the

into

we

same corridor

corridors leading

depth-first

we gave

description

Let's

pebbles.

first

we

the

is done

pebbles

We

approach

it divides

way

we

the

a deadend). When

through

the pebble from this intersection,

entered.
We will not visit

remove

we

which

from

corridor

the

this

call

We

the

another

try

it is

(unless

a pebble, we return
corridor.
If all the

has

been visited, then

which

from

corridors.
DFS is

and

corridor

another

already

again. (Removing
of the algorithm.)

corridor

visit

came,

that

have already

intersection

from

we continue

and

at an intersection

arrive

and

the

is marked

picked

and

for undirected
from

starting

An

r{

to

of

down

putting

vertex r,

arbitrary
from

try

usefulness

in the

given

graphs

an arbitrary

as visited.
a DFS

and

of walking

we first

which

is

(unmarked)

is performed
all the vertices

it reaches a vertex v such that

all the vertices
If, after the DFS for r { terminates,
DFS for r terminates. Otherwise,another
arbitrary

when

connected to r is picked,a DFSstarting

from

r2 is

performed,

and

so

on.

There
applications
postWORK,

is generally a purpose for traversing

the graph.
To incorporate different
the DFS framework, we associatetwo types
of work, preWORK and
with visiting a vertex or an edge;
is performed at the time the
preWORK
with

7.3 Graph

Traversals

191

and postWORK is performed after

we backtrack
from an edge or find
Both preWORK
leads to a marked
vertex.
and postWORK depend on the
of DFS. This notation
allows
us to present several applicationsby defining
application
and
The
DFS
is given
in Fig. 7.3. The starting
postWORK.
program
only preWORK
call
is
v.
For
of
the
we
first
recursive
that
assume
the
is
vertex
simplicity,
graph
is given in Fig. 7.4, where the numbers associated with
the
connected. An example
indicate the order in which the vertices could be traversedby DFS.
vertices
vertex

that

is marked,

the edge

(G, v);

Depth_First_Search

Algorithm

Input: G = (V, E) (an undirected connected graph),

Output: dependson the application.

and

v (a

vertex of

G).

begin

mark

perform

for all

preWORK
edges

(v,

ifw is unmarked

onv ;

{preWORK

on the

depends

application of DFS}

w) do
then

Depth

First

Search(Gf w) ;

perform postWORK
for (v, w)
{ postWORK dependson the application
performed only on edges leadingto newly

of DFS;
marked

is sometimes

it

vertices.}

end

Figure 7.4

\342\226\241
Lemma

Algorithm

First_Search.

Depth

7.1

then
all its vertices
// G is connected,
Depth First Search, and all its edges will
the execution
of the algorithm.

Figure 7.4

DFS

for an

will

be

be

looked

marked

at at

by

algorithm

least once during

undirected graph.

Graph Algorithms

192

Proof:

remaining
must

of

the

is visited,
whenever a vertex
all
vertices
Since
marked) too.
edges areconsidered,all edges
If

one marked
all the

vertices

to

adjacent

vertices

unmarked

vertex from U

one

it

visited

are

a vertex

(hence
all its

is visited

are considered.
not

\342\226\

we

connected,

have to modify
is connected
graph

first try, then the

after
the
with an arbitrary unmarked vertex, perform
whether
or not a graph
can use DFSto determine

Thus, we

of

least

situation cannot happen, since

start

we

Otherwise,

at

and since whenever

are marked

vertices

connected,
But this

vertex.

unmarked

are visited,

G=(V, E) is

a graph

G is

Since

the set

U denote

let

and

algorithm.

to at least

connected

be

the contrary,

Suppose

at the end

DFS

and we are

another
is connected

all

If

slightly.

done.
on.

and so

DFS,

and to

its

find

is given
in Fig 7.5. We will
The corresponding algorithm
components.
we
because
otherwise
can
deal with
connected
consider
usually
only
graphs,
generally
in Fig.
we
will
use
DFS
it
is
as
described
connected
each
component
separately. Thus,
it
run
several
times
as
in
that
have
to
be
7.3, without
may
mentioning
Fig.
specifically

connected

7.5.

Algorithm

Input: G
Output:
for

( G );
ConnectedjComponents
E) (an undirected graph).
is set to the number
v.Component
=(V,

every

of the

component

containing

v,

vertex v.

begin

Component

:=

Number

while there is an

1 ;

Depth First_Search{G,

( using

the following

v do

vertex

unmarked

v);

preWORK:

v.Component .= Component
Number

Component

Number

:- Component

;)
+ 7

Number

end

Figure 7.5

It is

Complexity

each end). Therefore,

since the

graph

them

must

time.

Therefore,

be

Constructing
Next,

we

present

and building a
DFS tree exhibit

easy to see that

the

may

running

contain

examined),

Connected

Algorithm

edge

is looked

proportional

to the

each

time is

Components.

twice

exactly

number

(once

of edges.

from

However,

all of
(and
many vertices that are not connected to anything
for the running
we must include 0(\\V |) in the expression

the overall

time

running

is O

(|

+ |E

the DFS Tree

two simple uses of DFS\342\200\224
numbering

special spanning
special

at

properties

tree,

called

that are

the DFS
useful

for

|).

the vertices

with

DFS

numbers,

tree. The DFSnumbers

many

algorithms.

Even

if

and

the

the

tree

Graph Traversals

7.3

193

it is easier to understand
many
explicitly,
algorithms
by considering it. To
these algorithms, we need only to describe
either preWORK or postWORK.
The
the
vertices
is
in
for
with
DFS
numbers
and
the
7.6,
numbering
given
Fig.
algorithm
DFS
tree
is
in
for
the
7.7.
These
two
need
not
given
building
Fig.
algorithms
algorithm

is not

built

describe

be performed separately.

(G, v);

DFS_Numbering

Algorithm

G = (V,

Input:

Output:

for

vertex

every

UseDFSwith

graph), and
v.DFS is set to the

v,

:= 1

Number

DFS

Initially

(an undirected

E)

the following

vertex of

v (a
DFS

number

G).
of v.

;
preWORK:

preWORK:

v.DFS := DFSNumber;
+ 1

DFSNumber

:=

DFSNumber

Figure 7.6 Algorithm

DFS

Algorithm Build_DFS_Tree (G, v) ;

G = (V, E) (an undirected graph),
Input:

Output: T
Use DFS

(a DFS

tree of

G; T is initially

the following

with

postWORK:
ifw was unmarked

{the

Depth

algorithm

A vertex
path

from

w to

in

T.

(line 4)

of

Search }

First

DFS

Build

Algorithm

If v is an

7.2 (The main

\342\226\241
Lemma

Let G = (V,

vertex of G).

the edge (v, w) to T ;

be included in the if statement

can

an ancestor

is called

v (a

empty).

add

then

Figure 7.7

unique

and

Numbering.

postWORK:

above

statement

of a vertex

w in

ancestor of

u\\

property

of

a tree

then

undirected

Tree.

T with

w is

root

r, if

is on

called a descendant of

the
v.

DFS trees)

undirected graph, and let 7=(V, F) be a DFS

eeE
Build
DFS Tree. Every
edge
either belongs to T (i.e., eeF), or connects
two vertices
one
which
G,
of
of
is the ancestor
of the other in T.
tree

of

E)be

a connected

constructed

by

algorithm

that v is visited by DFS before

u.
u) be an edge of G, and
suppose
from
all neighbors
of v that have not been
we perform DFS starting
marked yet. Since u is a neighbor of v, the DFS will either start from w, in which case
from v, in which case
(v, u) will belong to 7, or the DFS will visit u before it backtracks
u is a descendant
\342\226\2
of v in T.

Proof: Let

After

is marked,

(v,

Algorithms

Graph

194

In other words,

sideways across the

DFS avoids cross edges,which

tree.

cross

Avoiding

edges is

are

edges
for

important

vertices

connecting
recursive

procedures

performed on the graph, as we will see later.

we also include its nonrecursive
version.
Since DFS is a very
important
program,
is
a
which
a
recursive
for
stack,
The main tool
keeps information
program
implementing
all the local data
maintains
the recursive calls. A compiler
to \"unfold\"
needed
associatedwith every instance of the recursive procedureon the stack. Hence, when one
we can get back to the exact
in
ends,
recursiveinstance
point (with the exact information)
of
the
same
another
instance
recursive
be
the calling procedure (which
procedure).
may
on the stack, which is one reason
Frequently, not all local data need to be maintained
is more efficient. The nonrecursive
version
we give
nonrecursive
procedures
using
why
from
a
recursive
to
a
of
a
translation
nonrecursive
is a good example
next
program.
a recursive
version into a nonrecursive
we face in translating
One major difficulty
inside
a for
version is that we need explicit bookkeeping. We called DFSrecursively
the
in
the
the
to
remember
from
which
to
and
right
place
loop
program
expected
loop,
maintain
continue after the end of the recursive call. In a nonrecursive
version, we must
list of its incident
this information
explicitly. We assumethat each vertex v has a linked
will
The
list
is
order
follow
this
to by v.First.
in
a
certain
order).
(DFS
pointed
edges
two variables:
Vertex and Next. Vertex
is the
Each item in the list is a recordcontaining
and Next points to the next item. Next of
name of the vertex
on the other side of the edge,
the
last edge on the list points
to nil. DFS proceeds as before,
down the tree
traversing
until
A stack
no new vertices
is maintained throughout
are found.
the search.
The stack
contains all the vertices
on the path from the root to the current vertex (in the order
of the
Between
two
vertices
Parent
and
the
stack
contains
a
Child,
path).
every
pointer to the
from
is
Parent
that
the
next
one
DFS
traverses
when
it
backtracks
from
Child. The
edge
nonrecursive version of DFSis given in Fig. 7.8.

Directed
The

Graphs
DFS for

for

procedure

However,directed

directed

graphs

is

to that for undirected

identical

have different

graphs.

longer true that there are

properties.
no cross edges, as can be seen in Fig. 7.9. There are now four types of edges \342\200\224
tree
back
forward
The
first
three
and
cross
of
edges,
edges,
edges,
edges.
types
edges
connect
two
vertices
one of which
is a descendant
of the other in the tree: Tree edges
trees

DFS

to children

back edges

connect

parents

forward

edges connect ancestorsto descendants.

in the tree. Cross edges, however,

\"related\"

shown

in

the

next

in

the

It

tree,

is no

and
connect descendantsto ancestors,
cross edges connect verticesnot

Only
must

cross

from

\"right

\342\226\241
Lemma

7.3 (The main

property

of directed

as

left,\"

DFS trees)

Let G = (V, E) be a directed graph, and let T=(V, F) be a DFStree

< w.DFSNumber,
(v, w) is an edge in E such that v.DFS _N umber
is a

to

lemma.

descendant

ofv

in

the

tree

T.

ofG.

then

If
w

is

Graph Traversals

7.3

G =(V, E) (an

Input:
We

we

the

do that

not

and

graph),

of G).

(a vertex

application.

Pascal

use the

will

connected

undirected

Output: dependson
(

(G, v) ;

Nonrecursive_Depth_First_Search

Algorithm

this chapter.

rest of

the

here;

explicitly

symbolA

pointer
in

begin

while

there is an unmarked vertex

mark

do

v ;

perform preWORK on
Edge := v.First ;
v and Edge to the
push

Parent := v

of the

top

up to here;
stack is not empty
remove Edge from the

{ initialization

Edge

comes

now

while the
while

stack ;

of the

top

recursion

stack ;

is unmarked

mark

loop of the

do

* nil

Child := Edge*.Vertex
if Child

the main

do

then

Child;

preWORK on Child ;
to the top of the
push
Edge*.Next
{ so that we can return to the next
perform

with

Parent := Child
to

Parent

push

else {Edge

Edge

if

the

{the

is a

:=

stack

are done

the

top

of the

stack ;

back edge

Child)

postWORK

only on

Edge*.Next;

Child from

stack

when we

}
perform postWORKfor(Parent,
{this step is skippedif we perform
tree edges}

remove

edge

Child}

:= Child.First

Edge

stack

is not

the

top

empty
empty

becomes

let Edge and Parent

{ do not remove

of the

stack ;

when

Child

then

the top
them }
be at

perform postWORKfor

is the

of the

root}

stack

(Parent, Child)

end

Figure

7.8 Algorithm

Nonrecursive_Depth_First_Search.

196 Graph Algorithms

Figure

Since the

Proof:

(v, w) is

Since

time

was

v.

DFS number
in

edge

Hence,

\302\243,
(v,

(v, w)

unmarked,

is satisfied.

condition
from

an

A DFS

7.9

would

be

greater than that

be considered during
added
to the tree;

marked after v during

of v in the tree T.

w was

be a descendant

w must

graph.

w is

of

w) must

Otherwise,

tree for a directed

of v, w
the

was visited

DFS

hence, (v,

after

v.

of v. If at that
w)e

the recursive

F,

and the

call of DFS

\342\226\2

undirected graphs, starting

from
any vertex, traverses the whole
directed
Consider
in Fig. 7.10. If
the
directed
graph.
graphs.
graph
DFS starts at a, for example, then only the left column will be traversed.
DFS will
traversed
the whole graph of Fig. 7.10 only
if it starts at v. If v and its two incident edges
are deletedfrom the graph, then there is no vertex from which a DFS traverses the whole
We must start
from an unmarked vertex, and
continue
so until
all
graph.
again
doing
vertices
are marked. Therefore, whenever we talk about
DFS for directed graphs, we
assume that it is run until all the vertices are marked and all the edges are considered.
DFS

for connected

This is

not

so

for

Figure

7.10

#4

+*

<L

<L

*<

#<

An example of a directed

DFS

that does

not traverse the

whole

graph.

Graph Traversals

7.3

As

an

we show

example,

DFS to

use

to

how

determine

graph is

not a

or

whether

197

acyclic.

The Problem
whether
it

a (directed)

contains

graph G

a directed

Given

=(V,

determine
\302\243),

cycle.

7.4

\342\226\241
Lemma

Let G =(V, E) be a directedgraph,

and
let T be a DFS tree
Then,
ofG.
contains a directed cycle if and only if G contains a back edge(relative

G
to

back edge,then it leads to a vertex higher up in the tree, so it

Conversely, let C be a cycle in G and let v be the vertex in C with the
lowest DFS number.
We claim
that the edge (u\\ v) leading to v in C is a back edge. It
cannot
be a forward or a tree edge,
since it leads from a higher
DFS-numbered
vertex to
v
is
not
a lower DFS-numberedvertex.
that
an
ancestor
of
w
in
the
tree, and let
Suppose
u be the lowest
common ancestor of v and w. Since v has a lower DFS number than that
of u\\ it is in a subtree of u that was visited before the subtree
of u that contains w. This
implies that the only way to reach w from v is through u or an ancestor of u (since it is
left to right\.")
But, C contains a path from v to u\\ and C cannot
impossible to go \"from
in C
contain an ancestor
of v since v has the lowest DFS number
\342\226\2
Proof: If there is a

completes

The

a cycle.

algorithm

for determining

graph is

a directed

whether

acyclic is given

in Fig.

7.11.

Algorithm Find_a_Cycle (G) ;

Input: G =(V, E) (a directed graph).

Output: FindaCycle
Use DFS,

starting

from

(true

if G

contains

a cycle and

vertex,

an arbitrary

false

otherwise).

the following

with

preWORK

andpostWORK:

preWORK:
v.on_the_path

:=

{ x.onjhejpath is true
{ initially

x.onjhejpath

true

ifx

is on

= false

the

from

path

the root

to

the

current

for all vertices,and FindaCycle

postWORK:
then
FindjijCycle
ifw.onjhejpath
on v's list
vertex
last
is
the
ifw

Figure 7.11

:= true
then

halt

v.onjhe_path

Algorithm

Findji_Cycle.

:=false

vertex}
is

false

Search

Breadth-First

7.3.2

search

Breadth-first

by

7.12). The traversal

DFS, except

If we

level.

level includes

second

The

is

all the

to

visit

is replaced

to DFS numbers.

\"grandchildren,\"

to

similarly

implemented

stack

the

that

in what seems like a more organizedorder

the graph
start from a vertex v, then all v's children are visited

traverses

(BFS)

so level

\342\200\224
it does
first.

Algorithms

Graph

198

We can

a queue.

by

nonrecursive

the

and

on (see

so

implementation

associate BFS numbers

Fig.
of
with

is, a vertex w has BFS number k if it was the kih

We can build
a BFS tree by including only
that lead
vertex to be marked
by BFS.
edges
in Fig. 7.13. (The notion
vertices.
The BFS algorithm
is given
visited
of
to newly
it is for DFS, since intuitively
not
as
well
defined
for
BFS
as
is
the
search
postWORK
and up,\" but only down;
we therefore
omit it.)
does not proceed \"down
vertices

similarly

That

7.5

Lemma
\342\226\241

to a BFS tree, such that

u is a parent ofw,
If an edge (u,w) belongs
with
has the minimal BFS number
vertices
among
edges leading to w.
The

Proof:

claim follows from

For each vertex

the root to w in

The

Left to

level

traverses the

first-in-first-out

property

of the

T is a

shortestpath

queue.

\342\226\2

7.6

Lemma
\342\226\241

Proof:

the

then

of a

graph

w,

the path

from

the

root

to w in

from

G.
the

reader.

vertex w

level

\342\226\2

is the

length of

the

path

in the

tree from

by level.

Figure 7.12

BFS

tree

for a

directed graph.

the

root

to w.

BFS

7.4

Breadth_First_Search
(G, v);
E) (an undirected connected graph),

Algorithm

Input: G = (V,
Output: depends

and

199

Sorting

Topological

vertex of G).

v (a

the application.

on

begin

mark v
put

the

while

{ First In First Out};

a queue

v in

is not

queue

do

empty

the first vertex wfrom

remove

the

queue

perform preWORK on w ;
{ preWORK depends on the application
for all edges(u\\ x) such that x is unmarked
mark

do

x;

add

to the

x)

(u\\

x in

put

ofBFS

tree T ;

the queue

end

Figure 7.13 Algorithm

7.7

Lemma
\342\226\241

// (v,w)

is an

vertices

whose level numbers

Proof: Left

7.4

edge

to the

we know

that

Now

graphs.

Breadth_First_Search.

how

use the

We again

Topological

in E

that does

not

differ

to T,

belong

by at most

then

it

two

connects

1.

reader.
to

\342\226\2

a graph, we

traverse

present several algorithms

design-by-induction

involving

very heavily.

technique

Sorting

is a set of tasks that

that there
on
other
tasks
and
cannot
depend
they

need

Suppose

to be

be started

performed one

at

Some tasks

a time.

tasks are

completed. All
for
the tasks
a
schedule
we want to arrange
and
dependencies
performing
which is consistent with the dependencies
(i.e., every task is scheduled to be performed
after all the tasks on which
it is dependent
are completed). We want to design a fast
only
to generate such a schedule. This
is called topological sorting. We
algorithm
problem
other

the

are known,

the

can

associate

way.

Each

until

if y

task

cannot

sometaskscan

a directed graph
is associated

start
never

until

x is

with
with

finished.

be started.

the

a vertex

tasks and their

and there is a

Obviously,

the

dependencies

in the

directededgefrom

graph

must

task

be acyclic;

following
x to task

otherwise,

Graph

200

Algorithms

The

Problem

vertices,

label

vertices

Given a directedacyclic
n
G = (V, E) with
graph
from 1 to n such that, if v is labeled k, then all
be reached from v by a directed path are labeled with

vertices

the
can

that

>k.

labels

The straightforward

induction

is the

hypothesis

Induction hypothesis: We know

with < n vertices according to the
case of one

The base
remove

one vertex,

are free

to choose

will
It

apply

a vertex

is clearly

zero. This

(task)

be labeled

can

vertex

indegree zero? The

The following
lemma

A directed

go

and

We
it

a cycle,

a vertex

will see

shortly

it

its

algorithm

with

we

once

followed

variable

yes, since we must

and

that

easiest to label?
whose indegree is
of

a vertex

find

always

able

be

somewhere.

to start

with

edges, and
2 to n.

adjacent

decided
little

labels
1

to

how

traverse

could

we

then

0.

indegree

the

to select

in an acyclic

graph.

\342\226\2

with

vertex

0.

indegree

label the
(To be

instead

graph

however, we
(By the same

many vertices,

finitely

0.)

a vertex

to find

are

impossible

outdegree

how

indegrees,

stop. Sincethere
is

with

vertex

of 2
of

rest

Once we

of the

to

n,

but

indegree

graph

this

causes

0 for

it, we

find

the

label
is still

\342\200\224
which

precise,

completely

induction

no problems.)

the

reduction,

the

effort.

The only implementation

are
problems
to adjust the indegrees when
is
a vertex
with
each
such
that
Indegree
vertex,
v.Indegree
initially
variables
can be initialized
all the
Indegree
by traversing

Implementation

indegree 0
The

Can we

a vertex

choose

is the

a vertex

namely,

any problems.

positive

which

with

with
acyclic, of course \342\200\224
hypothesis assumed labels of

Notice that

without

had

vertices

the

there is

1, remove

to

this fact.

establishes

never have to

through

argument,

acyclic graph always containsa

Proof: If all
must

try

\342\200\224

dependencies

is intuitively

answer

and

hypothesis,

a graph with
n vertices,
extend
the labeling. We

consider

we

usual,

7.8

\342\226\241
Lemma

\"backward\"

no

with

graphs

above.

as the rtth vertex. Therefore, we should

need to label vertices.
Which
vertex

any vertex
our work. We

simplify

As

induction

the

all directed acyclic

to label

how

conditions

is trivial.

vertex

following.

how to find a vertex with

removed. We associatea
is equal to v's indegree.

edges

in

any

order

(using

DFS, for example), and incrementing

whenever an edge (v, w) is traversed.
w.Indegree
The vertices with indegree
0 are put in a queue (a stack will do just as well). By Lemma
7.8, there is at least one vertex v with indegree 0. It is easy to find v \342\200\224
it is simply
removed from the queue.
Then, for each edge (v, w) coming
out of v, the counter of w is
decreased
is put on the queue.
A removal of
by 1. When a counter becomes 0, the vertex
v leaves
the graph still acyclic. Therefore, by Lemma
one
7.8, there must be at least

7.5

of

vertex

in

indegree

becomes empty,

in

the

Initializing
Complexity
a vertex with
indegree
Finding
(v, w)

In

this

time of the

Indegree

algorithm

we deal

section,

with

in the

can

think

of

we

path;

it as

with

will

a directed

the

is called

that

such

edges (in opposite directions)

as
section applies to undirected graphs
of its edges.
lengths

directed

the

with

Input:
Output:

well.

(G)

Topological^Sorting

Algorithm

We

edges.

problem

graph

Let G

graphs.

weighted

(Length of a path
be careful to avoid

problem).

path

lightest

Paths

Shortest

nonnegative
weights associated
because traditionally the
section,
edges

is given

input.

Single-Source

the

algorithm

the queue

0 takes

is considered once (when

need to be updated

The running

7.5

when

algorithm

variables
+ |\302\243|) time.
0(|V|
requires
constant time (accessinga queue). Each
edge
v is taken from the
Thus, the number of times
queue).
is exactly
of edges in the graph.
equal to the number
is therefore O (| V \\ + \\ E |), which is linear in the size

the

variables

of the

terminates

graph.

7.14.

Fig.

the

The

vertices have been labeled. The

remaining

case all the

in which

201

Shortest Paths

Single-Source

G =(V, E) (a directed
acyclic
a
The Label field indicates

=(V, E) be a directed
the weights

call

will

the shortest

path

the

each undirected
same

length.
The length

graph

Thus,

of a

the

path

graph).

sorting of G.

begin

Initialize

GJabel

for all

v.Indegree

:= 0

vertices ; {e.g.,by

DFS

fori := Hondo

if

= 0

Vj.Indegree

then put

v,

in Queue

repeat

remove vertex v from Queue ;

GJabel := GJabel + I ;
v.label := GJabel ;

for all edges(v,

w.Indegree

if w.Indegree

w)

do

:= w.Indegree - I ;
= 0 then
put w in

Queue

until Queue is empty

end

Figure

7.14

this

in

(rather

than

number

of

the

is undirected, we

edge correspondsto

topological

lengths

problem

denotes

sometimes

also

confusion.) If

with

graph

Algorithm Topological_Sorting.

discussion
is

the

two

in this

sum of

the

202 Graph Algorithms

shortest

For

Given a

Problem

The

discuss

we

simplicity,

v to all

from

paths

directed graph G = (V, E)

other vertices of G.

only

find

to

how

and

of

the length

v, find

a vertex

shortest

the

a segment
through

it,

may

or to

actual

to its

correspond

of constructing

the cost

the

to

length,

on the

it, depending

time

expected

it

The

paths.

to find the actual paths. There are many

examples
algorithms can be extended
path problems. For example,the graph may correspond to a road map, and

of shortest

of

the length

to travel

takes

problem.

The Acyclic Case

Let's

G is acyclic. The problem

is easier in this case, and its
to
the
case.
We
on the
help
general
try induction
\342\200\224
of vertices.
The base case is trivial.
Let
number
n.
We can use topological
| V \\
in the previous section. If the label of v is k, then all vertices with
as discussed
sorting
<k need not be considered. There
labels
is no way to reach these vertices
from
v.
the order imposed by the topological
is
a
order
for
Furthermore,
the
sorting
good
first

that

assume

induction.

the graph

us to

will

solution

Consider

last

the

solution

find

the vertex z

vertex,
namely,
know the shortest

with

label

n.

Suppose

vertices except for z.

(inductively)
already
paths
Denote
the length
of the shortest path from v to w by w.SP. To find z.SP, we need only to
check those vertices
w with
to all other
edges leading to z. Since the shortest
paths
vertices are already
z.SP is equal to the minimum,
over all w with
an edge to z, of
known,
w.SP+ length (w\\ z). Are we done? We have to be careful that adding z does not shorten
the distance
to other vertices. But, since z is the last vertex in the topological
order, no
other vertex in the graph can be reached from z, so no other path is affected.
Therefore,
it back, we have
by removing z, computing the shortest
paths without it, then
putting
solved the problem.
The corresponding
induction hypothesis
is the following.
we

that

hypothesis: Given a topological

v
of the shortest pathsfrom

Induction
the

Given

lengths

an acyclic

graph

with

n vertices

the reduced problem

w.SP + length (w, z) over all w
We now improve the algorithm
with the shortest paths. In other
topological
sorting and one for
solve

Considerthe

by

that

such
such

the

z)e

E.

we want to

shortest

how to find

vertices.

we remove the /2th vertex,

the minimum of the values
The algorithm is given in Fig. 7.15.
order,

take

topological order can

combine the
paths, into one pass.

that the

words,

the algorithm

first

then

(w\\

we know

ordering,

to the

a topological

in

induction,

v to all

from

be
two

found
passes,

hand

in

hand

one for

the

recursively (after the topological

label of v in the topological order is 1.
The first step is the call to the recursive
v is
It will call itself repeatedly until
procedure.
reached.
At that time, the length of the shortest
path to v is set to 0, and the recursion
starts to unfold. The vertex
u with label 2 will
be considered
of its
next, and the length
shortest path will be set to the length of the edge from v to u if it exists; otherwise,
there
is no path from v to u. The next step will be to check the vertex x with label 3. In this
order

is found).

way

Assume, for

simplicity,

will

be

that

executed

the

7.5

Input: G =(V, E) (a weighted

acyclic
and
n (the number of vertices).

Output:
from

w e

vertex

every

vertex),

length of the

is the

w.SP

V,

v (a

graph),

shortest

algorithm,

path

to w.

that a topological sort has already been performed.

which computes the topological
order as well, is given

assume

We

For
v

(G, v, n)

Acyclic_Shortest_Paths

Algorithm

203

Shortest Paths

Single-Source

An
in

improved
7.16.

Fig.

begin

be

let z

ifz * v

n { in

labeled

vertex

the

the

order };

topological

then

Acyclic

_Shorte st_Paths

{ G-z

results

for all w such

if w.SP+

(u\\ z)

that

incident

its

with

edges from

G}

e E do
z.SP

z) <

lengthfw,

:= w.SP

z.SP

(G-z,v,n-\\);

removing

from

then

z) ;

length(w,

:= 0

else vSP

end
Figure7.15Algorithm
case, there may be

execute the

same

at the

algorithms

labeled

to m

need to

labels

assume

We

we

and

to

them

are already

path to

shortest

(w\\ z).

w.SP

u\\

+ length (w,

z).

sense

of labels

in advance, and
that the lengths of
edges

coming

vertices

with

the

each

such edge

the shortest

hence
length

of

as

before,

Each
0(|V|

when
vertex

its

tail

is

+ |\302\243|).

is removed

considered

checkedonce in

from the
only

once.

+ 1,

(u\\

the

Therefore,

the

the

induction

the

the length
of the
+
z
is
w.SP
to
edge

we know

over

minimum,
worry

no way

initialization

queue. The queue

To find

of the shortest

lengths

this

path to z is the
we need not
is

paths

call it z.

labels. By
the

z),

both

run

to vertices

The topologicalorder

into z.

through

path

the shortest

Furthermore,

Each edge is

Complexity

try to

This order

be able to

will
shortest

the

smaller

shortest paths
to vertices
with lower labels, since there
from z. The improved
is given in Fig. 7.16.
algorithm

once

be

will

now

v.

from

starting

we

labeled m

the vertex

consider

For

known.

paths
we

\"backward,\"

have already been considered;hence,

w.SP,

Therefore,

the corresponding

and

w,

order

increasing
the

know

same time.

are known,

in

from

in some

order of labels.

in increasing
is applied

these vertices

hypothesis,

of

v and/or

recursion

path to z, we need to check all

that all such edges come from

shortest
guarantees

length

x from

applying

steps

the

eliminate

paths

of

induction

The
will

edges to

Instead

compared.

Paths.

Shortest

Acyclic

to reach

any

all

w>

adjusting
of

them

of the indegrees and

in constant time.
time is
running

is accessed
worst-case

about

Algorithms

Graph

204

Algorithm Improved_Acyclic_Shortest_Paths (G, v);

v (a vertex of G).
Input: G = (V, E) (a weighted
graph),
acyclic
Output: For every vertex vt\\ w.SP is the length of the shortest path from
and it includes
is a nonrecursive version of the previous
algorithm,
{ This

topological sorting

to w.

begin

for all

vertices w

Initialize

; { e.g.,by

all vertices

for

v.indegree

:= 1 to n

fori

do

:= oo ;

w.SP

DFS

do

= 0

ifvj.indegree

then

put

v, in

Queue ;

v.SP:= 0 ;
repeat

vertex

remove

Queue ;

wfrom

for all edges(w\\

z) do

+ length(w,
z) < z.SP then
z.SP := w.SP+ length(w,
z);
z.indegree := z.indegree 1 ;
if w.SP

^ 0

ifz.indegree
until

then

z in

put

Queue ;

is empty

Queue

end

Figure 7.16

the

When

algorithms

use the

Improved_Acyclic_Shortest_Paths.

Case

General

The

Algorithm

acyclic, there is no such

is not

graph

we just discussed cannot be applied

ideas
of these algorithms for the general

we presented

If z

is a

is a

vertex

with

following
k, then

label

and (2) there

< k,

labels

with

of the

result

thing

directly.

case.

order, and the

be possible, however, to
The simplicity of the algorithms
as

a topological

It may

feature of topologicalorder:
(I) there are no paths

are no

from

paths

vertices

from

z to

vertices

with

labels

> k

to

z.

This feature

enables

vertices of a general

The ideais to
of their shortest
find

them

paths

during the
(v, x) be

of v. Let
positive, the

us to

after z

that are

vertices

shortest

in

graph
consider
from

the

find

the

shortest

topological

path

consider the
order

on the

will allow us to do something similar?

the vertices of the graph
in the order imposed by the lengths
v. We do not know
we will
these lengths initially,
of course;

that

execution of the
the

path from v to z without having to

order. Can we somehow define an

edge

from

of
v

algorithm.

minimum
to x is the

First,

we check

length among

edge (v,

x).

them.

All other

all

the
Since

coming

edges
all

paths from

lengths
v

are

out
are

at least

Shortest Paths

7.5 Single-Source

as long.

Let's

vertex? We

the shortest

we know

So,

induction.

choose

paths we need to considerare

the
first edge is (v, jc) and
or length
(vy y) (y*x)
length

consider any

is the

other

this

to jc, and

path

try one more step.

the vertex that is

can serve

How can we find

second closestto v

edges

from

or

paths

the
(x is

as

the

shortest

the

first

consisting

case for the

to
one more
path
base

The

closest).

of

two

only

\342\200\224
the

edges

is an edge from x. We choose

the minimum
of
we
do not need to
(v, jc) + length (xyz) (z*v). Again,
since this is the shortest
way to get out of v (except to x). Here

other

second

paths,
induction
hypothesis.

general

and
a vertex v, we
Induction hypothesis: Given a graph
vertices that are closestto v and the lengths of the shortest
paths

Notice that

205

is on

induction

the

been computed and

the closest vertices

not

to

above), so the
is
problem solved.
vertex (x

on
and
base

the

know

to them.

whose shortest paths

have
already
it assumes that these are
the size of the graph.
Furthermore,
that we can identify
We know how to find the closest
them.
k = | V \\ - 1, the complete
case, with k = 1, is solved. When
of vertices

number

the

closest vertices to v by Vk. The problem is to

not in Vk> and to find the shortest
among
v
v
from
w
the vertices in Vk. It
from
to
w.
The
shortest
to
can
only
path
go through
path
cannot include vertices not in Vk, since they would then be closer to v than w. Therefore,
vertices from Vk to vertices
to find vt\\ it is sufficient to consider only edges connecting
for now.
Let (w, z) be an edge such that u is in
not in Vk; all other edges can be ignored
of the
from
v to z, which
to a path
consists
Vk and z is not. Such an edge
corresponds
and
the
We
need
shortest path from v to u (already known
edge (w, z).
only
by induction)
to compare all such paths, and take the shortest among them.
is the following.
At each
The algorithm
by the induction hypothesis
implied
a new vertex is added. It is the vertex w such that
the length
iteration,
Denote

find

a vertex

the set containing

w that is closest

min (u.SP

+ length

minimal

over all

is the

closest vertex to v;
The

algorithm

thus,

to

and

the k

the vertices

<n i \\

(w, w))

not

adding

in Vk. By
it extends

is complete

now,

but

the
the

arguments

above,

induction

hypothesis.

its efficiency

w is

indeed

can be improved.

the

(k+

The main

l)th

step

the
the next closest vertex. This is done by computing
algorithm involves finding
minimal path length
all
the
it
is
not
to
check
to
However,
(7.1).
necessary
according
values
u.SP + length (w, w) in every step. Most of these
a
values
are not changed when
new vertex is added; only
those
that correspond
that go through
the new vertex
to paths
We can maintain
the lengths
of the known
shortest
may change.
paths to all vertices in
and
is
when
extended.
The only way to find better shortest
Vky
Vk
update them only
when
w is added to Vk is to go through w. Therefore,
we need to check all edges
paths
in Vk. For each such edge (u\\ z), we check the length
out of w to vertices not
of
coming
w.SP + length (u\\ z), and update z.SP if necessary. Thus, each iteration
involves
finding
a vertex with minimum
the SP values of some of the
SP value, and updating
remaining
as
vertices.
This algorithm is known
Dijkstra's
algorithm.
of the

Graph

206

Algorithms

to be able to find a minimum among a set of path lengths,

for finding
A heap is a good data structure
lengths
frequently.
Since
we
need
to
find
of
the vertex
elements.
and
lengths
updating
in
a
in
with
their
all
vertices
not
we
current
minimum
Vk
yet
heap
with
keep
path length,
all
of
v
but
one
the
as
their
from
known shortest path lengths
keys. Initially,
path
lengths
w is
order (except that v is on top). Finding
so the heap is ordered in no particular
are \302\253>,
All the edges (u>, w) can be checked
it from the top of the heap.
take
easy; we can simply
without
However, when a path length
to,
the path lengths can be updated
and
difficulty.
We need

Implementation

and to update
path
minimum elements

say, z is updated,

To do

accordingly.

heap

a heap is

the

in

the

structure;
heap can be

may change. We need to be able to modify the

to know z's position in the heap. (Remember that
to locate an element.)
not provide any facilities

heap
need

we

that,

a search

not

Locating z

z's place in

it

does

done

with

data

another

connected to

structure

the

heap.

in an array
ahead of time, we can put them
the identities of all verticesare known
to their location in the heap.
with
Finding a vertex in the heap thus requires only
pointers
of the heap are the vertices
of the graph, the
Since the elements
the array.
accessing
Since

of

O (|

is only

requirement

space

an element

the

heap

which

|),

smaller

becomes

is reasonable.
than its

This
its appropriate position is found.
The
shortest
maintenance
procedures (e.g., insert).

up until

parent,

Path

the same

is exactly
paths

as the

in

If

moved

and

regular

is given

algorithm

(G, v);
Single_Source_Shortest_Paths
and
E) (a weighted directed graph),
Output: for each vertex u>, w.SP is the length of the
are assumed to be nonnegative.
( all lengths
)

decrease.

only

lengths

be exchanged

it can

Fig.

heap
7.17.

Algorithm

Input: G

source vertex).

v (the

=(V,

shortest

path

from

to w.

begin

for

all vertices
w.mark

do

:= false ;

W.SP:= oo

:= 0;

v.SP

while there exists an unmarked

vertex do
let w be an unmarked vertex such that
w.mark := true ;
for

w.SP

is minimal;

all edges (w, z) such that z is unmarked do

z) < z.SP then
ifwSP + length(u\\
zSP := w.SP+ length(u\\
z)

end

Figure 7.17

Updating

Complexity
the

size

of the

the

most

|E |

of

length

heap. There are | V

There are also at

Single_Source_Shortest_Paths.

Algorithm

updates

a path

iterations,

takes 0(log
leading

(since each

to | V

edge can

where

comparisons,

m)
|

cause

deletions

the heap.
one update),

from

at most

m is

7.5 Single-Source
Shortest
leading to O (| E | log
O ((| E | + | V |) log | V

for acyclic graphs,

(arbitrarily

V |)

comparisons

|). Notice that

since the

heap.

algorithm

in the

vertex

next

Hence, the
is slower than
the

the

in

this

latter

was

algorithm

207

Paths

is

time

running

same

taken

algorithm

from

the

queue, and no updates were required.

ordered)
7.1

\342\226\241
Example

An example
line includes

of

is given

Single_Source_Shortest_Paths

algorithm

in

7.18.

Fig.

The first

only
edge from v. The shortest path is chosen, in this case,
paths
to
vertex
The
a.
second
line shows the update
of the paths including
now
all
leading
v or ay and the shortest path now
either
leads
to c. A new
vertex
paths of one edge from
is chosen in each
known
shortest
line, and the current
paths from v are listed to every
vertex. Thecircleddistances
are those that are known
to be the shortest.
\342\226\

of one

This type

Comments

of

is assigned a priority
(e.g.,
vertices are traversed according to
vertex

the

current

that

priority.

called

the

considered,

-\342\226\272#

Figure

oo

oo

oo

oo

oo

oo

oo

oo

oo

oo

12

oo

oo

12

oo

oo

12

11

oo

12

11

12

11

7.18

\302\251

\302\251

\302\251

\302\251 \302\251 \302\251

\302\251 \302\251 \302\251 \302\251

\302\251 \302\251 \302\251 \302\251 \302\251

\302\251 \302\251 \302\251 \302\251 \302\251

\302\251 \302\251 \302\251 \302\251 \302\251

An example

of the

single-source

12

\302\251

\302\251 \302\251

shortest-paths

and
all

\342\226\272#

\342\200\224
each

source),

++

search

priority

known distance from

When a vertex is

sometimes

is

algorithm

algorithm.

its

Graph

208

Algorithms

a change in some
are checked. That check may
The
priorities.
trigger
one
search
from
that
is
another.
for
what
priority
making
change
distinguishes
procedure
It is useful
for problems
search.
involving
Priority search is more expensivethan regular
weighted graphs.
one path at a
the shortest paths from v to all other vertices by finding
found
We
led from
was identified
a previously
time. Each additional
path
by one edge, which
v as its
All those edges together form a tree with
vertex.
shortest
known
path to a new
in dealing
with
root (Exercise 7.6). This tree, called the shortest path tree, is important
adjacent

edges

of path

variety

problems.

Minimum-Cost

7.6

Trees

Spanning

of computers connected through

bidirectional
links.
There is a
a network
Consider
positive costassociatedwith sending a message on each of the links. We assume that the
a message
We want
on a specific link does not depend on the direction.
cost of sending
We
to broadcast a message to all the computers starting from an arbitrary
computer.
assume that the cost of the broadcast is the sum of the costs of the links used to forward
the
(Another
message.
possible definition of cost is the time it takes to complete the
see Exercise 7.63.) The network
can
be represented
broadcast;
by an undirected graph
with

on

costs

positive

the

The

edges.

(corresponding to the links

the sum of the costs of the

is to

problem

in the

used

broadcast),

find

fixed
all

containing

connected
the

subgraph

vertices, such that

not difficult to see

subgraph is minimum. It is
we could
have
subgraph
any
cycle had been present, then
broken it by deleting one of its edges;
the graph would still
be connected,
but the cost
would
be smaller
since all costs are positive. This subgraph
is called the minimum-cost
and
it has many uses besides broadcasts. Our
is to find an
spanning tree (MCST),
goal
efficient

the

ties

are

a tree.

For simplicity,

MCST.1

an

that

implies

to discuss.

If

The

the MCST is

algorithm

unique

the

remains

we assume
(Exercise

the

that

7.11),

same without

costs

are

which makes

this

assumption,

equal-cost edges are encountered,

any one of them can be chosen(i.e.,
The proof of correctnessis more complicated
in this case.
arbitrarily).

that, when
broken

Problem

The
G =

(Notice

find

assumption

easier

problem

except

to

algorithm

distinct. This

in the

edges

must be

this

that

(V,

find
\302\243),

that we

now

an

Given

a spanning

call

tree T of G of

the weights

connected

undirected

minimum

costs.) The

straightforward

weighted

graph

cost.

induction

hypothesis

is

the following.
1
is known to us. The complete
We assume here that the whole graph
network
topology of a communication
and all current costsare usually unknown only at the local sites; therefore, a distributed algorithm
is needed.

Induction

graphs

< m

< m

with

If it

is

the MCST

Given

We claim

edges?
not

how

to find

this

problem
the

that

then

included,

edge from

other

any

removing

Minimum-Cost

Spanning Trees

the MCST

209

for connected

edges.

The base case is trivial.

a problem
MCST.
the

know

1: We

hypothesis
with

7.6

adding

it

creates

cycle

m edges,

with

minimum-cost
to

the

reduce it

how do we

to

edge must be included in

MCST would create a cycle;

a tree again,

but

with

smaller

cost,

of the MCST. So, we now

know
one edge that
which is a contradiction to the minimality
We
can
to
the
MCST.
remove
this
the
and
from
induction
to
belongs
edge
graph,
apply
Is
rest
of
the
which
now
contains
less
a
valid
use
that
induction?
of
the
graph,
edges.
This
is not a valid use of induction,
after we remove an edge,
the problem
because,
we need to solveis not the same as the original
First, the selection of one edge
problem.
we remove
an edge, the graph
the selection
of other edges. Second,after
limits
may not
We cannot emphasize this
issue
the induction
too strongly \342\200\224
be connected any more.
and followed.
hypothesis has to be preciselydefined
The solution is to adjust
We know
how
the induction
to select the first
hypothesis.
since
rest
the
but we cannot simply remove it and forget about it,
of the selections
edge,
of removing
it, we mark it as being selected and use this
depend on it. Therefore,instead
The algorithm
one edge at a
fact
for the algorithm.
(its selection)
proceeds by selecting
but rather on the
time to the MCST.
Thus, the induction is not on the size of the graph,

already selected in

of edges

number

Induction hypothesis 2:
how

to find

tree that

is a

of

subgraph

Given

T ofG

subgraph

the

fixed graph.

a given

connected

with k

MCST

graph

G=(V, E), we

edges (k<\\V\\-\\),

such

that

know

T is

ofG.

is choosing
the first
discussed the base case for this hypothesis, which
We assume that we have already found the tree T satisfying the induction
and we need to extend T by one more edge. How can we find another
edge
hypothesis,
that is guaranteed to be in the MCST?
that was used to find
We apply the same argument
the first edge.
there must be at least
T is already known
MCST.
to be part of the
Hence,
one edge in the MCST
will
T
in
T.
We
to
vertices
not
try to find one such
connecting
in
T.
We claim that the
edge. Let Ek be the set of all edges connectingT to vertices not
this edge by (w, w) (see
Denote
cost in Ek belongs to the MCST.
edge with minimum
a unique path from u to w
is a spanning tree, it contains
Fig. 7.19). Since the MCST
If (w, w) does not
between
(there exists a unique
path
every two vertices in a tree).

We have already

edge.

belong

to the

belong to T and

MCST,

it is

then

w does

not included

not belong to

in

that

7, there must

u to w. But,
one edge (jc,

path from
at least

since

u does

y) in this

path
a vertex not in T. The cost of this edge is higher than the cost of
cost among all such edges. But now we can use the
since (w, w) has the minimum
(w, w),
the
same
to
first selected
we
that
argument
applied
edge. If we add (w, w) to the MCST
and
remove
tree with smaller cost, which
is a
the edge (jc, y), we get another
spanning
that

connects

be

T to

contradiction.

Implementation
algorithm
presented

This
in

the

algorithm

previous

is

very

section.

similar

The

to
first

the

single-source

chosen

edge

shortest-path

is the

edge

with

Algorithms

Graph

210

\342\200\224#

(1

>u

fg f

\342\200\224i
>

7.19

Figure

the

Finding

then defined as a tree

T is

cost.

minimum

\342\200\242

\342\200\242

next

with

edge

MCST.

of the

this edge.

only

each

In

we need

iteration,

of T. In the shortest-path
edge connecting T to verticesoutside
the minimum-length
outside of T. Hence, the only
leading
path
and
the shortest-path
difference
between the MCST algorithm
algorithm is that the
minimum
is taken not on the length of a path but on the cost of an edge.
Tue rest of the
in 7, the
is
same.
We
each
w
included
the
for
vertex
not
maintain,
algorithm
virtually
\302\253>
w
in
T
if
minimum-cost
a
such
to
from
vertex
no
(or
edge leading
edge exists). In each
we
w to
choose the minimum-cost
vertex
iteration,
edge and connect the corresponding
T. We then check all the edges incident
to w. If the cost of any
such
edge (u\\ z) (for z
in T) is smaller than
not
z's cost.
the cost of the current
best edge leading to z, we update
The algorithm is presented in Fig. 7.20.

to

find

minimum-cost

the

we

algorithm

found

The complexity of this

Complexity

shortest-path algorithm
+ |\302\243|)log|V|).
0((|V|
Example

7.2

An example

of algorithm

of

is the

\342\226\241

the

table

connectedto

v are

is chosen

each

vertices

in

are updated

Comments

one, of a method
elements, each

elements

with

presented

MCST

one

that

is

listed along with

line.
The current
at each

step (only

The algorithm for

the greedy
called
with

maximum

an

to

the single-source
time is
running

of

that

previous section. The worst-case

is illustrated
added

is identical

algorithm

in the

in

Fig.

7.21.

The vertex

at that step. The first

The vertex
their
costs.

best edges (and

the

tails

of the

their

vertex
with

costs)

the

in

the

first column

is v, and

the edges

minimum-cost

leading

edges are listed).

to

edge

unmarked

\342\226\2

an MCST
not a pure
is an example, although
method. Suppose that we are dealing with a set of
associated
the set of
cost, and that we are interested in finding
cost satisfying
In the MCST
some
constraints.
(or minimum)
finding

7.6

Algorithm MCST

Input: G

Minimum-Cost

Spanning

Trees

211

(G);
undirected

(a weighted

Output: T (a minimum-cost

graph).
of G).

tree

spanning

begin

for all vertices

w do

w.Mark

let (jc,

; { w.Mark

:=false

cost edge

be a minimum

y)

:=

ifw is

in

T}

z.Edge

loop }

cost edge from

{
of z.Edge }
exists an unmarked vertex do
vertex such that w.Cost
an unmarked

T to

; {

:=

z) do

{ a minimum

z);

(xy

let w be
ifw. Cost = oo

the

z}

cost

is minimal

then

\"G is

print

marked

be

will

z.Cost := cost(x,z);
there

in

in the main

true

for all edges(jc,

while

is true

.= oo ;

W.CoSt

x.Mark

set;

the empty

T is

Initially

not connected\" ;

halt

else

:= true

w.Mark

add w.Edgeto
{ we

now

for all
if

T ;

the costs

update
(w,

edges

not

z.Mark

of unmarked vertices connectedtow}

z) do
then

if cost(w, z) < z.Costthen

:= (V, z);
z.Edge

z.Cost := cost(w,

z)

end

Figure

were the

7.20

Algorithm

MCST.

was that the edges

of the graph, and the constraint
is
The
to
be
method
correspond
greedy and take the
spanning
greedy
maximal-cost
In
some
element
at
MCST
we introduced
the
possible
any step.
algorithm,
more constraints
of edges, specifically,we considered
on the selection
only edges that
were connected to the current
tree. Therefore,
the MCST algorithm
is not purely greedy.
We can also, however,
find the MCST by selecting, at each
step, the minimum-cost edge
this
in the graph, provided that
anywhere
edge does not form a cycle (Exercise
7.59).
lead
to an optimal solution. It is usually
not
The
does
method
always
greedy
just a
solutions.
heuristic to find suboptimal
Sometimes,
however, as in the MCST example,
to the best solution.
the greedy method
lead
does
problem,

the elements
to

tree.

edges

212 Graph Algorithms

V

/?

C t

,
i

d]

\342\200\242

Figure 7.21 An

consider

We

now

in

a graph.

The

/?

/i

oo

oo

a{2)

v(9)

oo

oo

oo

oo

v(6)

c(10)

oo

oo

v(6)

c(4)
-

oo

the problem

with

of the

d{l)

c(10)

rf(12)

oo

6(3)

c(10)

</(12)

oo

c(10)

</(12)

*(5)

c(10)
-

MID

MID

minimum-cost spanning-tree algorithm.

Paths
of

shortest

computing

Given a

Problem

undirected)

/
oo

example

All Shortest

7.7

e
oo

v(6)

v(9)

v(6)

v(D

11

oo

\342\200\242

\342\200\242

13

72

70

graph

weighted

weights,

nonnegative

find

paths

between

all pairs of

vertices

(directedor

G =

(V, E)

the

minimum-length

paths

between all pairs of vertices.

This
the
as lengths.
weights
how
For
we
discuss
shortest-paths
all-pairs
problem.
simplicity,
to find only the lengths
We
of the shortest
than
the
themselves.
rather
paths,
paths
assume that the graph is directed; the same arguments hold for undirected graphs. We
assume
this section that all weights are nonnegative; Exercise 7.73deals with
throughout
Again,

since

we are

problem is called the

negative

lengths.

talking

about

shortest

paths,

we refer to

As

or on the

edge, say

(w,

with

is involved

graph? First, the

may be other shorter

there

We can

induction.

straightforward

What

vertices.

to a

w),

Furthermore,

start

let's

usual,

the edges

edge

in

of shortest

terms

paths

worst

on

either

in

a new

adding

u and

between

path

In the

use (w, w).

that

use induction
paths

a shorter

may form

213

Paths

All Shortest

7.7

w.

we need

case,

of the shortest path

to check, for every pair of vertices
V! and v2, whether the
length
to
u
the
the
of
of (w, w) plus
the shortest path from w to v2 is
from
v,
plus
length
length
than
the known path from v, to v2. Overall,
for every
new edge, we may have to
shorter
time of O (| E \\ \\ V |2).
O (| V |2) checks, leading to a worst-case
make
(Since
running
of edges
the number
may be as large as O (| V |2), this is an O (| V |4) algorithm.)
What is involved in terms
a new vertex u to a graph?
of shortest paths in adding
of the shortest paths from u to all other vertices and
from
We first need to find the lengths
do not involve u are already known,
all shortest paths that
all other vertices to w. Since
find the shortest path
from
u to w in the following
to
we can
way. We need only
If this edge is (w, v), then the length of the
the first edge out of u in this path.
determine
from u to w is the length of (w, v) plus the length of the shortest
path from v to w
path
these
for all vertices adjacent
(which is already known). We thereforecompare
lengths
The shortest path from w to u can be found
to w, and take the minimum
length.
similarly.
We still have to check, for any pair of vertices, whether
But again, this is not enough.
exists
a shorter path between the two using the new vertex w. For each pair of
there
of getting
of getting
from v to u plus the length
vertices v and w\\ we check the length
to the length of the
known
shortest
from u to w\\ and we compare this length
previously
and
additions
are needed for each added vertex,
path. Overall 0(|V|2) comparisons
is thus better than
on vertices
the
to an 0(|V|3)
leading
algorithm. The induction
method
for this problem.
on edges, but there exists an even better induction
induction
The trick is to leave the number of edges and vertices
fixed, and to put restrictions
on the type of paths
of these restrictions
The induction addresses the removals
allowed.
on the paths until, at the end, all possible paths
We label the vertices
are considered.
from 1 to | V |. A path
for u and w\\ the highestfrom
u to w is called a k-path if, except
labeled vertex on the path is labeled k. In particular,
a 0-path is an edge
(since no other

vertices can appear

the path).

on

Induction hypothesis:
all pairs of vertices
such

The base of

the

the

is obvious.

it

solution
to

We

m + 1.

that

paths

between

all

Denote by

1, in

vertex

to consider

consider are

induction
all

u and

&-paths

only direct
hypothesis
such that

have

whether

they

shortest

w-path

is the

shortest

the

some k

We

m-paths.

and to check
labeled m. Any

between

w-path

case

which

We assumethe

have

of

lengths

only k-paths,for

of vertices,

pairs
vm the

The shortest
and

now

we need to

that

is m =

induction

the

know

We

shortest

between

paths

are considered.

< m,

edges can be consideredand

for m,

k <m
to find

improve

and

+ 1.
the

on the

must include

\302\243-path (for

some

So, the
shortest
&-paths
vm

k <

k <m\\

and

hence,

between u

only

extend
new

m-paths
for k < m.

exactly

once.

m) betweenu
be equal to k)

some j < m, where j need

not
(for
by the shortest y-path
of all shortest A:-paths
the lengths
for
By induction, we already know
to find the shortest m-path
above
we need only to sum the two lengths

vm appended

betweenvm

try to

we

to w.

w.

Not only is

this

algorithm

faster

(by a constant

factor) than

the

one

using

Graph

214

Algorithms

Weight (an n
[a\\ y ] is the

Weight

Weight

also simple to

program. The algorithm

(Weight);

All_Pairs_Shortest_Paths

Algorithm

Input:

it is

7.22.

in Fig.

on vertices, but

induction

straightforward

the

is given

jc ]

[jc,

Output:

adjacency
of the

weight

for all x )
end, the matrix

matrix representing a weighted

graph).
or \302\260\302\260
otherwise;
edge (jc, y) if it exists,

is 0,

the

At

shortest

x n

of

the lengths

contains

Weight

the

paths.

begin

for m:=

{the induction

to ndo

forx := I to

:= I to n

fory

sequence }

n do
do

y] < Weight[x,

m] + Weight[m,

ifWeight[x,

y] :=

Weight[x,

m]

Weight[x,

then

y]

+ Weight[m,

y]

end

Figure 7.22 Algorithm

The inner
this

that

of

independent

two

loops

can be

check

of the

algorithm are

applied to

the

the

others.

Such

For

each m,

the

Paths.

All_Pairs_Shortest

pairs

used to

check

of vertices
is

flexibility

in

any

important,

all pairs

Notice

of vertices.

check is
order,
for example, for parallel
since each

algorithms.

Complexity

algorithm

involves

only one sum

and

one

comparison

of vertices. The induction

of
is of length
number
| V |, so the total
sequence
(and comparisons) is at most | V |3. Recall that the running time of the singlesource algorithm is O (| E | log | V |). If the graph is dense such that the number
of edges
is Q(a?2), then using this algorithm is better than using the single-source
algorithm for
vertex.
it
is
to
in time
the
every
Although
possible
implement
single-source
algorithm
O (| V |2) (Exercise 7.43.), which will lead to an O (| V \\3) algorithm for all-pairs shortest
the algorithm
in this section is better for dense graphs
it is so simple to
because
paths,
per pair
additions

On

implement.

O (| E | | V

log

the other
| V

hand, if

the

|), resulting from

graph
using

is relatively
the

sparse,

single-source

then

the

algorithm

running

|V

\\

time of
times,

is

better.

7.8
Given

graph
w

in

problem

Transitive

Closure

the transitive
closure C=(V, F) of G is a directed
graph G=(V, \302\243),
that there is an edge (v, w) in C if and only if there is a directedpath from v to
G. The transitive closure is related, for example,
to the user-accounts security
of this
mentioned
at the beginning
chapter. The vertices correspondto the users,
a directed

such

and the

edges correspondto

permissions.

The

transitive

closure

identifies

for

each

user

7.8 Transitive

all

the

users with permission

other

There are

other

many

(either directly
of

applications

or

transitive

the

to use his or her

indirectly)

and so

closure,

215

Closure

account.
is

it efficiently

finding

important.

The Problem

a directed

Given

= (V,

graph G

find
\302\243),

its transitive

closure.

of the
instance
We solve this problem
is, we transform
by using a reduction. That
any
of another problem that
we already
know how
transitive closure problem to an instance
of the
to a solution
the solution of the other
to solve. We then transform
problem
is from the all-pairs shortest-paths problem.
transitive closure problem. The reduction
Let G' = (V, E') be a complete directed graph (i.e., all vertices are connected in
0 if e e \302\243,
1 otherwise.
Each edge e in E' is assigned the length
and
We
both
directions).
for
If
there
a
from
v
w
in
G'.
is
to
now solve the all-pairs
G,
shortest-paths
problem
path
in G' is 0, since all edges
of G have length
0 in G'. Therefore, there is a
its length
then
path

is

easy

algorithm,

and

v and w if

between

is 0. Thus, an answer
into an answer for the
The idea of using
10. We used reduction

to the

closure

mainly

problem.
two problems is

to illustrate

the

between

and

can be transformed

explored in
with

technique

detail

a simple

directly to a

algorithm

shortest-paths

all-pairs
in Fig

path

between

reductions

to modify the
as is shown

shortest

the

all-pairs shortest-paths problem

transitive

here

length of

if the

only

w in

G'

directly

in Chapter

example.

transitive

It

closure

7.23.

Algorithm TransitivejClosure (A) ;

A (an n x n adjacency matrix representing a directedgraph).
Input:
if the edge (jc, v) belongsto the graph,
and false otherwise;
{ A [jc, v ] is true
A [jc, jc] is true for all jc }
of the graph.
the transitive
closure
At the end, the matrix
A represents
Output:
begin

for m:= 1 to ndo

forx := 1 to

:= 1 to

fory

if

the

induction

sequence

ndo

ndo

[jc, m ] and A

{this step

[my

is improved

A [jc, v ] ;= true
the next algorithm }

v ] then
in

end

The
first

problem

fact that we
is general

Figure 7.23 Algorithm

TransitiveJClosure.

can reduce one problem

enough to embody the

solution

to another

means that the solution

other. But, more

of the

of the

general

216

are usually more

to solve; in

solutions

expensive.

is easier

problem

to

Algorithms

Graph

When

it.

for

pay

cases,

many

used

is

reduction

where a more

cases

seen

have

We

general

however, the more you get the more you have

the resulting
we should always
try to improve

solution
by using the special characteristicsof the problem.
It consists of two checks,
the {/statement.
Consider the main step of the algorithm:
if
both of these checks are
An action is taken
for A[my y].
for A[x, m] and
only
for each pair of vertices. Any
This
satisfied.
if statement is performed n times
would
to
a
substantial
statement
lead
this
of
improvement of the algorithm.
improvement
all the time? The first check
on
Do we really need to perform the two checks
depends
on
m
and
we
check
the
second
whereas
can
and
x
Therefore,
m,
y.
depends only
only
x and a certain m. If the
first check
the first check only once for a certain
fails,
perform
for
of
check
value
If
the first check
the second
then there is no need to perform
y.
any
This change is incorporated in the
it again.
then there is no need to perform
succeeds,

but this

unchanged,

The

7.24.

in

Fig.

run

about

Input:

{ A [jc, y ]

[jc, x ]

n xn

(an

the

end,

the

complexity

remains

(A) ;
matrix

(jc, y)

edge

is true for all x

At the

Output:

adjacency

is true if

asymptotic

twice as fast.

Improved_Transitive_Closure

Algorithm

presented
algorithm will

algorithm

(improved)

a directed

representing

belongs to

the

graph,

the

transitive

graph).
otherwise;

and false

matrix

A represents

of G.

closure

begin
{
for m:= 1 to ndo
forx := Hondo

if A

[jc, m ]

the

induction

sequence

then

fory :- I to

ndo

if A

y ] then

[my

A [jc, y

true

:=

end

Figure 7.24

The

Implementation
however,

Algorithm

implementation

that the last line has

the

same

matrix. Each entry

row
(jc, y) in the jcth
These operations are equivalent
to setting
row.

mth

time, a row
in

practice,

Since
or

(the number of
of a

bits

parallel algorithm.

that

of

algorithm is straightforward.
as an or operation on the

the

effect
is set

the

Closure.

jcth

to

the

row

value

of itself

the or
operation on
to be

Notice,

or

of the

can perform
an or
many
performed faster than several bit-by-bit
of steps for this algorithm
is 0(az3/u>), where w
can be or'd together in one step).
This is a very
issue is also discussed in Section
This
9.5.3.

many computers
can be
operation

the number

Transitive

Improved

jcth

row

that

of

jcth

row

of the
(m, y)and the

same
So,
operations.
size
is the word
bits

simple

at the

example

7.9

have

components.

In

when we need to

design an

consider each subgraph

have seen severalalgorithms

that

algorithm

and

that

require

its

that the

the
be
to

these

graph

into

Then,
be possible to

For example, we

property.
connected.
apply

the

property.

it may

graph,

desirable
graph

partition

By partitioning

the

algorithms to each

many complications. This sectionpresentstwo

and strongly connected components.
components

to avoid

thus

\342\200\224
biconnected

decompositions

to use

into connected

partition

desirable

certain

manipulates

components, we were able

component separately,and
other

decomposition

graph

separately

its connected

into

graph

is to

subgraphs satisfies a

of the

each

that

\342\200\224
the

decomposition

graph

idea of

the

general,

such

subgraphs

of

seen one form

already

217

of Graphs

7.9 Decompositions
We

of Graphs

Decompositions

and
the second one to directed graphs. Both
applies to undirected graphs
in
In
both decompositions
on
are
designing algorithms.
particular,
depend heavily
in
the
and directed cycles respectively).
the cycles
(undirected
Therefore,
graph
there is a problem that involves
or another
whenever
(and many graph
cycles in one way

one

first

The

useful

problems involve
not

always

that the

section

cycles),

are

graphs

Biconnected

7.9.1

The notion

a good

it is

but they should

useful,

at

idea to

these

consider

They are

decompositions.

We assume

be considered.

least

throughout

this

connected.

Components

in a natural
extends the regular
connectivity
concept
way.
graph is connected if there is a path from every vertex to every other
are at least two vertex
vertex. An undirected
disjoint
paths
graph is biconnected if there
from every vertex to every other vertex. Biconnected
thus
exhibit
a
level
graphs
higher
of connectivity:
If for some reason one of the paths connecting two
vertices
can no
if a graph is not
It turns out that,
longer be used, then the two vertices are still connected.
is
We
of
which
biconnected.
into subgraphs,
each
then it can be partitioned
biconnected,
will
is called kbe mainly
In general, an undirected
interested in that partition.
graph
between
connected if there are at least k vertex disjoint
every two vertices. We
paths
of

An

undirected

first

study

biconnectivity

of ^-connectedgraphs.
property of ^-connected graphs

several properties
first

The

[1927]

important

relates

that

the number

of vertex

required to disconnect the

vertices

a theorem

vertices

graph.

E)

undirected connected graph, and let u and

The minimum number of verticeswhose
G.

be an

nonadjacent verticesin
from
disjoint

are
A simple

G disconnects
paths

removed
corollary

due to Menger
of
to the number

Theorem

\342\226\241
Menger's

Let G = (V,

paths

disjoint

is

between

from

from

v is

u to v. (When

equal

vertex

to

the

maximal

is removed,

v be

number of vertex
all its incident
edges

as well.)
of Menger's

two

removal

\342\226\

theorem

is the

following,

due to

Whitney

[1932].

218

Graph

Algorithms

Theorem

\342\226\241
Whitney's

An undirected graph is k-connected

if and only if at least
removed in order to disconnect the graph.

must

vertices

be

\342\226\

is equivalent to the condition

in
theorem
kWhitney's
defining
of
these
conditions.
For
a
of
either
one
these
we
can
use
theorems,
proof
connectivity,
is clear: If
and
Lesniak
see for example Chartrand
[1986]. (One side of the theorems
removal
disconnects
the
then
there
cannot
whose
be
more than
vertices
k
there are
graph,
is
more complicated.)
k vertex
disjoint paths; the other direction
is one of the most important
in graph theory. For our
theorems
Menger's theorem
is
of
two
theorems
that
a
is not biconnected
main
the
the
if
implication
graph
purposes,
disconnects
the graph.
Such a vertex
is
and only if there is a vertex whose removal
7.25
illustrates
the structure of a nonbiconnected
an articulation
called
point. Figure
*
contains
one
or
more
articulation
The blocks, 'between\"
a
Such
points.
graph
graph.
in the figure,
are
are
articulation
the
highlighted
by themselves
points, which
of the graph.
the biconnected
We make
biconnected. These blocks form
components
next.
more
this notion
precise
Since

the

condition

Definition:

biconnected

such that its induced

that

biconnected

components.

it and

contains

component
Indeed,

each

is a

component
subgraph

is

maximal subset of

biconnected

(namely,

the

edges

there is no subset

induces a biconnectedgraph).
is defined as
articulation

Figure 7.25

The structure

a set of

edges.

point belongs to more

of a

can belong to

A vertex

nonbiconnected

than

one

graph.

component.

several
(In

7.9 Decompositionsof

219

Graphs

characterization
of articulation
description providesanother
points.) The set of
be
into
can
in a unique
biconnected
partitioned
graph
edges
components
way.
one component.
The following
two
claims
Each edge belongs to exactly
prove the
and its uniqueness.
existence of the partition

fact, this

of every

7.9

Lemma
\342\226\241

to the same biconnected component if and only

Two edges e and f belong
is a cycle containing
both
there
a biconnected
(Note that
of them.
consist
one
this
claim addresses only
component
may
of only
edge;
with at least two edges.)
biconnected
components

First, we show

Proof:

component. If the cycle

then we can extend each

extended subgraph
vertex. This contradicts
theorem, if the

is

is always

a cycle

that

these

of

components
biconnected
since

still
the

biconnected

one

than

by adding

a cycle cannot
be
component. For

of the

maximality

of

rest

the

biconnected

in one

contained

entirely

edges from more

contains

if

component,
the cycle. The
by one
of the

disconnected
side

other

the

belong to the same biconnected

component, then we can obtain
in
the
We
add
them
two
new
vertices to
(artificial)
following
way.
cycle containing
=
if
of e and /. (That
e
we
add
a
new
z
\"middle\"
vertex
and
e by
the
is,
(v, u>),
replace
as a subgraph,
the two edges (v, z) and (z, w)\\ we do the same for/.) The component,
remains biconnected since it still contains no articulation points. (Removing any of the
as removing
the old edges, which
the
new vertices is the same
cannot
disconnect
vertex
has
same
as
are
an
old
the
effect
there
Therefore,
before.)
component; removing
two vertex-disjoint paths
the two new vertices, but these
between
paths exactly complete
a cycle containing
\342\226\2
e and /.
two

edges

the

7.10

Lemma
\342\226\241

Each edge belongsto

Proof: Each
component,
A

edge definitely

combination

want

We

with the

It

to

find

belong

Induction

hypothesis:

connected

graphs

and

edges

one

than

from both
edges of

containing

two

biconnected

components.
components.

\342\226\2

components. Let's

start

as

usual

hypothesis.

We know

with < m

with one

it

into biconnected

partition

induction

straightforward

one biconnected component

to more

is impossible.

this

the

component.
at least

to

cannot

be cycles
since there would
containing
of the two cycles is one larger
cycle

already seen that

We have

belongs

itself).

only

containing

(possibly

one biconnected

exactly

how

to find

the biconnected componentsof

edges.

edge is biconnected.Considera

m edges and
the
induction,
graph
arbitrary
by
to
have
now
We
determine
what
effect
x
would
have
biconnected
adding
components.
x connects
on the partition.
two vertices from the same
The easiest case is when
In this case, adding
x has no effect
(for example the edge (a, n) in Fig. 7.25).
component
A

pick

connected

an

graph

edge x.

We

remove

x from

the

and

graph

find,

with

220

Graph Algorithms

that one particular component even more

(it only makes
connected).
is
x
the
when
the edges
case
disconnects
(for
Another
graph
example
easy
completely
that
of
jc's
in
In
is
both
it
clear
this
and
case,
(az, o)
endpoints are
Fig. 7.25).
(h, i)
a
x
is
itself.
as
a
biconnected
result,
and,
(Such an edge
articulation
component by
points
none
of
the
other
is changed.
a
called
is appropriately
bridge.)
components
Obviously,
is when x does not disconnect
the graph and connects verticesfrom
case
two
The difficult
An example of such an edge is edge (b, e) in Fig. 7.25. We also
different
components.
that x merges the two components
case in Fig. 7.26(a). It is clear
this
it
illustrated
\"in between,\"
are
into one larger
connects, plus several other components that
component.The problem is thus to find all the \"in-between\" components and to merge
on

the

partition

them efficiently.

Fig. 7.26, we can see that the biconnected

in
the
Each biconnected component is
way.
following
them
from
the original
associated with a node (we call them nodes to distinguish
vertices). We start with an arbitrary component R as the root of the tree (the component
of/?
are those biconnected components
containing a, b, and h in Fig. 7.25). The children
articulation
with
the
are those biconnected
have
common
that
R\\
points
grandchildren
have
in
that
not
been
included
the
tree
which
have
common articulation
components
yet,
with the children, and so on. In other words, we construct the trees
in a breadthpoints
first
fashion.
We cannot simply say that
two
biconnected
are connected if
components
have an articulation point
in common,
be common
because an articulation point
they
may
to more than two biconnected
and
we
do
not
to
form
want
components,
cycles. It is not
difficult to prove that a tree is always formed by this construction
(Exercise 7.17). This

Looking back at Fig.

define a tree
components

7.25

and

(b)

(a)
Figure 7.26 An
components

edge

corresponding

The biconnected

that connects
to the

component

two

different

biconnected

graph of Fig. 7.25 with

tree.

the

components,
articulation
points

(a) The

indicated, (b)

7.9

tree
the

is called
graph

x in

edge
an

edge

tree.

the biconnected

in Fig.

7.25,

Fig. 7.26

and

Fig.

illustrates

a and k

connecting

in

shows

7.26(a)

Figure

shows the

7.26(b)

the

addition

the

original

Decompositions

of Graphs

the biconnected

components of

corresponding biconnectedtree.

of an edge;

it

can

correspond,

221

The

for example, to

graph.

tree now, we see that an edge connecting vertices

All the nodes
generates a cycle in the tree.
in that cycle must
be merged
into one component. So we
(correspondingto components)
We add to the induction
have an algorithm.
the assumption that we
now
hypothesis
and then we can handle
of the three cases we
each
know how to construct the tree,
discussedearlier. We omit the details because there is a better algorithm.
we just described
is the time it takes to find the
The
with the algorithm
problem
in
tree. Finding a cycle in a tree may
by the added edge the biconnected
cycle generated
the whole
tree, which in the worst case requires looking at all the edges
require traversing
as O (| V |) edges
in the tree, and we have to perform
of the tree. There may be as many
for each edge of the
this
Thus, this algorithm
step
original
graph.
may
require
a precise
We would
like to avoid searching for
O (| V | \342\200\242
| E |) time (this is not
analysis).
a cycle in each step.
a straightforward inductive
is to choose
One common way to improve
algorithm
order
of induction. In the preceding
we picked an arbitrary
the
discussion,
carefully
if we pick the edges
in an order that will
We may be able to improve
the algorithm
edge.
would be to use a
make it easier to handle
tree. A natural
first attempt
the biconnected
in
DFS
is excellent for
as
we
shall
see
a
that
It
turns
traversal.
out,
moment,
good graph
DFS
this
7.25.
Assume
that
starts
at
vertex
and
consider
Consider
a,
again Fig.
purpose.
B
b
DFS
first
the articulation
b.
Let
be
the
\"below\"
which
the
visits
component
point
of the edges connectingvertices
after visiting b. (In Fig 7.25, this component
consists
b,
b is indeed
an articulation point? By
How can we determine that
c\\ d, e, f and
g.)
b is an
of the graph pass through
if all paths from B to the
rest
b, then
definition,
out of
whether there are any edges
articulation point. So, we want to determine
coming
If we

from

think

of

the

different

two

biconnected

components

B to the rest of the graph.

next
Assume that the vertices in B are visited
by the DFS. If there are no edges out
of \302\243,
will be traversed and
b will be
the traversal
will be local to B. All of \302\243'sedges
DFS eliminates
reached again. Furthermore,
since
cross edges, the only edges that may
connect B to the rest of the graph are back edges. In other words, b disconnects B if and
that reach the tree above b. (The
only
if there are no back edges out ofB
exception
only
to this rule occurs
Let's see now
at the root of the DFS tree; we discuss this case later.)
how

we can

determine

this

fact.

DFS tree we

reach from a subtree. We

one whole subtree below v,
then
be a subtree
rooted at a child of v such that the DFS
another,
visits this child first. Suppose that we find not only all the biconnected componentsin
in the tree that is connected
to T] by a back edge. (This is
vertex
7*1, but also the highest
induction
as
will
be seen
in a moment.) Let's
the
hypothesis,
really
just
strengthening
in the
DFS
is
tree
that
vertex
a back edge,
the
denote
connected,
by
High(v)
highest
by
to either v or a descendent of v (in the DFS tree). Assume that the children of v in the
We

traverse

want to

know

how

high

using DFS. At
and so on. Let Tx

the graph

in the

each

vertex

\\\\

we

can

first visit

Graph Algorithms

222

DFS tree

are

High(Wj)

for all wt: It

v.

edges from

is simply
will describe

(We

back edge from

Now suppose that
articulation

is an

than

higher

v.

Indeed,

subtree rooted

(The

beauty

point
u>, to

at

of

the

7.27,

Fig.
v

if
if

how

DFS,

perform

hence

and

High

vertices

higher

the tree;

v in

than

the

We claim

values.

High

= w$;

edges from

hence,

in exactly

graph

is an

that a

the

vertex v

that High(Wj) is

v such

of

u>,

then there are no

w, exists,

vertex

the

(v) = r.

computed all the High

only if there is a child

such

whether one

can easily compute all

= v, and High(w2)
High(w2)
we

have

is that it traverses

DFS

= rJ

High(wl)

to q,

goes
we

(v) if we know
all the back
among

High

compute

all High{wt) and

among
to determine
efficiently

highest

shortly

So, if we

than another.)
is higher
For
values.
example, in
highest

We can easily

w {, u>2, \342\200\242\342\200\242\342\200\242,
wk (see Fig. 7.27).

the

articulation
order

right

not

in the

vertices

point.
for

our

purposes.)

Computing

the

induction

following

High

goes hand

values

we visit the kth vertex

Induction hypothesis: When
to find the High values of vertices that have already
below

this

of the induction
follows
a
perform (recursively) DFSfor all
whether

compute

High(y)

a vertex

the

DFS,

by

we know

been visited

how

and are

vertex.

The order
and

according to

the DFS,

with

hand

in

hypothesis.

is an articulation

Figure

children

to

according

of DFS.

the order
the

of v,

find

(by

At

definition.

we

When

induction)

the same

point.

7.27

Computing

the

High

values.

a vertex v, we
their High values,

reach

time, we can

decide

7.9

223

of Graphs

Decompositions

DFS tree presents a special case. Obviously,

no High
value can
see
is
to
that
the
root
an
if
articulation
and
easy
point
only if it has
more than one child in the DFS tree. Of course, this is easy to determine.
The key to the efficiency of the algorithm
for computing
the High values is that all
is
information
available
when
DFS
is
we
the necessary
performed. The only problem
in the DFS tree. We use
than
another
have is how to decide whether one vertex is higher
All the vertices involved in the computation
of
DFS numbers to make this determination.
in the
values
are ancestors
tree.
the High
Therefore,
they already have a DFSnumber.
lower
the higher an ancestor is, the
its number is! This is not true for
Furthermore,
in the tree; fortunately,
we care only about
vertices that are not related
back
however,
is
a
to
the
to
DFS
values
use
the
numbers.
We
So, practical way
High
manipulate
edges.
it
as
that
refers
not
to
the
vertex
but
to
that
define High(v)
before,
itself,
except
highest
to describe the algorithm
in terms of DFS numbers,
vertex's DFS number. It is confusing
vertices
because
we define
Therefore,
correspond to lower DFS numbers.
higher
of
is
| V | and the number
decreasing DFS numbers: the root has a DFS number
time
we
visit
a
new
vertex.
We
also
can
use
decreased
numbers:
we
every
negative
and we decrement the number
a
of-1,
assign the root a DFS number
every time we visit
new vertex.
The advantage of the latter scheme is that the value of | V | need not be
known in advance.
the actual
The only remaining task is to find
biconnected
components. We could
It is

root.

the

by brute

them

find

of the

root

The

exceed

Noticethat,

at the

force,

edges of B were the most

vertices
on a stack and add
be an
stack
now
program
defined

to be

the edges

constant

|). The
be remembered
| + |E

that b is

traversed.

the

During

an

7.25.

back at Fig.

look

Let's

way.

articulation

we put

traversal,

the

point,

new

the

as

are

they

encountered.

per

space
as they

amount

vertex.

of

the

Hence,

also

are

requirements

in addition to

work,

running

O (|

V | + |E

work

the

this

of

time

|) since

the

involved

in

algorithm

is

components

traversed.

are

7.3

\342\226\241

Example

An example

of

Fig. 7.29. The first

Each
DFS Numbers.

in

(decreasing)
when

is discovered

a new

to be

for

Biconnected^Components

algorithm

repeated here, is given

gives their
as updated

the extra

Clearly,

the DFS, is

it

recent

elegant

determined

algorithm

fully.)

Complexity

must

the

to
When a vertex is found
all the top edges going back in the
we can remove from the stack
articulation point,
We can
until that vertex is reached. This is exactly
the biconnected
component!
in the
and continue
same way. The complete
remove
those edges from the graph
is given in Fig. 7.28.
for
biconnected
(The algorithm can be
components
in
for completeness, we
terms
of
and
of
DFS,
but,
preWORK
postWORK
merely

presentit

O (|

is also an

there

but

point where

an

call to

the

articulation

recursive

point.

line

gives

the graph in
the vertices

Fig.
and

successive line presentsthe

procedure

is made.

vertex

7.25,

High

is

which

second

the

line

numbers

is circled

when

\342\226\2

Graph

Algorithms

(G,
Algorithm BiconnectedjComponents
connected
G = (V, E) (an undirected
Input:
the

root

Output:

n)

v,

graph), v (a vertex serving as

DFS tree), and n (the number of vertices in G).
and the High values are
the biconnected components are marked
of the

computed.
begin

for

vertex vofGdo

every

v.DFS Number
{the DFS numbers

serve to

also

indicate whether

vertices have been visited

corresponding
DFSN

0 ;

:=
will

not the

or

:= n ;

DFSnumbers;
{ we use decreasing

the explanation

see

in

the

text.}

BC(v)

end

procedure BC(v) ;
begin

:= DFSN

v.DFSNumber

DFSN := DFSN -1 ;
v into Stack ; { Stackis initially
insert
v.High := v.DFSNumber ; { initial
for all edges (v, w) do
insert (v, w)

into

Stack

twice (for
{ each edge will be inserted
is
not
the
then
ifw
parent
ofv

if w.DFS Number
BC( w) ;

if w.High
{v

= 0

disconnects

wfrom

and mark

reached,

{
v.High

else

ofw's

:= max

{ (v, w)
v.High :=

of the

the

is a

until

Stack

back edge

, w.High

Algorithm

is

or a forward
, w.DFSNumber

edge

end

Figure 7.28

they form

subgraph

component and possiblyothers

( v.High

max ( v.High

graph

from

biconnectedcomponent
back into Stack ;

is part

directions)}

then
the rest

edges and vertices

remove all

insert

both

then

< v.DFSNumber

as a

empty
value

Biconnected

Components.

7.9

Decompositions

of Graphs

J
9e

16

15

14

13

16

16

16

15

16

15

16

15

16

15

16

15

16

15

16

15

16

15

16

15

00

16

15

16

15

16

15

16

15

16

15

16

15

16

15

16

15

16

15

14
14
14
14
14
14
15
15
15
15
15
15
15
15
15
15
15
15

16

15

16

15

16

15

16

15

15
15
15
15
15
15
15
15
15
15
15

7.29

An

example

00
b
a
m

16
16
16
16

15
16
15
15

16

15

16

15

00

16

15

\302\251

Figure

e
12

11

10

J
7

15

CO

wl

wk

13

13

15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15

15
15
15
15
15
15
15
14
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15
15

14
14
14
14

15

14

15

13

15

14
14
14
14
14
14
14
14

15

14
14
14
14
14
14
14
14
14
14
14

of computing

16

15

16

15

16

15

16

15

16

15

16

15

16

15

16

15
15
15
15

16
16
16
16

8
8
8
8

8
8
8
8

8
8
8
8

15

16

16

15

16

16

15

16

16

15

16

16

16

15

16

16

16

15

16

16

16

16

15

16

16

16

16

High values and

biconnected

components.

Graph Algorithms

226

7.9.2 Strongly Connected

In

we discuss

section,

this

if, for every pair

Definition:A
vertices such
no subset that
that,

Notice

as a set
connected
An

in a

components

in the

edge

components.

We

are similar to

the

(we

can

vertices

A circuit is

contained

and a

only

path

necessarily

a circuit.

in

Proof: If
combining

contained

into

may

the

strongly

one component.
two separate

connect
two

which

claims,

connected component if and only

(Recall that a circuit is a
not necessarily simple; that is, it may include a
of them.

is a simple

itself

circuit.)

connected.

strongly

of the

vertices of a circuit,

vertices of the

the same strongly

By the definition

to the

circuit

connected
of

component).

component

not be

it would

component,

connectivity,

strong

connected

strongly

since

Now,

maximal

given

we claim

that

there is a path
results
in a circuit

any

two

they

are

v to w

from

(but

not

\342\226\2

7.12

Each vertex belongs to

are

it

by the following
section.
previous

Putting together these two paths

since the paths
not be vertex disjoint).
may

a cycle,

\342\226\241
Lemma

there

connected component is defined

w to v.

from
in

by

a subset

w from

and

graph).

be partitioned
vertex belongs to exactly

in

of the

there is

strongly

both

is

that

add all the other

v

(i.e.,

can

graph

partition

case

same

once. A cycle

than

more

include

subset

7.11

closed directedpath
Proof:

the

component

Two vertices belongto the

is a circuit containing
if there

cannot

of

maximal

component, or

to one

belong

the existence

prove

Each

way.

unique

biconnected

Lemma
\342\226\241

vertex

of every

w to v.

from

path

connected
connected

a strongly

components,

may

graph

is a

strongly

induces a strongly

vertices

The

vertices.

and a

any other vertex.

component

is

connected

is strongly

graph

v to w

from

from

subgraph

it and

contains
biconnected

unlike

of

induced

its

a path

vertex

connected

strongly
that

A directed

graphs.

u\\ there is

v and

possible to reach any

it is

words,

In other

directed

only

of vertices

Components

a vertex

circuits
those

in only

v belongs

containing
circuits

one strongly

results

connected component.

one strongly

exactly

to more
and
in

connected

than

vertices
another

one

from
circuit,

component.

strongly

connected

component,

then

the other components. However,

be
which, by Lemma 7.11, must
\342\226\2
This is a contradiction.

We can define the strongly connected

to the
similarly
component (SCC)graph
biconnected component tree. (This
is also called a condensation graph.) The
graph
nodes
of the SCC graph
call them nodes to distinguish
the original
them
from
(we
vertices) correspondto the strongly connected components; there is a directed edge from
node a to node b if there is a directededge(in the original graph) from any vertex in the
to a to any
to b.
vertex
in the component that corresponds
component that corresponds
since
The SCC graph
is acyclic
involve
cycles cannot
more than one component. Figure
a directed graph G and its SCC graph.
7.30
presents

7.9 Decompositions

Figure7.30A
As was

directed

biconnected

with

case

the

and its

graph

connected

strongly

graph.

component

we can

components,

of Graphs 227

design an

by

algorithm

induction.

Induction

hypothesis: We
of graphs with

components

how

know
<

m edges,

to find

and

the

connected

strongly

construct

to

how

their

SCC

graphs.

The basecaseis
remove x from
now

have

case is

and find,

graph

to determine

what

effect

adding

and pick

m edges

with

x would
the

have on the

same

an

arbitrary

edge

x. We

connected components.

the strongly

induction,

by

two vertices from

x connects

when

a graph

Consider

trivial.
the

Again,

partition.

In this

component.

We

the easy

case,

adding

is when x connects
the SCC graph. The difficult
case
partition
in Fig. 7.31, in which
an
verticesfrom two different components. This case is illustrated
x
is
in the SCC graph
of Fig. 7.30.
edge
Clearly, x merges
connecting two components
has no

effect on

these two

if and

components

this case, all

one

or on

the

the

components

and we

component,

no changes are made

we can improve
this
DFS

plays

a major

Let's
algorithm,

try

and

to

are

only

if

it completes

corresponding
done. If x does

the component.

to

nodes

cycle
the

in

in

cycle

SCC graph.
are combined

In

the

into

not complete a cycle in the SCC graph, then

As was the case with biconnected components,

by considering

algorithm

a (directed)

to the

the

in a

edges

particular order.

Again,

role.
the same steps as we
them when necessary. When

follow

modify

want to determine whether

than
vertices that are higher

it

of a circuit
the DFS tree. The

is part

it in

did
we
with
notion

the biconnected component

a vertex through
we
DFS,
other vertices \342\200\224
in particular,

in

visit

of High

values can

be used in

Algorithms

Graph

228

Figure 7.31

Adding

a similar way.
of the graph

from

can

the

We

identify

an

edge

connecting

two

different

are looking for vertices such

them or from their
descendants.

\"breakpoints\"

in a similar

way

is no

there

that

We
to

connected

strongly

need a

to

way

components.

the articulation

other parts

reach

mechanism

by

which

points. Consider

we
the

connected
of the tree
parts
strongly connected components occupy
in
That
all
the
a
connected
vertices
must
is,
(Exercise7.88).
component
strongly
belong
to one connected subtree of the DFS tree.
For a given component, consider its highest
in the tree; we call this vertex
vertex
the root of the component. The root
is the first
vertex of the component
to be visited by the DFS.
(For example, the roots in Fig. 7.30
are a, d, #, and /.) If we can identify
the
roots
similarly to the way we identified
articulation points,
to
then
we can find the partition.
We will see that the roots are similar
DFS

The

tree.

articulation

The

points.
algorithm

is based

on induction

that

follows

the

order of

DFS. Let r be the

component visited in its entirety by the DFS. It is the lowest leftmost

component in the usual picture of DFS (r -d in Fig. 7.30). The component must consist
of all of r's descendantsin the tree (none of the descendants
can belong to a smaller
If,
traversal
would have been completed first).
component, since that component's
the DFS, we can identify
r as the first root, then
we can identify the component,
during
of the first

root

removeit

the graph, and continue

This is not as simple as we stated
by induction.
the main idea. Let's first see if we can identify
r.
for
a
vertex
r
there
cannot be any back edges
to be a root of a component,
First,
a
from
a
descendantof r to a vertex higher than r. Such a back edge completes
leading
with
same
the higher
to the
vertex
which
cycle
vertex,
belongs
implies that the higher
to the
whether such back edges exist similarly
component as r. We can determine
\342\200\224
biconnected
the High values. However, we need to be more
using
component case
careful here since DFS in directed
does not eliminate cross edges. Consider
Fig.
graphs
is
7.32.
Vertex g does not have any back edges, but it has a cross edge to e, which
a
vertex
in a cycle with
contained
h. Consequently, g's parent
higher
(/) is not a root of a
is
no
there
even
back
though
component,
edge from any of its descendants. Thus, we
it,

but

from

this is

7.9 Decompositionsof

7.32 The effect of

Figure
must consider

cross

the

words, they must

looking for
found
both

e. If

/ and

before we

take

that

means

to go

way

that

be a

vertex

higher from e.
not a root.
/is

back edge \342\200\224

we need
of the edge from g to e,

only to
is not

it

which

it would

then

the component

as a

the effect

considering

of /,

that

fact

the

must

It

been an ancestor

before /implies

visited

account

into

be /.

it cannot

In particular,
that there is

/.

was

that

not

had

it

reached

discovered yet
a vertex

root.

we claim

then

yet,

as well.

cross edges? Crossedgesmust go from right to left; in other

that have already been visited. Rememberthat we are
vertices
If there is a cross edgefrom
root
has not been
g to e and the

to

point

first

the

edges

edges.

effect of

is the

What

cross

229

Graphs

is an

have been

discovered

e has

containing

not been

So, a crossedgefrom
But

this

is just

important

this

g to

as easy to

consider DFS numbers!

whether

of

ancestor

When
edge

is a

back edge. Only the DFS number of e (and its value relative to that of/) is important.
We can define the High
for an
values
as in the biconnected component case by looking
edge leading to a vertex with the lowest DFS number. The High value of a vertex is the
those of its children
is
and among its back edges or crossedges.A vertex
highest
among
the first root if it is the first vertex whose High
Notice
that
is not higher than
value
itself.
the High values do not really point to the highest
The High value of g will be
vertices.
the DFS number of e, even
it is possible to reach b from
e (and thus from g). We
though
care
vertex
whether
we
reacha
can
than
it
is not important
to know
higher
g (or /);
only
do
the identity
we
vertex.
of the highest
want
to
chase
once
a
back
(Nor
pointers
edge is
encountered.)

Once

consists

component
edges,

and

the

we find

of all the
from

the

all the

first

root,

we can

descendants of
graph.

edges

that

the

This is
point

the

find

done

by

to them

connected component \342\200\224

it
We
can
then remove this
the component's vertices and

first strongly

in the

root

deleting

DFS tree.
all

from other vertices.

We

can

ignore

edges

Graph

230

since there is no way

since we now have

vertices,

other

from

Algorithms

be done by

induction

assumptions are still

values is dynamic. Since we remove
verify

all the

that

to

of

get outside

a smaller

the

graph!

Notice

valid.)

component.

The

reader

should

(The

that the

of

definition

rest can
carefully

the

High

discovered
pointing to the newly
later.
of
the
values
in
the computation
(This is
High
they will play no part
component,
values for the biconnected component
the \"static\" definition of the High
from
different
there is no
case, which did not depend on any of the previous components.) In practice,
vertices or edges. We can simply mark the vertices of
remove
either
need to actually
to marked
later
on ignore edges pointing
as they are discovered, and
each
component
is
in
7.33
connected
The
vertices.
(we use
algorithm
given
Fig.
component
strongly

decreasing DFS numbers

The algorithm

Complexity
is

complexity

to avoid

again

the

The

same.

the

edges

confusion).

to the biconnected componentalgorithm

space complexitiesare O (| V | + | E |).

is similar

time and

An example of algorithm
StronglyjConnectedComponents
The
in Fig. 7.34.
which is repeated here, is given
second

line their

In

it

this

to be

of the

call to

whether

determine

seen

that

a cycle

Given

for

procedure
connected

Use of Graph

must be

line presents
is made. A

the

High

is

vertex

\342\226\2

component.

Decomposition

where
problem

the use
involves

undirected

connected

the graph in Fig. 7.32,

the vertices and the

gives

successive

recursive

a root of a strongly

it contains

line

first

Each
the

section,

The Problem

We have

a new

we present two examples

simplifies the solutions. The first
one involves directed graphs.

short

second

when

is discovered

Examples

significantly
the

(decreasing) DFS numbers.

as updated

circled when

7.9.3

its

7.4

\342\226\241
Example

numbers

and

a cycle of

even

contained

in

of

graph

decomposition

undirected

graph G=(V,

graphs

and

\302\243),

length.

Hence, we
component.
then consider each
components,
that the graph is biconnected!
assume
at least
it contains
than one edge, then
a biconnected

can first partition

the graph
into its biconnected
component separately. In other words, we can now
If the graph is biconnected and it contains
more
one cycle (in fact, every two edges are containedin a cycle). Let's find an arbitrary cycle
C l = v,, v2,..., v*, v,. If A: is even, we are done. If there are no more edges \342\200\224
is,
that
the graph consists of exactly
one
odd cycle \342\200\224
then
the answer is obviously negative.
Otherwise,there is an edge not in the cycle such that one of its vertices is in the cycle.
Let that edge be (v,, u>). Since the graph is biconnected, the edges (v,, w) and (v,, v,+i)
are contained in another
at w until we meet C{ again
cycle C2. We traverse C2 starting

7.9

Input:
of

(G, v, n)

Strongly_Connected_Components

Algorithm

G =(V, E) (a directed
the DFS tree), and n
the

marking

Output:
the

of vertices

number
connected

strongly

vertex serving as the

v (a

graph),

(the

of Graphs

Decompositions

in

components,

root

G).

and computing

values.

High

the case with directed

{ As is always
have to be called several
times
until

this procedure may

DFS,

all vertices

have been visited.

begin

for

every vertex

vofGdo

v.DFS Number

:=

Current

{ we
while

:= 0

Component

DFS_N := n

0 ;

:= 0

v.Component

use decreasing DFS numbers;

there
exists a vertex v such

see

that

the explanation
v.DFS Number

in Section

7.9.1.}

= 0 do

SCC(v)

end
SCC(v);

procedure

begin

:= DFS_N

v.DFSNumber

:= DFS_N

DFSN
insert

v into
:=

v.High

1 ;

STACK ;

v.DFS

Number

for all edges (v, w) do

if w.DFS Number =

; {the

initial

value

0 then

SCC(w);

:= max

v.High

( v.High

, w.High

else
> v.DFSNumber
if w.DFS Number
{ (v, w) is a cross edgeora back

, w.DFS Number
) ;
=
v
is
root
a
v.DFS
Number
then
a
v.High
{
of component}
Current
:= Current Component + 1 ;
Component
{ mark the vertices of the new component}
repeat
remove xfrom the top of STACK ;
v.High

if

edge

and w.Component = 0 then

that we need to consider }

:=

max

( v.High

x.Component := Current
until

x =

Component;

end

Figure

7.33

Algorithm

Strongly

Connected

^Components.

<\302\273\"

>

a>

&)

61

0-

-\302\251

o a. a a. o

3
\"2.
cd*

\342\200\242-h

O
O

3
\342\200\242a

oooooooooooooooooooo

ooooooooooooooo

sO

5'
CFQ

oooooooooooooooo^

3Q
IS\"

ooooooooooooooooo

<
CD
-J

&3

-J

-J

^J

-\302\253J
-J

a.
~~-&>-^-&>*\302\273>.P>*\302\273>

OS.

3CD
o

cd

a.

\342\200\242a

CD

\342\200\224
\342\200\224
z
z

^J

-J

^J

^J

sO

sO

sO

a*

o\302\273

7.9

at,

of Graphs

Decompositions

233

7.35). Clearly, v, ^ v7. The path v,, u>,..., w, v7 defines two cycles, as is
It is easy to see that one of the three cycles in the figure must be
the following theorem.
proved

v7 (see Fig

say,

in

shown

even.

7.35.

Fig.
have

We

7.13

Theorem
\342\226\241

Every biconnected graph that

odd-length cycle contains an
second problem is

The

whether
it

a (directed)

contains

that a

we know

Again,

similar

one,

cycle

but for

directedgraph
cycle of odd length.

be

must

edge and is not

merely

an

cycle.

even-length

Given a

Problem

The

than one

more

has

\342\226\

directed graphs.

G =

(V,

in a

contained

determine
\302\243),

connected

strongly

so

component,

We perform DFS starting

an arbitrary vertex r and
we mark
r as
from
vertices with either
even or odd. We mark
r
for
each
we
mark
w
with
of
v.
Since
can
the
mark
be
even, then,
edge (v, u>),
opposite
we
claim
that
there
reached
from
vertex
the
is
a
any
(by
strong-connectivity
assumption),
if and only if we try to mark
a vertex that is already marked by the
cycle of odd length
opposite mark (the most notable example is if we reach r again and try to mark it as odd).
We leave the proof
of this fact to the reader.
It is strongly dependent on the
strong
we

of these

Both

worthwhile
in

questions

to

are

problems

Since both

decomposition.
usually

graph

is strongly

connected.

assumption.

connectivity

the graphs

that the

assume

well

as

might

start

more

much

thinking

either

are

about

a given

biconnected or

to

difficult

decompositions can be achieved

efficiently

problem

with

strongly

connected.

the

v*

vi+\\

Figure

7.35

Finding

an

even-length

cycle.

solve

without

in linear
extra

time,

the
it

is

that

assumption
is especially

This

Graph Algorithms

234

for

true

that involve

problems

determining

efficiently

open (see the

7.10

Bibliography

cycles.

It

a directed

whether

is

to note

interesting

graph contains

an

the

that

problem
cycle

even-length

of

is still

section).

Matching

= (V, \302\243),
is a set of edgesno two of which have
a matching
Given an undirected
graph G
is that an edge can be thought
of as a
The reason for the name
in common.
a vertex
We insist that no vertex belongs to more than one edge from
match of its two vertices.
A vertex
to any
that is not incident
so that it is a monogamous matching.
the matching
We
also
that
the
does
not
is
called
unmatched.
vertex
the
in
matching
say
belong
edge
are matched. A
is one in which
all vertices
A
to the matching.
perfect
matching
is one with the maximum
number of edges. A maximal
maximum
matching
matching,
is a matching
of an edge.
that cannot be extended by the addition
on the other hand,
occur in many situations (besides social). Workers
be
Problems involving
matching
may
matched to jobs, machines to parts, and so on. Furthermore,
many problems that seem
in terms of matching
formulations
to matching have equivalent
unrelated
problems.
In this section, we limit
in
is
a
difficult
our
general graphs
Matching
problem.
The
first
to two specific matching
is
so
not
it
discussion
problems.
problem
important;
in special very dense graphs. The solution
involves finding
to this
matchings
perfect
an interesting
problem, however, illustrates
an important problem concerningmatching

Perfect

7.10.1
In

this

Matching

we consider

example,

a very

approach,
in bipartite

in Very Dense
restricted

case

which we

then

to solve

generalize

graphs.

Graphs

of the

G = (V,

perfect matching

Let

problem.

is at
and the degree of each vertex
E) be an undirected
graph such that | V \\ -In
least n. We present an algorithm
As a
in such
to find a perfect matching
graphs.
exists.
corollary,we show that, under these conditions, a perfect matching
always
We use induction
on the size m of the
The base case, m = 1, is handled
matching.
of size one. We will
that we can extend
show
by taking
any arbitrary edge as a matching
any

edge
result

with

two

perfect either by adding

edges. In either case, the

is not

that

matching

new

another
size

of the

edge or

by

replacing

an existing

matching is increased,

and

the

follows.

Consider a

matching

in G

with

m edges

such that m < n.

We

first

check

all the

whether
If we find such an edge,
edges
any of them can be added to M.
then we are done. Otherwise,
there are
M is a maximal matching. Since M is not perfect,
at least two nonadjacent
vertices,
v, and v2, that do not belong to M. These two vertices
have at least In distinct
out of them. All of these edges lead to vertices
edges
coming
that
are covered
an edge
could be added to M. Sincethe
by M, since otherwisesuch
number
of edges in M is < n and there are 2n edges from
v, and v2 adjacent to them, at
\342\200\224
least
one edge from M \342\200\224
is adjacent
to three edges from
(\302\253i,
w2)
say
v, and v2.
of generality,
that those three edges are (\302\253,,v,), (\302\253i,v2), and
loss
Assume, without
the edge (\302\253,,w2) from M
(w2, v,) (see Fig. 7.36(a)). It is easy to see that, by removing
not

in M

to see

7.10

235

Matching

v,

----.Jw'

^#

\342\200\242^

fa)

7.36

Figure

a matching.

Extending

v i), we get a
this algorithm

and adding the two edges (w,, v2),

We leave the
implementation

and

algorithm is

greedy approach. At most

to a larger one.
matching

in

each
but,

may

affect

of a

example

extension of one
is
general,
finding a good matching
choices of other edges far away
in
this approach to other matching
problems.

in

generalize
7.10.2

\302\243
is a

The

\302\243,
U)

set of

larger matching
as an exercise

This was

the graph.

This

(7.21).

edges were involved

three

difficult.

more

7.36(b)).

(Fig.

in this

sufficient

A choice of
Next, we show

one

edge

to

how

Matching

Bipartite

Let G = (V,
and

of

in the

step

case,

another

(m2,

be a

bipartite

that

such

graph,

edges connecting

V and U
V to

from

vertices

Find a maximum-cardinality

Problem

are two

vertices

in

disjoint

sets of

vertices,

U.

in

matching

a bipartite

graph G.

in terms
of real matching: V is a set of girls, U is a set of
of
we want
to match boys to girls so as to
boys,
\"possible\" pairings;
maximize
the number of matched boys and girls.
A straightforward
no
until
approach is to try to match according to some strategy
more matches are possible,in the hope that the strategy
will
guarantee
optimality, or at
least
come
close. We can try different
For example, we can try a greedy
strategies.
with small degrees, hoping
that the other vertices
approach by first matching the vertices
will
be more
later
on. (In other words, first match
the
likely to have unmatched partners
boys that are the most difficult to match, and worry about the rest later.) Instead of trying
We

this problem

formulate

can

and

\302\243
is

a set

problem.

such strategies
Suppose that we

maximum

matching.

to

analyze

Can

(which

we

is

hard),

we

try

the

approach

maximal matching,
somehow improve it? Consider

start

with a

which

Fig.

used

in

is not

7.37(a),

the

previous

necessarily a
in

which

the

236

Graph Algorithms

matching

replacing
applied
edges.

in

is depicted
by bold
2A with the
edge

the

the

If we

is
and

clear
2B.

can improve
similar to the

that we
This is

the

matching

by
we

transformation

to replacing one edge with

But we are not restricted
problem.
situation where k edges can be replacedby k + 1 edges,

previous
find

lines. It
edges 1A

a similar

two

then

the
we can improve
further by
matching
5E
and
edges iC, 4D,
(Fig. 737(b)).
replacing the edges
these
Our goal is to add more matched vertices. We
transformations.
Let's study
a match
for it. If we already have a
with an unmatched vertex v and try to find
start
so we must try to break
then all of v's neighbors are already
matched,
maximal
matching,
which
We
choose
to
was
another
vertex
match.
w,
v,
a
adjacent
previously matched
up
v
match
to
break
the
match
between
u
and
w.
We now have to
We
u
and
w.
to, say,
up
w
If
w.
is
to
an
unmatched
then
we
are
for
connected
done (this was
match
a
find
vertex,
if
this
matches
we
can
continue
and trying
the first case above);
not,
way
by breaking
we have to do two things.
into
an algorithm,
To translate this attempt
rematches.
First,
we have to make sure that this procedure
terminates, and second, we have to show that, if
then this procedure will find it. First, we formalize this idea.
there is an improvement,
M is a path
from
a vertex v in V to a
An alternating
path P for a given
matching
in A/, such that
are
the
unmatched
of P are
vertex u in U, both of which
edges
E
-M
in
and
in
the
first
P
M.
That
of
does
not
to M
is,
(v, w)
alternatively
edge
belong
v
not
to
the
to
does
second
and
so
until
the
on,
A/),
(since
belong
edge (w, x) belongs A/,
last edge of />, (z, w), which
Notice
that alternating
does not belong to A/.
paths are
The
number
of edges in P must be
exactlywhat we used already to improve a matching.
odd since P starts in V and ends in U. Furthermore,
there is exactly one more
of P
edge
in E-M
than there is in A/. Therefore,
if we replace all the edges of P that belong
to M
by the edges that do not belong to A/, we get another matching with one more edge. For
the matching
in Fig. 7.37(a) was
example, the first alternating
path we used to improve
which
was
used to replace the edge A2 with the edges 1A and 2B\\ the
(M, A2, 2\302\243),
second alternating
was (C5, 3D, D4, 4E, \302\2435),which
was used to replace the edges
path
3D and 4E with
the edges
C5, D4, and E5.

we have

an

For

improvement.

3D and

12

4E

with

example,
the

12

(b)

(a)

Figure7.37Extending

a bipartite

matching.

7.10

It

then

now

be clear

should

M is

not maximum.

It

out that

A matching is
will be

claim

The
matching

a matching.

proved, in
maximum

with the

of a

context

the

theorem

path

is not

only if it has no

if and

maximum

alternating

that

is an alternating path
the opposite is also true.

for

a given

matching

A/,

Theorem

\342\226\241
Alternating-Path

This

if there

that,

turns

237

Matching

has

more generaltheorem,
suggests

immediately
an

path and

alternating

\342\226\2

paths.

alternating

an

any

in the
algorithm,

alternating

next section.
because

any
can
extend
path

edges to the matching

an alternating path,
the
until
no
more
The
and modify
matching accordingly
alternating paths can be found.
alternating
resulting
matching is maximum. Since each
path extends a matching
by one
of vertices),
(where n is the number
edge and there are at most nil edges in any matching
is at most nil. The only
of iterations
is how to find
the number
remaining
problem
We
We
solve
this
as
follows.
transform
the
undirected
problem
alternating
paths.
graph
the edges in M to point from U to V and directing
G' by directing
G to a directedgraph
for
the edges not in M to point from V to U. Figure 7.38(a) shows the matching
obtained
in Fig. 7.37(a),
and Fig. 7.38(b) shows the directed
the graph
graph G'. An alternating
from
an unmatched
vertex in V to an
exactly to a directed path
path
corresponds
can
unmatched vertex in U. Such a directed path
be found by any
graph-search
of a search is O (| V \\ + \\ E |); hence, the
for example, DFS. The complexity
procedure,
as possible,

We

start

until

we

complexity of the
An

greedy

We

get a maximal matching.

is O

algorithm

(|

as many

adding

algorithm,

(| V \\ + \\ E

for

search

then

|)).

Improvement

in
Since a search
can traverse
the whole graph
traverses one path, we might as well try to find
We have to make sure, however,
that these paths
the independence of such alternating
guarantee

the

same worst-case

several
do

that

paths

12

(a)

(b)

Figure 7.38

Finding

alternating

it

alternating paths with one search.

one another. One way to
is to restrict them to be vertex

not modify

12

time

running

paths.

Graph Algorithms

238

disjoint. If

the

vertex

are

paths

disjoint,

improved
applied concurrently. The new
First, we perform BFS in G'
following.

different

modify

they

algorithm

the set

so

vertices,

for finding

alternating

of all

unmatched

U are

found.

they

can

be

is

the

paths

in V, level
we
extract
from
Then,
level,
by
in
a
set
of
vertex
G'
the
BFS
maximal
induced
(which are
the
disjoint paths
by
graph
in G). This is done by finding
any path, removing its vertices,
alternating
paths
finding
is not a maximum set, but
and so on. (The result
another path, removing its vertices,
set in order to maximize the
number
a maximal
of
set.) We choosea maximal
merely
with
one
search
to
the
added
adds
(each
vertex-disjoint alternating
matching
path
edges
we modify the matching
one edge to the matching).
Finally,
using this set of alternating
no more alternating
the new
This process is repeated until
(i.e.,
paths can be found
paths.
in
V
from
the
the unmatched vertices
unmatched vertices
G' disconnects
directed graph
a level

until

vertices

unmatched

which

in

from

in

vertices

inU).

Complexity
0(v\\V\\)
[1973].

The overall

7.11

Network

The

Furthermore,
E)

G=(V,

indegree
weight

0,

that

1.

of

/ (the

an

edge.

0 to

satisfies

the

proof,

which

is thus

O ((|

+ \\ E

\\

|).

| )V|V

of

the

two

capacity of e.
We

call such

The

flow

through

is a

the amount

function / on

an edge cannot

the

with

source)

(the

an associated

exceed

the

total

flow

that

can

we assign a
the network

of

edges

positive

of flow

For convenience

a network.

graph

nonexisting edges. A flow

following two conditions:

0<f(e)<c(e):

measures

E has

network-flow

as follows. Let

is defined

vertices, s

distinguished
capacity

posed as

can be
problem

extensions.

and

variations

0. Each edgee in

The

35 years,
many

problems

network-flow

and combinatorial
and many
algorithms

theory

graph

last

the

It has

it.

for

with

graph

in

extensively for

sink) with outdegree

called the

pass through
capacity

directed

be a

of the improved
algorithm is
is due to Hopcroft and Karp

iterations

basic problem

unrelated

seemingly

basic variation

and

c (e),

flows is a

been studied

many

The

time

have been developed

data structures

problems.

the

Flows

It has

optimization.

and

worst-case running

network

of

problem

We omit

case.

worst

of

that the number

out

turns

It

the

in

of

capacity

that edge.

2.

For all veV-{s,

vertex

is equal to

/),
the

\302\243/(m,
u

total

flow

v) = \302\243/(v,
w

exiting

w):

The

this vertex

(exceptfor

the

entering a
source

and

sink).

These two conditions

s is equal to the total
flow
imply that the total
leaving
t. The problem is to maximize
this flow.
(If the capacities are real numbers,
even clear that maximum
flows exist; we will
that they indeed always
show
is to think of the network
as a network of water
way to visualize this problem
goal is to push
to the wrong

as
area,

much

water through

the pipes

will

burst.

the

pipes

as possible.

If too much

flow

water

then

entering
it is not

exist.)
pipes.

One
The

is pushed

7.11

be

how

to solve

The

reason

posed

problem
the

Given a

problem of

network-flow

problem

the

(namely,

similarities

matching

the

is in

reduction

the wrong

direction).

the techniques for solving

is that
the

solving

239

in the previous
to
seem
be a fruitless
may
but
we
do not know
problem,

This

problem.

to solve

how

Flows

discussed

matching,

bipartite

this wrong-order reduction

are similar to those for

present

Understanding

know

network-flow

the

we

network-flow

as a

we already

since

exercise,

that the

show

we

First,

section, can

Network

in understanding
can be helpful
in which we
G = (V, \302\243,
U)

the

matching problem.
network-flow
algorithms.

bipartite

want
to find
a maximumtwo
s
new
vertices
connect
s
to
all
vertices
in V, and
and
f,
cardinality
connect all vertices in U to t. We also direct all the edges in E from V to U (see Fig.
7.39, in which all edges are directed from left to right). We now assign capacities of 1 to
on the modified graph
and we have a valid
network-flow
the edges,
G'.
Let
all
problem
in G. There is a natural
M and a flow in G\\
between
M be a matching
correspondence
of 1 to all the edges in M and to all the edges
s or Mo
We assign a flow
connecting
in M. All the other edges are assigneda flow
of 0. The total flow
vertices
is
matched
of edges in the matching.
It turns out that M is a maximum
thus equal to the number
if and only if the corresponding flow
is a maximum
flow in G\\ One side is
matching
If
flow
is
to
a
the
maximum
and
it
clear:
matching, then we cannot have a
corresponds
since
it would correspond to a larger flow.
For
the other side of the
larger matching,
have to adapt the idea of alternating paths to network
and to
claim
we somehow
flows,
flow
is
maximum.
We
show that, if there are no alternating
then
the
paths,
corresponding
to
that.
do
proceed
just
from
s to t
An augmenting
to a given flow
/ is a directed path
path with respect
which consists of edgesfrom G, but not necessarily in the same direction; each of these
edges(v, u) satisfies exactly one of the following two conditions:

we add

matching,

1.

(v, u) is

in

Reducing

left

same

the

case, the edge

Figure 7.39
edges are from

graph

bipartite

(v,

direction

u) is called

bipartite

to right).

as it is
a forward

matching

in G,

and /(v,

edge.

to network

flow

forward

(the

w)<c(v,
edge

w).

In

this

has room for

directions

of all the

240

Graph

The difference c(v, w)-/(v, u) is calledthe slack of the edge.

and /(w,
the
v) > 0.
(v,
opposite direction in G (namely, (w, v)e \302\243),
to
this
In
case, the edge (v, u) is called a backwardedge. It is possible
borrow some flow from a backward edge.
flow.

more

2.

Algorithms

is in

u)

of

and they serve the

same
paths,
for
do
If
there
exists
alternating
bipartite
matching.
paths
purpose
an augmenting path),
an
path with respect to a flow / (we say that / admits
augmenting
can
more
flow
We
maximum.
not
then
moving
modify
/by
through the augmenting
/is
If
all
of
the
the
are
forward
edges
path
edges, then more flow
way.
path in the following
and all the constraints
are still satisfied.
The extra flow in
them,
can be moved through
slack of the edges in the path.
The case of backward
that case is exactly the minimum
are

paths

Augmenting

for network

flows

extensions

alternating

as

with two
Fig. 7.40. Each edgeis marked
the current flow.
It is clear that no more
from
s to t that consists of only
be pushed forward, since there is no path
can
flow
is
a
to
flow.
there
extend
the
forward
However,
way
edges.
An additional
is an augmenting path.
flow of 2 can reach
The path s-v-u-w-t
u
slack
over all forward edges until
from s through this path (2 is the minimum
w). We can
is now satisfied for w, since
deduct a flow of 2 from / (w, w). The conservationconstraint
in through
of 2 coming
the augmenting
u had an additional flow
path, and a flow of 2
deducted from the backward edge. We now have an extra flow of 2 at w that needs to be
can continue pushing flow
We
from
w in the
pushed, which is exactly what we want.
same way, pushing
it forward on forward
and deducting
it from backward edges.
edges,
In this
case, there is one forward
(w, t) that reaches f, and we are done. Since only
edge
forward
to
is equal
r, the total flow is increased. The increase
edges can leave s and enter
the minimum of either the minimal
flow
slack of forward edges or the minimal
current
7.41
shows
the same network
with
the modified flow.
through backward edges. Figure

edges is a

little

more

complicated.

numbers alb, such that

(This flow

is in

a is

Consider

the capacity

and

b is

fact maximum.)

Figure 7.40

An

example

of a

network

with

a (nonmaximum)

flow.

Network Flows

7.11

Figure7.41The
The
not

The

The
\342\226\241
A

path,

is a

a cut
V such

sum

and prove

set of edgesthat
e A and 14

of the

w) e E]

capacities of

if it

only

not

augmenting

the flow

then

path,

is

edges.

v e

path.

augmenting

of the

direction

theorem \342\200\224
if the
now

assume

Let's

maximum.

that

a flow

flow admits
/admits

no

maximum. We use the concept of cuts. Intuitively,

s from t. More precisely,let A be a set of verticesof

that/is

that

admits no

one

shown

separate

its

is an

Fig. 7.40.

Theorem

A. Denote

such

there

if

of

flow

is also true:

maximum if and

that s

of edges {(v,

that

establish

We have already
path, then it is

augmenting

augmenting

opposite

Augmenting-Path

f is

flow

Proof:
an

above

arguments

maximum.

the

of augmenting

result

241

the

rest

A and
It is

of the

vertices

we B. The

clear

that

no

capacity

flow

by

B =

V -A.

of the

can exceed

cut

is the

cut is defined as
the

capacity

set
the

of any

cut. (If you disconnect

the pipes, no water
can
flow through them.) Hence, if we find a
flow whose value is equal to the capacity
this flow must be maximum.
of a (any) cut, then
We
admits
no augmenting
to prove that, if a flow
paths, then it is equal to the
proceed
it is maximum.
of a cut, and hence
capacity
Let / be a flow that admits no augmenting path.
Let A c V be the set of vertices
such that for each v e A there is an augmenting
with
path,
respect to the flow / from s to
v.
and
that
no augmenting
we
assumed
seA,
t$A (since
Clearly,
/admits
path).
a cut.
We claim that,
for
all edges
Therefore, A defines
(v, w) in that cut,
and there would be an
/(v, w) = c(v, w). Otherwise,
(v, w) would be a forward
edge
to our assumption that
w $ A. By the same argument,
augmenting
path to u>, contrary
there
cannot
be an edge (w, v) such that w$ A and v e A, and/ (w, v) > 0 (sinceit would
extend an augmenting path). Hence, the value
be a backward edge and
it could
of the
of the cut defined by A, and /is maximum.
flow /is equal to the capacity
\342\226\

Algorithms

Graph

242

We have provedthe

Theorem

Min-Cut

Max-Flow
\342\226\241

The value of a maximum

of a cut.
The

The
\342\226\241

//

the

theorem

implies

maximum

matching,

augmenting path
there is a maximum
vertex

the

to the

the

is a

there

then

theorem.

augmenting-path

to an integral flow
a flow of 0,

lead

paths

will

since

we

with

start

if

In fact,
all the
each

and

total flow.

\342\226\

Clearly,
versa.

problem.

bipartite-matching

in

path

alternating

any

The augmenting-path
augmenting
path
in the previous
the alternating-path theorem given
section. If M is a
which
then there is no alternating
for
it,
path
implies that there is no
in G', which
is
the
flow
maximum.
On the other hand,
that
implies
to a matching
since
each
flow, and it clearly
integral
corresponds
in

of

flow

by only one
1. The

same

since, if

it

vice

and

G\\

edge (with

only

be maximum

must

matching

obvious

an

connected

V is

in

can support

integer

return to

now

We

G corresponds to

from

directly

is

This

are integers,

integer.

uses only augmenting

an

capacity

theorem.

following

network

the

in

is an

follows

integers.

path adds

augmenting

edges

value

theorem

that

algorithm

capacities are

minimum

Theorem

whose

flow

Proof: The
any

also implies the

of all

capacities

network is equal to the

\342\226\

Integral-Flow

maximum

in a

flow

theorem

augmenting-path

theorem.

fundamental

following

could

of V
s; hence, each vertex
the vertices of U. This
be a larger
then there would

1) to

capacity

for

holds

argument

be extended,

flow.

The augmenting-path
0, searchfor
no more augmenting
flow. Searching for
flow of

residual

theorem

paths,

augmenting

We

paths.
augmenting

with respect to

graph,

and augment

are always making

can be done
paths

network

in an augmenting

the

possible

belongs to F
or a backward
directed

regular

requires

\\E

the

augmenting
it is either

if

edges

possible

flow through

forward

edge,

flow

Their

path.

those edges.
case

More
its

/,

the

precisely,

is c(v,

in which

graph

edge

w)-/(v,
path

the

with

to

correspond
an

case its capacity

w). An augmenting
is/(v,
path from s to t in the residual graph. Constructing
since each edge has to be checkedexactly
once.
steps
edge,

increasing

residual

capacities

capacity

are

We define the
as the network

the

in

with

start

until there

same edges, but

and the

The edges

capacities.

in which

and a

sink,

are

way.

following

E)

G=(V,

correspond to

different

the

in

We

accordingly,

progress since we

the same sourceand

= (V,

flow

the

F) with the same vertices,

different directions and
possibly

an algorithm.

suggests

immediately

residual

(v, w)

w),

is thus

graph

lead to a very
in an arbitrary
way
may
selecting augmenting paths
be a
The worst-case running
time
of such an algorithm may not even
function of the size of the graph. Consider the network
flow
in Fig. 7.42. The maximum
is obviously
a
2Af. However, one might
start
with the path s-a-b-t,
which
can support
one might take the augmenting
1.
flow of only
which again
Then,
path s-b-a-t,
Unfortunately,

slow

algorithm.

Figure 7.42
augments

even

very large,
of M can

case,

the

size of

number

of

(IV 13-

IV

next

has only

be

edges.
I

)/4

polynomial

An

this algorithm is

above may be unlikely,

to minimize the number

and Karp

upper

They

Some

this

We

started

the chapter

by taking

path

proved

chapter
with

Problem

that includes

worst

the

in

precautions to avoid
in

path

to speed

order

an

complexity
not

describe

these

worst

whose

algorithms
network

most

at

then

simple

of

up

things)

with the minimum

algorithm

others are relatively

will

other

maintained,

different

Many

input.

are complicated;

is

policy

have been
are really

(none

been
algorithms here
flow has

Bibliography section).

Tours
with a

a discussion

discussion of a tour
of a

tour

containing

all edges

containing

all the

also a famous problem, named

after
the Irish mathematician
who designed a popular
based on this problem in 1857.
game

The

to take

augmenting

This leads to

required.
the

the

this

if

that,

on the
of 0(|V|3)
of these algorithms.
We

in the

have

we

of augmentations

bound

Hamiltonian

end

exponential,

[1972], for example,suggested(among

are
augmentations
in the
size of

achieved by several
(referencesare given

2M times, where M may be

and five edges. (Since the value

repeated

vertices

four

bits,

O(logM)

augmenting

suggestedsince then.

7.12

processcan

input.)

Edmonds

the

selecting

the

we want

algorithm.

simple).

graph

of network flow.

example

the scenario

Furthermore,

case is

the

though

Although
it.

1. This

by only

be representedby

the

in

flow

the

bad

243

Tours

Hamiltonian

7.12

Given a graph
find
G=(V,
\302\243),
once.
every vertex of V exactly

vertices of

Sir

a simple

of a graph.

a graph.

William

cycle

R.

in

We

This

Hamilton,

is

Graph Algorithms

244

Such a cycle is called a

graphs.

Hamiltonian

only the undirected

consider

the

Unlike

characterizing

interesting

Eulerian-tour

reversed

induction.

7.12.1

Reversed

We

have

(e.g., S =

set S
the

theorem
that

proving

an

algorithms

solve
that

the

do

[2k),

base
on

= 1,

affect

in

In

of finding

this

It

the

Hamiltonian

to

belongs

we present
graphs that are

section,

special

only

is

example

use

as

hand,

base

values of

is much

case

the

it

Section

in

induction

2,...)

from n to

the other

problem

not

we

version;

will

an interesting

of

cycles

(or

of NP-

the class

a simple example
The

dense.

very

called

technique

Induction
k

going

infinite

cycles

this

P(n) holds for all

of
the validity

mathematics,

11.

in Chapter

seen reversed

already

are called

such cycles

undirected

an

is very

graphs)

of

the problem
difficult.

problem,

discussed

part

directed and

version.

find Hamiltonian

we

which

most

containing

Graphs

cycle.

has a

problem

Hamiltonian

problems

complete
in

Hamiltonian

The

case
that

P(n)
implies
- 1 is not easier

more
is almost

difficult

Then, we go
of

validity

than going from

than

can

n-\\ to w,

from

many

that

\"backward,\"

and

in

proving

When designing
namely, to

nton-\\,

introduce
in

infinite

Usually

P(n-\\).

one.

simple

always easy to go

for smaller inputs. For example,we

the outcome. As a result, it is sufficient

use an

is, we prove

That

induction.

to S.

belong

the

The idea is to

2.11.

for the

\"dummy\" inputs
cases to design the

not for inputs of all sizes, but only

for sizes taken from an infinite
set. The
algorithm
most common use of this principle is designing algorithms
for inputs of size n which
only
is a power of 2. It makes
the design much cleaner and eliminates
many \"dirty\" details.
will have to be resolved eventually.
But it is more convenient to
Obviously, these details
solve the main problem first. We use the assumption
that n is a power of 2 in several
the
book
Sections
and
8.2,
algorithms throughout
9.4).
(e.g.,
The same method is also useful
when
on the number
of possible
there is a bound
elements. The base case of the theorem can be the instance with the maximal number of
elements (rather
For
than
the minimal number). The proof can
then
\"go backward.\"
to apply
about
and we want
example, supposethat we want to prove a theorem
graphs
induction on the number
of edges.
has the
We can start with the complete graph, which
maximal number of edges for a fixed number of vertices. We can then prove that the
theorem continues to hold even if we remove an edge (as opposedto the usual adding of
an
This gives us extra flexibility
in applying
The next algorithm
induction.
edge).
illustrates

this principle.

7.12.2

Finding

Hamiltonian Cycles in Very Dense Graphs

of the
let d(y) denote the degree
and
G=(V,
E) be a connected undirected graph,
vertex v. The following
involves
problem
finding Hamiltonian cycles in very dense
is
We will show that the conditions of the problem
that the graph
graphs.
guarantee
the
to illustrate the principle of reversedinduction.
We introduce
Hamiltonian.
problem
Let

Given
The Problem
n > 3 vertices, such
that
satisfies d(v) + d(w)>n,

The

algorithm

is based on

the

complete

graph.

Hamiltonian

and

cycle

connect them

in a

Induction
satisfying

undirected graph G = (V,

of
nonadjacent vertices
pair

a connected

each
find

a Hamiltonian

reversedinduction

Every complete
it is easy to find

cycle

with

one

with

E)
v

245

and

G.

of edges. The basecase is

three vertices contains a
all vertices in an arbitrary
order and

the number

on

graph

in

Tours

Hamiltonian

7.12

(put

at

least

cycle).

hypothesis:
the

given

how to find a
with > m edges.

know

We

conditions

Hamiltonian

cycle

in graphs

how to find a Hamiltonian

cycle in a graph with m -1 edges that
a graph.
of the problem.
Take any pair of
Let G =(V, E) be such
w
v
in
is
and
and
consider
the
which
the same as G
vertices
G',
G,
graph
nonadjacent
w
we
the
know
how
v
are
connected.
induction
to find a
that
and
By
hypothesis,
except
a
in
in
Let
be
such
G'
G'.
Hamiltonian
(see Fig. 7.43). If
cycle
x{, jc2,...,Jt\342\200\236,xx
cycle
in G and we
in
is
contained
the cycle, then the same cycle
the
(v, w) is not included
edge
we can assume that v=x{
and w=xn.
loss
of generality,
are done. Otherwise,without
The stage is now set to find a new
for
G, d(v) + d(w)>n.
By the conditions given
We

to show

have

satisfies

the conditions

Hamiltonian

cycle.

Consider all the

the

two

conditions

vertices

of the

jc, and

edges

in G

problem).

jc/+1

, which

of

out

xh

the

edges

not
*/_!,...,

Figure 7.43 Modifying

v and

n-2

G contains

are neighbors

and w is connected
to xh Using
does
Hamiltonian
cycle that

v(=x!), xi+l,*/+2,...,w(=xn\\

coming

But

in

the

cycle,

(v, jc/+1)

use
v (see

w. There are at least n of them (by

other vertices. Therefore,there are
such that

and (w, *,), we

the

edge
Fig. 7.43).

Hamiltonian

(v,

w).

cycles.

can now

find

to jc,+1
a new

is

the

cycle

is connected

It

Graph Algorithms

246

Implementation

The

complete graph
much smaller graph
and

edge at a

one

replaces

of this

implementation

straightforward

as follows. Take

time.

graph G,

proof

do better

can

We

with

starts

by starting

the
a

with

path (e.g., by DFS),

to complete this path to a Hamiltonian cycle.
a Hamiltonian
have a larger graph
now
We
cycle.
Usually,
only few edges
n
1
will be added. We
be added.
However, even in the worst case, at most
will
edges
with
is
G', until a Hamiltonian
can apply the proof above iteratively, starting
path
of steps to replace an edge
is O(n).
There are 0(n)
number
obtained for G. The total
add the

and

edges

from

(not

edgesto

replace;

7.13

Summary

hence,

the

input

a large

find

G) necessary
has
G', which

the algorithm

in time

runs

0(n2).

among pairs of objects.Sincemost algorithms

an
is
input, the first issue involved in graph
algorithms
require
We studied two types of graph
traversal.
traversals:
search
depth-first
frequently graph
search
DFS
was more
(BFS). We saw several exampleswhere
(DFS), and breadth-first
we suggest trying
DFS
first (although
there are many
suitable than BFS.
Therefore,
BFS is superior). DFS is especiallysuited
where
for recursive
examples
algorithms on
BFS
also
more
this
is
not
a rule \342\200\224
it
usually
space (although again,
graphs.
requires
on
We
the
have
also
seen
an
of
which
was
search,
depends
graph).
priority
example
from
a single source. Priority
used to compute shortest paths
search
is more expensive
to model relationships

used

are

Graphs

examination of

than

for

have

for

useful

major difficulties
directed acyclic graphs are

It is

important

different

many

for

much

usually

cycles (Exercise

7.54).

even

with

involving

to design
a small number

easier

that

Algorithms

graphs.

weighted

Therefore,

algorithms.

graph

that graphs

realize

to

problems

optimization

usually cause

Cycles
trees or

execute.

It is

search.

regular

whole

the

components,

and

consideration

may

properties
not have

by

algorithms.

We

Network-flow

discuss

problems
also

discuss a class

help

of

connectivity),

reasonably

biconnected

Decomposition basically allowsus to

even
the graphs under
though

them.

this
and

us to

components.

(such as

technique
matrices
there

represented

Reductions

connected

useful

Another

in

strongly

certain

assume

all or

checking

require

to

faster

of edges can

a graph
can be very slow for most graphs.
cycles
useful.
it is also usually
Graph decomposition is very
Fortunately,
We have seen decompositionsinto
connected
inexpensive.
components,

large

of the

fraction

algorithms

and

for graph

is a

called
running

reductions

and

whether

Since graphs

is reduction.

relationship

problems

matching

solved by algorithms whose

worst case. This class includes

natural

relationship

determine

problems,

algorithms

between

in general

are excellent

a problem

graph
in

and

can be
matrix

10.

Chapter

source for reductions.

is difficult.

In

11, we

Chapter

which probably
NP-complete
problems,
in the
size of the
times
are polynomial

be

cannot
input

in

the

numerous
The differences betweeneasy
graph problems.
sometimes seem minuscule. For example,we have seen an
efficient algorithm
whether a directed graph
to determine
a simple cycle of odd
contains
the same
vertex
length;
problem with the extra constraint that the cycle contains a given
to understand and develop
an intuitive feeling
(or edge) is NP-complete. It is essential
problems

and

hard

problems

Bibliographic Notesand Further

for

differences.

these

Thus,

is a

theory
only

this

field,

and

[1979],

for

important

very

now

graph
books

the

Berge [1962], Ore [1963], Harary

[1976], Chartrand [1977], Capobiancoand
and Chartrand and Lesniak [1986]. There
[1984],
including

algorithms,

graph

and related

Even

results

basic

were

theory is a developedand
on graph theory have been
[1969],
[1973], Deo
Berge

Murty

Tutte

books devoted to

Most of

in mathematics.

field

new

Nevertheless, by
century.
with thousands of results. Many

them

among

[1974], Bondy
Bollobas

relatively

in

well-understood
published,

11 is

Chapter

and Further Reading

Notes

Bibliographic
Graph

in

graph algorithms.

understanding

discovered

material

the

247

Reading

Molluzzo

[1979], Golumbic

[1978],
also

are

several

[1980] (which

classes of

Gondran
and Minoux [1984]
graphs),
Gibbons
Nishizeki
and Chiba
[1985],
(which
[1988]
emphasizes
optimization
problems),
and a survey paper by van Leeuwen
(which is devoted to planar
[1986].
graphs),
The notion
of Eulerian
graphs is due to Euler [1736], and it is regarded as the first
An algorithm
in graph theory.
for finding Eulerian paths
result
can
be obtained
quite
Even
from
the
for
or
Ebert
search
(see,
[1979]
[1988]).
easily
proof
example,
Depth-first
was first described by Lucas
work by Tr6maux)
and Tarry [1895],
[1882]
(describing
where it was used to design algorithms
to traverse
a maze. The importance
of depth-first
in
the
of
who
also
the
search was made evident
work
algorithms
presented
Tarjan [1972],
for biconnected and strongly
connected
components.
tree
has been studied extensively. The
The minimum-cost spanning
problem
is due to Prim
in Section
7.6 (although
not
its implementation)
algorithm
presented
is
the
of
Exercise
is
due
to Kruskal
7.59)
[1957]. Another
(which
algorithm
subject
were
for
the
minimum-cost
tree
Other
[1956].
finding
algorithms
spanning
developed by
and Gabow, Galil,
Yao [1975], Cheriton
and Tarjan
[1976], Fredman and Tarjan
[1987],
emphasizes

perfect

graphs

Tarjan

[1986]

Spencer,and

in Section
7.5 was
for single-source shortest paths
presented
a heap is due to Johnson [1977]
The implementation using
[1959].
the case in practice, this
When the graph is sparse, as is usually
(see also Tarjan [1983]).
is a fast algorithm. If the number
time
of edges is proportional
to | V |2, then the running
of this algorithm
with
a
is O (| V | 2log
V
for
dense
A
better
|
\\.
graphs,
implementation
The
time
of
is
the
of
Exercise
7.43.
best-known
0(\\V |2),
running
asymptotic
subject
time
for this
data structures) is
running
(using
problem
quite
complicated
due
to Fredman
and Tarjan [1987]. The all-pair
0(\\E | + |V|log |V|), a result
in
Section
7.7 is due to Floyd [1962]. It works
algorithm
shortest-paths
presented

The

algorithm

developed by

correctly

Dijkstra

for weighted

negative weight
on the average
0(|
and
time
survey

V
an

|2 log2

\342\200\224

Spira

|V

earlier

is 0(\\

graphs

(Exercise

cycles

|),

and

by Deo

negative weights, provided that

It is possible to find all the shortest

possibly

7.73).

[1960] to obtain
|). For more information
and Pang [1984] (which
includes,

|2 log

|V

there

presented an algorithm whose average running

Takaoka [1987] used a hybrid
of Spira's

and

by Dantzig

algorithm
V

[1973]
Moffat

with

an
on

algorithm
shortest-path

among

are no

paths faster

whose average running

see
algorithms

other things,

is

time

algorithm
the

222 references).

248 Graph Algorithms

7.8 is due to Warshall
[1962].
algorithm presented in Section
to
network
flows
its
theorem
and
were
application
augmenting-path
of
data
the
excellent
and
Fulkerson
An
Ford
structures
[1956].
discovered
description
by
A new
for
flows is given in Tarjan
network
[1983].
and combinatorial algorithms
and
flow
was
for
network
Tarjan [1988].
recently developed by Goldberg
algorithm
on the network-flow
More information
problem and many of its extensions can be found
Christofides
in Ford and Fulkerson [1962], Hu
[1975], Lawler [1976], Minieka
[1969],
The

closure

transitive
The

and Plummer
and

theory

in

Hamiltonian

finding

[1986] covers
for various

algorithms
algorithms

matching

[1982], and

and Steiglitz

Papadimitriou

[1978],

Lovasz

bipartite

in dense

cycles

mathematical

problems.

matching

and

and

Gondran

the

both

Galil

Minoux

[1984]. A book
foundations of matching

general
graphs

[1986] presents

in
graphs. The algorithm
is based
on a theorem (and

of

survey

7.12.2 for

Section
its

by

by Ore

proof)

[I960].

Two important

subjects

isomorphism.

The

algorithms were not discussed

of characterizing planar
graphs
in graph theory.
is one of the oldest
in
the plane
problems
Early
Auslander
and
Parter
and
were
[1961]
developed
by
problem
to
determine
whether
Cederbaum [1966]. A linear-time
algorithm
graph

developed by

and Tarjan

Hopcroft

a graph

decompose

polynomial-time

isomorphism is

[1974].

It

a linear-time

uses

3-connected

the

NP-hard) is still
see, for example,

unknown

(more

Hoffman

[1982],

on that
or Luks

in

11).

Chapter

here:
and

algorithms
Lempel,
a graph

and
them
for this

planarity

embedding

and

Even,

is planar was

to
(DFS-based) algorithm
and Tarjan [1973]). This
and data structures. A

components
(Hopcroft
of many
other
development
algorithms
for graph isomorphism has not been
algorithm
one of the very few major problems whose status
into

motivated

algorithm

in graph

problem

For a

found
(either

Graph

yet.

or

polynomial

discussion on

this

topic

[1982].

Bruijn sequences (Exercise7.28)can be found in Even [1979].

and Vitter [1986]. Exercise7.55 is motivated
Sedgewick
by an
exercise from Bollobas [1986], and Exercise
from
7.58 is motivated by an exercise
Lovasz [1979]. Ford
of
contains
an algorithm that
satisfies
the requirements
[1956]
Exercise 7.75. The algorithm
for transitive
in Exercise
closure hinted
7.81 is from
Warren
Exercise
7.97 is from Lovasz and Plummer
Gabow and Tarjan
[1975].
[1986].
The
for
in
the
bottleneck
Exercise
7.100.
[1988] presentan efficient
algorithm
problem
theorem presented in Exercises
7.101
and 7.102 is known
as
theorem.
Gomory's
Exercise 7.105 is from
of
Lovasz
Exercise
7.121 is related to a problem
[1979].
in Manber and McVoy
is solved
designing
space-efficient
[1988].
routing tables, which
A

on de

discussion

Exercise7.46 is

Drill

7.1

from

Exercises

Considerthe

problem

Solve this problem using

7.2

LetG = (V,

E) be a

of finding balance factors

in binary trees discussed in Section
DFS.
You need only to define pre WORK and postWORK.

connected undirected graph,

and

let T

be a DFStree

of G

rooted

at

5.8.

v.

a. Let H be an
b. Let R be
Prove that

7.3

7.4

7.7

7.8

Characterize

all

and

identical

traversed

is

H is

let S be
tree of S.

the

of G

subgraph

induced

vertices

in R.

the

by

graphs

at v that
if

immaterial

is identical

7.14) in the following way. Assume that you

if the graph is cyclic, a
graph is acyclic. Obviously,
that will output the topological-sort
sort is impossible.
Design an algorithm
topological
if the graph is acyclic,and will output a cycle if the graph is not. The running
time
labeling
of the algorithm should be O (| E \\ + \\ V |).
Topological

algorithm

Modify

know

or

whether

Sorting (Fig.

not the

Consider algorithm
Source Shortest Paths (Fig. 7.17) Prove
Single
consisting of all the edges that belong to shortest paths from v, found
of the algorithm, is a tree rooted at v.

the

that

Prove or show
works correctly
there are

a counterexample:
for

weighted

no negative-weight

7.11

Prove

MCST (Fig.

algorithm

graphs some of

weighted

7.20) to

graphs someof whose

7.12 a. Give
such
rooted
b.

an example
that

graph.

minimum-cost
find

a counterexample: algorithm
have negative
edges

or show

the

Single Source Shortest

whose

of a weighted

minimum-cost

Prove that,
tree.

if

all the

a maximum-cost spanning

MCST (Fig. 7.20)works

costs.

connected

undirected

graph

spanning tree of G is

the

same

(Fig.

7.17)

weights, provided

costs are distinct,

then

tree.
for weighted

correctly

G =(V, E) and a vertex v,

as the shortest-path tree

at v.

connected
undirected
an example of a weighted
graph G
the minimum-cost
spanning tree of G is very different
tree rooted at v. Can the two trees be completely
disjoint?

=(V, E)

that

Describe the changes in

deleting the vertex c from

7.14 a. Run

a constant

spanning

Give

such

Paths

have negative

edges

tree rooted at
by

cycles.

Let G = (V, E) be an undirected

there exists exactly one unique
Modify

Algorithm

subgraph
execution

the

during

and let T be the shortest-paths

Let G = (V, E) be an undirected weighted
graph,
a vertex v (Exercise
7.6). Suppose now that all the weights in G are increased
tree from v?
number c. Is T still the shortest-paths

7.10

7.13

of T and

v such that there exists a DFS

that contain a vertex
to a BFS spanning tree rooted
at v. (Two spanning
same
contain
the
set
of
the
order
in which they
are
they
edges;
here.
However, both trees must have the same root v.)

undirected

rooted

tree

trees are

that

7.9

7\\

the intersection

input

no longer

7.6

of

be a DFS

R could

that

Show

of H.

tree

is a connected undirected graph

a spanning
tree T of G, and a vertex v.
G = (V, \302\243),
T
a
to
is
a
determine
whether
valid
DFS
tree
of G rooted at v. In other
Design
algorithm
T can be the output
of DFS under some order
whether
of the edges
words, determine
should be O (| E | + | V |).
starting with v. The running time of the algorithm

The

spanning

7.5

a subtree

G.

subgraph of

induced

arbitrary

not necessarily a spanning

249

Exercises

Drill

the

biconnected

the

biconnected
components
in Fig. 7.25.

and

from the

and biconnected

tree

a vertex

shortest

resulting

from

the graph

components

algorithm on

the

graph

in Fig.

7.44. The

v,

path

algorithm

250

Algorithms

Graph

should
b. Add

7.15

that

7.16 a.

the edge

there

are no
the

Run

(4,8) to the

in the figure. Show

given

the

High

values

as

graph

and discuss the

changes this

to the

makes

algorithm.

of a biconnected
tree in Section 7.9.1 is valid.
You have to show
are connected.
cycles, and that the set of all biconnected
components

strongly
should

algorithm

b. Add the

connected components
the DFS numbers

follow

values as computed

7.17

are

that

in each step.

the algorithm

the definition

that

Prove

by

DFS numbers

the

follow

computed

by the

edge (4,1)to

algorithm

the graph

in

each

algorithm

the graph

on

are given

that

in

the

in Fig. 7.45. The

Show the High

figure.

step.

and discuss the

changes

this makes

to the

algorithm.

connected
Let G = (V, E) be a strongly
graph and let T be a DFS tree in G. Prove that, if all
from G, the resulting graph is still
the forward edges in G, with respect to 7, are removed
strongly

connected.

Figure 7.44 An

undirected

graph with

DFS numbers

for Exercise

7.14.

Figure

7.45

A directed graph

with

DFS

numbers

for Exercise

7.16.

Creative Exercises

7.18 a.

the

Prove

of the algorithm for

correctness

an odd

finding

length cycle

in

a directed

251
graph

(Section 7.9.4).

b. Show
7.19

7.20

of a graph

an example

that is

not

connected

strongly

for which the algorithm

does

work.

not

Show an implementation of the algorithm

discussed
in Section 7.10.2 to find a perfect
in
with
In
with
a
each
at least n. Your algorithm should
vertices,
matching
degree
graph
in time O (| V \\ + \\ E \\) in the worst case.
run
somewhat the proof of existence of perfect
in dense
matchings
are given a graph with 2n vertices such that not all of them have
the sum of their degrees
is at least In. Is
vertices,
but, for any two nonadjacent
high
degree,
it still true that a perfect matching
exists?
Is
the
obtained
in Exercise
7.19
always
algorithm
still valid?
exercise

This

generalizes

graphs. Suppose that

you

Exercises

Creative

we assume that the graphs are given in an adjacency-lists

Unless
otherwise,
specified
O (| V | + | E \\) space; hence, we say that an
requires
representation. Such a representation
time if its running
time
is 0(|V | + |\302\243
Unless specified
runs in linear
|).
algorithm
are
In
times
worst
case.
some
time is
all
the
cases, a particular
otherwise,
running
running
for an
that
in
other
we
ask
and
the
exercise
time;
cases,
requires
achieving
only
given
\"efficient algorithm.\" In the latter case, the reader should
try to find the best possible
is
of course, the best running
time
is unknown when
a problem
In practice,
algorithm.

encountered.

7.21

an undirected

Given

H of G

such

that

induced

subgraph

includes

all edges

induced
subgraph
graph G = (V, E) and an integer k, find the maximum
that it does not exist. (An
each vertex in H has degree > k, or determine
F
// = (\302\243/,
and
of a graph G = (V, E) is a graph
F) such that (/cV,
E both of

in

time. (This problem

7.22

is discussed

whose vertices are

in Section 5.3.)

in

U.)

The algorithm

should run

in linear

undirected
(V, E) be a connected
graph. We want to pick a vertex of degree 1 of G,
another
vertex
it and its incident edge from
G, and continue this process (i.e., taking
of degree 1 in the remaining graph, removing
all edges are removed. If
it, and so on) until
this procedure
is always
for certain graphs, then designing
possible
algorithms
by induction
for these graphs
easier.
that
these
be
Characterize
connected
undirected
may
graphs
satisfy
In other words, find necessary
conditions.
and sufficient conditions for a graph G on which
the procedure
described above is possible.

Let

G =

remove

7.23

Describe

7.2.

an efficient
implementation of the Eulerian
graph algorithm discussed
The algorithm should run in linear time and space.

7.24 Let G = (V\\

linear-time

equal to the
7.25

A directed

E)

be an

algorithm

in

Section

undirected graph such that each vertex has an even degree. Design a
to direct the edges of G such that, for each vertex, the outdegree
is

indegree.
Eulerian

circuit

is a directed circuit

that

contains

each edge exactly

once. Prove

252

Graph Algorithms

an efficient

circuit if and only if the indegree of each

contains a directed
Eulerian
to its outdegree, and the underlying undirected graph is connected. Design
algorithm to find such an Eulerian circuit if it exists.

Let G = (V,

E) be

a directed graph

that

is equal

vertex

7.26

a. Prove
b.

7.27

graph

of odd degrees.

with k vertices

even.

k is

that

Design

an

exactly

one of these

to

algorithm

k/2

find

paths such

open

that

each

in G is

edge

included

in

paths.

a vertex in a connected undirected

whose removal does
graph
The
not
algorithm should run in linear time. (Do not use the
graph.
As a consequence, prove that every
connected
biconnected components algorithm.)
graph
contains such a vertex.

Design an algorithm to

de

A binary

s of size

unique index i

somewhere

is represented

s =

that

such

sequence of 2\"

is a (cyclic)

sequence

Bruijn

string

binary

find

the

disconnect

7.28

an undirected connected

at

ai+x

\342\200\242
\342\200\242
\342\200\242

in

(where

fl/^-i

the

bits

\342\200\224
\342\200\242

a2\" such

axa2

that is,

sequence;

the indices are

taken

there

modulo

that each
exists

2\.") For

11010001 is a binary
de Bruijn sequence for n =3.
Let Gn = (V, E)
as
follows.
The
vertex
V
defined
set
to
the
of all binary
set
corresponds
graph
\342\200\242
\342\200\242
- 1
A vertex corresponding to the string axa2 \342\200\242
(| V \\ =2\"~l).
strings of size n
an.x has
'''
to a vertex corresponding to the string bxb2
an edge leading
if and only if
bn_x
\342\200\242
''' an-\\ -bxb2
\342\200\242
\342\200\242
Prove that Gn is a directed Eulerian
and discuss
bn_2.
#2^3
graph,
the implications for de Bruijn sequences.

example, the

be a

7.29

sequence

directed

/', =0, and

= 1 for 1

ik

integers
n vertices of

positive
with

dn.

d2,...,

dx,

Let (/',, o!),(/'2,o2),...,(/\342\200\236,on)

a.

that

such

dny

degreesexactly
7.30

Given
algorithm for the following
problem:
\342\200\242\342\200\242\342\200\242
construct a tree
+dn = 2n-2,
dx+d2+

efficient

an

Design

dx>d2,...,

be a sequence

of pairs

of integers

such

that

< k <n

b.
Find

= n-\\

^Oj

tree

a rooted

7.31 Let G = (V,

An

greater

undirected

than

the label

such labeling

of vertex k

algorithm

is

is ikt and

its outdegree is

(n).

of

at

is possible,

graph G = (V, E)
k different
colors

Design a linear-time
7.33

indegree

be a directed graph (not necessarily

acyclic).
of the graph with distinct
labels from

v is

colored using
are not

the

that

run in time O

vertices
that no

determine

7.32

should

E)

to label the
each vertex

such

n vertices

with

ok. The algorithm

to

said

Design
to | V

\\

an efficient algorithm
such that the label of

one of v's predecessors(if v has any), or to

is
(w a predecessorof v if (w, v) e E.)

least

be ^-colorable

if

all

the vertices

of G can be

the same color.

two adjacent vertices have
to color a graph with two colors or determine that two colors

such

that

no

sufficient.

Let G =

two colors.
It may be
graph that can be colored with
colors
in
Exercise
several
in
different
the
Use
possible
ways.
algorithm
7.32 to prove that the coloring of G is unique
for interchanging the colors, which
(except
can always be done) if and only if G is connected.

(VyE) be
to

color

an

undirected

G with two

Creative Exercises

7.34

whose root is r. Each vertex in T is

The tree is represented by
alphabet.
lists.
Let
P
be
a
an
of characters).
pattern
adjacency
(represented
string
by
array
Design an
the pattern appears at least once in a path from the root to a leaf.
algorithm to find whether
in the worst case, where n is the number of
The algorithm should run in time 0(n+m)
in the tree and m is the size of the pattern.
vertices
undirected tree

T be an

Let

associated

7.35

253

binary)

necessarily

(not

taken from a fixed

a character

with

finite

undirected graph G = (V, E) that contains exactly one cycle, direct

the
indegrees of all vertices are at most 1. (Such directed graphs are called
What is the complexity of your
functions.)
they correspond to injective

a connected

Given

edges such

the

that

since

injective

algorithm?

7.36

7.37

Let G = (V, E) be an undirected

to determine whether
graph.
Design an algorithm
of
such
that
the
to
direct
the
G
of
possible
edges
every vertex is at least 1. If
indegree
possible, then the algorithm should show a way to do it.
a connected undirected graph
conditions are satisfied:

Given

a.

b.

The resulting directed

away from the root).
which

edge,

Any

does not

7.38

direct
\302\243),

a rooted

contains

graph

edges of the tree.

What is the complexity of your

G =(V,

to the

belong

its edges

such that

tree (i.e., a tree

tree above,

the

completesa

it is

two

following

whose edges

all of

directed

cycle

is

it

point

with

algorithm?

find a simple (directed)

Given a directed acyclic graph
G = (V, \302\243),
maximum
number of edges among all simple paths in G. The algorithm

in G

path

should

that has
run in

the

linear

time.

7.39

a. Solve the

problem in

your algorithm work for

c. Will

your

7.40 Let G = (V,

path of G.

each two

E)

simple
shortest
not

7.42

7.43

(not

necessarily

v and

The algorithm

in the

are now

acyclic) graphs?

number
be a directed acyclic graph, and let k be the maximal
to divide the vertices into at most k + 1 groups
algorithm

vertices

7.41 Let G = (V,

subgraph

That is, you

graphs.
paths.

cost edges?

negative

work for general

case of weighted

maximum over all

of edges in a
such that for

Design an

w to v.

from

is the

weight

b. Will

algorithm

7.38 for the

Exercise

for a path whose

looking

same group there is no path from v to

run in linear time.

and

there is

no path

should

the following
property. G consists of an acyclic
back edges, such that every
vertices, and additional
a linear-time
path in G contains at most one back edge. Design
algorithm to find all
all
source
to
other
a
fixed
of
from
vertices
G.
that
the identity of H is
(Note
paths

//,

E)

be a directed graph with

all of G's
contains

which

known.)

and let v and w be two vertices in G. Design a linearLet G = (V, E) be a directed

graph
time algorithm to find the number of different shortest paths (not necessarily vertex disjoint)
on the edges.)
v and w. (There are no weights
between
Design

requires

an

implementation
running time of

of algorithm
\\2) in the

O (| V

Single Source Shortest

worst case (for any size

Paths

of E).

(Fig.

7.15)

which

254

Graph Algorithms

7.44

Let

7.45

7.46

E) be a weighted
minimum weight. The algorithm
G = (V,

an

to find a cycle

algorithm

(|

in

G of

|3).

graph such that each vertex corresponds to a

edge is equal to the distance between the points it
has been suggested to find the shortest path between two
heuristic
connects. The following
for single-source
vertices .v and t in a Euclidean graph. Use Dijkstra's
algorithm
given
shortest paths, except that, at each iteration, choose the next previously unchosen vertex x
the sum dist(s, x) + Euclid'_dist(x,
minimizes
that
t), where dist corresponds to the shortest
to
the
Euclidean
distance
is assumed
to be given).
and
(which
path
Euclid_dist corresponds
the shortest path from s to t is found.
t is chosen, then
When

graph is an undirected
plane and the weight

Euclidean

in the

weighted

of an

a.

How
would
you implement this algorithm? You
from the implementation of Dijkstra's
algorithm.

b.

Explain

c.

Give

why this method will

an example

running

The

is a

input

cost

positive

so

if

(v, //)

7.48 LetG = (V,

be two
from

path

work for general

not

this
an

heuristic

is much

example

where

number

g E,

the

cost

as

(a,).
\302\243<

to mention

have

the

only

differences

(non-Euclidean) graphs.
faster

it is

(by

than

more

faster.

not

What

a constant) than
is the worst-case

of vertices?

directed graph G =(V, E) with

associated with
each
c{w)

a*, // is defined

|, a2

minimum-cost

where

and
algorithm,
time in terms of the

Dijkstra's

7.49

should

7.5 break ties arbitrarily.

The algorithms for finding shortest paths described in Section
these algorithms
such that, if there are several different
how to modify
Discuss
paths of the
same length, then the one with the minimum number of edges (hops) will be chosen. You
of the same length
and the
can use O (| E \\) additional
space. (Ties between several paths
same number of edgescan be broken arbitrarily.)

point

7.47

Design
graph.
run in time O

directed

vertex

a distinguished

The costs

of

the

two

of getting from v to // is 0. Design

v to all other vertices.

\\\\

that there

such

The cost of a

w.

vertex

endpoints
an

efficient

v and

is a

directed

path

are

ignored,

algorithm

to

find

the

paths from

such that all the weights are positive.

Let r and
E) be a directed weighted graph
vertices in G and k< \\ V | be an integer. Design an algorithm
to find the shortest
v to w that contains exactly k edges. The path need not be simple.

There is a large
form. The input
this case, shortest

which have the following

called bottleneck
problems,
a
We
are
interested
certain
weighted graph.
property of the graph (in
of
of a subgraph as the weight
We define the bottleneck weight
paths).
the maximum-weight
edge in that subgraph, as opposedto the usual definition of sum of the
In this
we consider
is the bottleneck.)
weights. (This maximum-weight
problem,
edge
bottleneck
shortest
cost of an
is
as
the
maximum
defined
the
cost
of
the
paths (i.e.,
path

edge

in

class

of problems,

is a

the path).

a. Design an algorithm
to solve the single-source shortest-paths
costs
are defined as above. Can you say something special
paths obtained
by this algorithm?

b.

Design

an algorithm

defined as above.

to solve the

all-pairs

shortest-paths

where

problem
about

the

the

path

tree of shortest

problem where the

path

costs are

Creative Exercises 255

7.50

(V, E) be a

G =

Let

a linear-time
source v.

7.51 Let d(v)

directed

acyclic

weighted

to solve the

algorithm

graph

with

single-source

weights. Design
from a given

negative

possible

problem

shortest-paths

to sort all the

algorithm
degree of a vertex v. Design a linear-time
vertex
That is, if
given directed graph G = (V, E)by increasing
degrees.
< d(v), then edges to u precede edges to v in all the adjacency lists that contain both.
d(u)
Ties are broken arbitrarily.
The algorithm can use linear
space.

7.52

of a

Find necessary and sufficient

graph G = (V,E) can be
to a simple

corresponds

7.54

cycle.

disjoint
an efficient

Design

an undirected connected graph

G = (V,
weights). The length of the smallest cycle in

that there are undirected graphs

Prove
different cycles. (This claim implies

to solve the

an algorithm

Design

Input:

algorithm

the

S from vertices

Output: A function
conditions are satisfied:
1. If(w,

2. Ifw

Consider
any

the

(and

words, prove

7.57

that

a simple cycle of minimum

length
is called the girth of the graph.

in

(no

n vertices and

O(n) edges that contain 2a{n)

sparse graphs may have an exponential
that requires checking
all the cycles is inherently

n + \\ vertices

with

edge

to

(w,

subsets

S(u){j{X(u,

and

w)

is labeled

of

{1, 2,...,

whose

n edges,

a unique

with

n)

such

underlying

integer X(u, w)

the following two

that

w)}

S(u)\302\261S(w)

subset

of {1,

2,..., n)

including

the empty

set or the

whole

set.)

in Exercise 7.55. Prove that the problem cannot be solved for

In other
which contains a cycle (not necessarily
directed).
the restriction of the problem
to trees is necessary.
problem

labeling)

any

in a directed

are

an edge,

V\\

by

7.58 LetG = (V,

E) be a

=0
2 f(tk)
each edge

\302\243,

even

graph G = (V, E) is a subset V'cV such that no two vertices in V

and for every vertex w g V there is an edge (v, w) such that v e
is a directed graph G = (V, E) with n +1 vertices and n edges, whose underlying
undirected
an algorithm
to find a kernel in G or determine
that
graph is a tree. Design
kernel exists.
kernel

connected
The input

7.59

S(w)

can be any

again

graph

then

then

*w,

S(u)

(Note:

7.56

w)e\302\243,

the set of edges E of an undirected

subsets Ex,...,Ek such
that
each
algorithm to find such a partition

problem:

following

A directed graph
G = (V, E)
a
is
where
each
tree,
graph
range 1, 2,..., n.

find
\302\243),
a graph

with
that

undirected
in

which

into

Given

an
number of cycles;therefore,
inefficient
for general graphs.)

7.55

under

conditions
partitioned

these conditions.

that satisfy

graphs
7.53

the

denote

adjacency lists

if

directed graph and let /be a

e i,...,

(v, w), we

Here is a sketch
one edge at a

of a

time,

e* is a circuit
have/(v,

in

G.

function

Find a function

on all

defined

on

the

edges of G such
of G

vertices

such that

no

that
for

w)=p(w)-p(v).

of keeping one
different MCST algorithm.
Instead
of disjoint trees (which are all
we keep a collection

tree

part

enlarging it
of the MCST)

and

256

Graph Algorithms
them, adding one edge at a time. Initially, all the vertices are considered as
the minimum-cost
of size 0. In each step, the algorithm finds
edge that
the edge. Prove that
these two trees by adding
two separate trees, and combines
connects
Describe an implementation of an algorithm
based
such an approach is feasible and correct.
on this approach. What is the complexity of your algorithm?
(Hint: The Union-Find data
combine

and

trees

disjoint

structure is helpful
7.60

here.)

Let G = (V, E) be an

(i.e.,F doesnot
contains

G that

undirected

graph, and let F be a subgraph

to

weighted

algorithm
any cycles). Design an efficient
has minimum
cost
the edges of F, and that

contain
all

that is a forest
tree in

of G

a spanning

find

among

all

trees

spanning

containing F.

7.61 Let G = (V,

spanning

Suppose
under which T is no
MCST or to determine

conditions
a new

find

7.62

be a connected

E)

tree of G.

the

longer

that T

undirected

cost of one
an MCST.

is still

and let

graph,

edge e
Design

an MCST.

(e

in
an
may

T be a

minimum-cost

is changed.

efficient
or may

Discuss

algorithm

the

either to

not belong to

T.)

that can be modeled as a weighted

undirected
Consider a communication network
connected
as a vertex and each line of
graph G = (V, E). Each site in the network is represented
is bidirectional and has a cost associated with it. The cost may correspond
communication
to the expected delay
on the line, or to the tariff for using this line. Each site has only local
to it. An MCST of the
information; that is, it knows only the edges (and vertices) adjacent
If
network
can be used to broadcast
to
all
sites.
we
the messages by
broadcast
messages
the
of
cost
is
minimized.
that such an
the
then
the
total
Assume
MCST,
using only
edges
MCST
is computed
of the edges adjacent
by some method and that each site knows which
to it belong
to the MCST. Assume
now that sites in a certain subset U <zV share between
them the information that
In other words, every
is known to all of them.
site in (J knows
not only about the edges and vertices adjacent
to itself, but it also knows all the edges and
vertices
in U. Furthermore, assume that the partial MCST restricted
adjacent to all vertices
to U is connected (i.e., it is a tree). Consider an edge
to the MCST,
e e U, which
belongs

a.

has just

cost

whose

U.

In

other
the

modify

b.

under the change in e's cost is guaranteed not to affect the MCST.
that can be checkedwith the information
known to the sites in
that no action
needs
to be taken to
words, how can the sites in U determine
MCST after the change?

under which
from the original one
the modified
MCST is different
that belong to U (hence, the change can be handled locally). Consider only
that can be checkedwith the information
known to the sites in (J.

the

in edges

conditions

c. Describe
the

conditions

conditions

Find
only

within

changed.

conditions

the

Find

Consideronly

7.63

weighted
that

an algorithm
in parts a and b (again only
to check for the conditions
to modify
does not need to handle
the MCST accordingly. The algorithm
case where the change to the MCST may be outside
U.
briefly

U), and

is
Consider the problem of broadcasting
interest
now that the main
in a network, but assume
fast dissemination of information
rather
than minimum cost. In other words, the costs
to the time it takes to forward a message, and we want to minimize the elapsed
correspond
time
of broadcast.
A message can be sent concurrently on separate
links.
Assume that one
messageis sent from a fixed source and is forwarded to all other sites such that each site
receives only one copy of the message. You can assume that you are a controller
with full
of the network.
information about the topology

Creative Exercises

a. Design
b.

to determine

an algorithm

are associatedwith

7.64

of t(v)

Let G = (V, E) be a

connected

weights are

and

of G

7.65

7.66

positive

units

to find T(e)

undirected

of

variation

only delays

there
are also delays
forwarding when
site v to forward a message to one
then it takes kt(v) time. (The
neighbors,

an

edge

Assume for simplicity

that the
graph.
of G. Denote by T(e) the spanning tree
trees of G that contain
e. Design an

cost among all spanning

e e E. The algorithm
edges

for all

should

run in time O

to find the minimum bottleneck weight

Design an efficient
algorithm
connected
undirected
(Recall that a bottleneck weight
weighted
graph.
In other words, you
maximum
of an edge in the subgraph.)
weight
is minimized.
spanning tree in which the maximum weight

Solve a

the

that

of time for

weighted

Let e be

distinct.

minimum

has

that

algorithm

optimal

It takes t (v)

If v forwards the message to

are known for all v.)

neighbors.

values

the

determine

associatedwith
of its

assuming

forwarding,

optimal

the links.

an algorithm to
the sites.

Design

the

257

the problem

in

7.65

Exercise

directed

for

(|

|2).

tree of a

spanning

is defined
are

asked

The

graphs:

to

as the
a

find

is a

input

a distinguished
vertex v. Find a rooted spanning
G = (V, E) with
weighted directed graph
tree, with v as the root, such that the maximum-cost edge in the tree is minimized. (Recall
that
in a rooted tree the directions
from the root.)
of all edges are away

7.67

Let G = (V,
that

is it

how difficult
*

7.68

Let G = (V,
Suppose that

in

G are

to modify

weighted graph,

weighted

add a new vertex v

we now

and

increased by a constant
T into an MCST?

be a connected

E)

of G.

vertices

7.69

be an undirected

E)

all the weights

Design a linear-time

undirected

to G,

let T be an MCST of G. Suppose

number c. Is T still an MCST? If

some

with

to find a new

algorithm

and let

graph,

together

T be

weighted

MCST that

an

MCST

now

of G.

edges from

includes

not,

to

v.

that
the cost of a spanning tree is not the sum of the costs of the tree's
edges but
an efficient algorithm to find
rather the product of their costs (all costs are positive). Design
this assumption.
a maximum-cost
(You can assume that all costs are
spanning tree under

Suppose

distinct.)

7.70 Let G = (V,

Design

E)

vertices numbered
components contain
edges incident to it.
7.71

undirected graph with n vertices numbered from 1 to \302\253.

k such that successively
the
to
find
the smallest
deleting
algorithm
in a graph all of whose
connected
1,2,...,/:
(in that order) results
all the
at most nil vertices. Deleting
a vertex also includes deleting

be a connected

an efficient

(Hint:

data

union-find

structure.)

Let G = (V, E) be an undirected graph. A set F c E of edges is calleda

every cycle of G has at least one edge in F. Design an algorithm to
feedback-edge set.

7.72 Let G = (V,

E)

Prove
weighted

cycles.

that

algorithm

graphs

undirected graph
in Exercise
set (defined

be a weighted

to find a feedback-edge
7.73

Use the

All_Pairs_Shortest

with possibly

negative

with

positive weights.
7.71) of G of minimum

Paths given
weights

provided

in

feedback-edge
find

set if

a minimum-size

Design an

algorithm

weight.

7.22 works correctly

Fig.
that there are no negative-weight

for

258

Graph Algorithms

7.74 LetG = (V,

directed graph

be a weighted

E)

but there are

no

(i.e., there are

cycles

negative-weight

that some

such

tree
is negative). Let T be a spanning
T
contains
whether
the
to
tree
determine
algorithm
vertices of G. You need to output only yes or no.

edge

7.75

weights

The following

are hints

with

graphs

weights

negative

to compute

an algorithm

for

but

no negative-weight

no

of the

may be negative
in which the sum of the

weights

cycles

of G

rooted at

only

shortest

single-source

v.

to all

paths in

shortest

cycles. The

a linear-time

Design

paths from

weighted
with an

starts

algorithm

other

the algorithm in that

tree, as in Exercise 7.74. It then applies
arbitrary rooted spanning
the tree is the shortest path
tree.
The algorithm obtained in
exercise to determine whether
in case the tree is not the desired
7.74 should provide some evidence
tree. This
Exercise
is applied until
evidenceis used to make a modification to the tree, and the same procedure
tree.
the tree becomesthe shortest-path
a. Describe
b.

exact algorithm.

detail the

more

in

Prove that the algorithm

terminates after

to improve
c. Suggest a way
not change
need
improvement

7.76

Let G = (V,
Design an

need

a. LetG = (V,

includes

7.79

(The

Algorithm

one (the

a good

\"common\"

tree.

initial

The

case.

whether

determine

path.

The algorithm
Transitive

Improved
outer

one)

vertex of V. Each edge of E is colored

to determine whether G has a simple
\342\200\224
colors
each red (black) edge in the
namely,
such a cycle exists, then the algorithm
should

If

the restriction that

is a simple path

path

to

Hamiltonian

7.80

selecting
the

the cycle

has to

include

the special

undirected graph G = (V, \302\243),

find a spanning tree of G with
of a tree is the maximum
distance from a root to a leaf.)

height

Hamiltonian

algorithm

\\E \\) steps.

algorithm

alternating

(red) neighbors.

without

problem

a connected

height.

v, with

has two black

at least one.

b. Solvethis
Given

by

case, only

\342\200\242

let v be a

graph, and
Design a linear-time

cycle,which
find

7.78

be a directed

E)

or red.

cycle

the algorithm
the worst

\\

weighted directed graph such that some of the weights may be negative.
to determine whether
the graph contains a negative-weight
algorithm
to output only yes or no.

black

either

E) be a
efficient

cycle. You
7.77

O(|

chooses

that

vertex.

minimum

all the vertices of the graph.

Design an
G = (V, E) contains a
acyclic directed graph
in linear time.

includes

a given

should run

in Fig 7.24 has three nested

second one choosesa row,

Closure given
a column, the

loops.
and

The first

the third

one

row. Suppose that we exchange the first two loops such that the first
row
In other words, we simply
and the second one choosesa column.
the
is shown in
first
as
two lines in
the
exchange
algorithm
program,
WRONG Transitive Closure in Fig. 7.46. Show that this modification
does not work,
by
the transitive
closure is not computed.
giving an example for which

operates on the
one chooses a

7.81

chosen

was
for the transitive closure algorithm
(which
Exchanging the order of scanning the matrix
7.80) is desired for the following reason. If the matrix
attempted unsuccessfully in Exercise
is very large and thus cannot be stored
it from
in main memory, we need to access
that the matrix is stored
secondary memory. Assume
by rows such that each row occupies a
page. We want to minimize the number of pages that need to be fetched from secondary

Creative Exercises 259

(A) ;

WRONG_Transitive_Closure

Algorithm

A (an n x n adjacency matrix

a weighted graph).
representing
to the graph, and false otherwise;
{ A [jc, y ] is true if the edge (jc, y) belongs
A [jc, x ] is true for all x )
A represents
At the end, the matrix
the transitive
closure
of the graph.
Output:
Input;

begin

x :=

for

to n

for m := 1

do
to n

for

do
then

A [jc, m]

if

:= 1 to n

if

A [m,

do

y ] then

[jc, y ]

Figure 7.46 Algorithm

:= true
Closure.

Transitive

WRONG

loop scans the matrix by columns, then we need to bring all the rows to
On the other hand, if we exchange the first two loops and we find that
a certain entry
(jc, >>) is false, then there is no need to fetch the >>th row in the next step.
if the matrix is sparse (i.e., if it contains
Therefore,
only a few Is), fewer pages need to be
fetched. The algorithm in Fig. 7.46 is wrong as is, but it can be fixed.
the first

memory. If
look at each

a.

Show

7.82

if we

that,

closure
b.

column.

Show

this

times,

0(\\ogn)

algorithm

the

it computes

then

Let G = (V,

is sufficient

to run

the

only

algorithm

twice.

an undirected
namely,
graph that may have more than one
E
a
and | E | is the total number of
vertices.
is
in
this
case
multiset,
edge
E
+
v of degree 2 by replacing
an
V
O
to
delete
each
vertex
\\
(|
\\
|)
algorithm
edges. Design
the edges (w, v) and (v, w) by an edge (w, w), and to eliminate
multiple
copies of edges by
that removing
them with a single edge. (Note
multiple copies of an edge may
repacing
of degree
has to be removed, and removing a vertex
create a new vertex of degree2, which
which
must
be
removed
2 may create multiple
too.)
edges,

be a multigraph,

E)

a pair of

undirected

connected

if removal
graph G = (V, E) is callededge-biconnected
to
determine
a
linear-time
Design
algorithm

of any

one

whether

edge leaves the graph connected.

graph is edge-biconnected.

7.84

Given

should run

in

Let G = (V,

E)

Prove
of edgesthat
that

7.86

undirected graph G = (Vy

G that contains both
linear time.

a connected

there exists a

7.85

transitive

correctly.
that, in fact, it

between

7.83

run

cycle in

be a connected
intersection

the

forms

undirected graph

of F

a spanning

and
\302\243),

with

the

tree of the

three edges,

b but

and

and

set of edges

does not

let T = (V,

of any

a, b,
contain

F) be a

biconnected

and

c\\

c.

find

whether

The algorithm

spanning

component

tree

of G.
is a set

component.

G = (V, E) is a biconnected
G' = (V, E') such that
graph
=
an undirected
biconnected extension;
graph G (V, E), find the minimum
the minimum
number of edges. (Hint:
Start
that
is, find a biconnected extension with
by
work
and
to
your
way up
general graphs.)
considering very simple graphs,

A biconnected
Given
E<^E'.

extension

of a graph

Algorithms

260

Graph

7.87

Suppose that
Show how

are given

you

undirected

an

biconnected

find the

to

with a list of

graph

without

components

all

resorting

articulation

the

to

points.
whole

the

running

biconnected component algorithm.

7.88

DFS tree

and let T be a
Let G = (V, E) be a directed
graph,
of the edges of T with the edges of any strongly

Prove

of G.

intersection

the

that

connected component

of G form

a subtree

of7\\

7.89

A High

value

serves

linear-time

connected.

a. Prove

the

only
may

appear
an

G=(V,

undirected

connected

theorem:

following

if there

b. Design

is a

in G

circuit

more than
efficient

should find

such

A directed graph
that includes every

once in

that

to

algorithm

Design a linear-time algorithm to

such that the resulting directed graph is
an orientation if it exists.

graph.

be oriented

can

E) is

G =(V,
edge

connected

strongly

if and
an edge

at least

once.

a given

strongly connected

(Note

that

circuit.)
find

a circuit

such

in

graph

\302\243).

BcV
with the
subset
E) is a minimum-size
is a vertex b in B such that there is a path of
a
length 0 or more from b to v. Prove the following two claims, and then use them to design
linear-time
algorithm to find a vertex basis in general directed graphs.

basis of a

A vertex

property

that,

a.

b.

An acyclic

directed

each

for

graph

vertex

not on

that is

vertex

a cycle

G=(V,

in V, there

and has nonzero

directed graph has a unique

vertex

indegree

basis, and

cannot
it

be in

is easy

to

any
find

vertex

basis.

it.

A directed graph G = (V, E) is called unilateral

if, for any two vertices v and w in G, at least
is
one of them is reachable
from the other. In particular,
every strongly connected graph
unilateral.
On the other hand,
there
are many unilateral
not
that
are
strongly
graphs
connected.
For example, a graph that consists
of two vertices connected
by one edge is
but
it is not strongly connected. Design
a linear-time
unilateral,
(and linear-space)
to determine
whether a given directed graph
is unilateral.
algorithm
(Hint: Consider the
connected

strongly

7.94

(based on

the edges of G
The algorithm

whether

strongly

7.93

algorithm

E) be

Let G=(V,
determine

7.92

It

whether a strongly connected component

has been found.
to find, for each vertex v in the graph, the vertex with the
a fixed DFS tree with
DFS numbers) reachable
decreasing

v.

from

7.91

not

\"highest\"

DFS number

largest

7.90

the

an indication

as

only

Design a

by algorithm Strongly_Connected_Components (Fig. 7.33)does

vertex reachable from the vertex under
consideration.

computed

point to

actually

one

simple

given graph

7.95

Design

7.96

Prove the
Given

w is reachable
from v, there is
if, whenever
=(V, E) is calledunipathic
a
to determine whether
from v to w. Design an efficient
algorithm
=
G (V, E) is unipathic.
Solve the problem first for acyclic graphs.)
(Hint:

graph G

directed

only

components graph.)

path

a linear-time

algorithm for finding

alternating-paths

two

matching

them; namely, the

M\\

theorem
and M2,

set of all

edges

a maximum

directly
study

the

that appear

matching

in

a tree.

network flows or cuts.

symmetric difference
in exactly one of them.)

without

properties

the use of

of the

(Hint:
between

Creative Exercises

7.97

G be an

Let

a.

the

Prove

undirected bipartite

graph,

M be an

and let

arbitrary

the

b.

edges

a maximum
matching
following theorem: There exists
M covers.
(A vertex is coveredby a matching M
of M.)

the

Convert

in part a to

proof

an

for finding

algorithm

in G.

matching

vertices that

covers all the

incident to one of

G that

in
if

261

it is

such a maximum

when

matching

G and M are given.

*

7.98

Prove

the

that

7.99

time of

running

improved algorithm

to

and

Hopcroft

7.10) is O

Section

in

Karp

((m +n)w)

in

bipartite

matching

the worst

case.

(the

algorithm

matching in a graph. In other words,

are
looking
edges,
looking for disjoint star graphs, which
trees with one vertex
One edge is a specialcaseof
(the root) connected to all other vertices.
which we do not consider
a star graph, but one vertex
alone with no edges is a trivial
graph
to be a star graph. Let G = (V, E) be an undirected connected graph.
The goal is to design
an algorithm
that finds a collection of vertex-disjoint stars in G, each with at least two
should
be included in one of the stars, but not all the edges need to be
vertices. Each vertex
In other words, the stars should cover all the vertices, but not necessarily
included.
all the

Suppose

want

we

that

edges. (There are no

a.

the

Find

a nonmonogamous

find

for disjoint

of

instead

are

we

constraints.)

or maximality

minimality

in the following algorithm both

error

by exhibiting a

by

pointing

out the

wrong argument

and

counterexample.

is that we know how to

We use induction. The induction
hypothesis
algorithm:
a graph G=(V, E) with
n
the problem for a graph with <n vertices. Given
from the
vertices, we first pick an arbitrary vertex v and remove v with all its neighbors
may not be connected, but we can consider each connected
graph
graph. The remaining
the same algorithm by induction.
separately and apply
component
Wrong

solve

b.

Design

an efficient

7.100 Consider the

G =(V, E) with

(and correct)algorithm

for this

problem.

is a weighted bipartite graph

problem. The input
M to
and m edges. We define the bottleneck
weight of matching
of the maximum-weight
be the weight
edge in M. Design an algorithm to find, among all
run in
should
bottleneck
maximum matchings, one with minimum
weight. The algorithm
time 0(4n m \\ogn).

7.101

following

aniVxAf

Consider

Prove, by using
and one arbitrary
size 2 x 1).

7.102 Prove
the

the

board

bottleneck

n vertices

board of

alternating

black

and white

theorem, that if
then the rest of the

the alternating-path
white

square,

in Exercise 7.101 by finding

in the following way: The vertices

theorem

squares (such as
one arbitrary

a chess

one removes
board

can be covered

a Hamiltonian

cycle

are the squares

and

in

by

square

dominoes

a graph

any

board).

black

defined

(of

by

two neighboring

squares are connected.

7.103 Let G =(V,

find

the

differs

E)

shortest
from the

be a connected

undirected graph.

sequence of trees

previous one by

Given

T0,Tl9...,Tk,such

an addition

of one

edge

two spanning trees T and R of G,

T() = T, Tk=R, and each tree
a deletion of one edge.
and
that

262

7.104

Graph Algorithms
that a round-robin tournament
is played among n
other players. There are no draws,

Assume

once against all n -1

given in a matrix. It

is not

in general

possible

to sort

each player
results of all games

That is,

players.

and the

since A may

the players,

plays

are

B, B may
We are

beat

other words, the results are not necessarily

transitive).
an algorithm to arrange
the players in an
sorting as follows.
Design
- ,Pn
sucn tnat ^i beat P2, Pi beat P3, and so on (concluding with Pn_x
order
P\\,P2,
The worst-case running
time
of the algorithm
of results.
beating Pn\\ given the matrix
should be O(n logn).
time.)
(Any entry in the matrix can be accessedin constant
beat

C may

C, and

interested in

7.105 Given

a \"weak\"

0<d,
<d2
we have dj < dx

integers

2</<\302\253,

vertices of
in

implies,
*

A (in

beat

<

\342\200\242
\342\200\242
\342\200\242

^dn,

+d2 + ' \"

degrees exactly

+ dj_x.

+d2+

' ''

dn is

the

for

every

with

multigraph

which

algorithm

exists.

always

multigraph

even, and,

an undirected

Construct

Prove the correctness of

...,dn.

d\\,d2,

that such a

particular,

such that dx

is an assignment of colors to the edges (one color per edge),

7.106 An edge coloring of a graph
colors.
that two edges incident to the same vertex have different
such
Design an algorithm
to find an edge coloring with k colors for undirected bipartite graphs all of whose vertices
time of the algorithm should be
have degree k such that k is a power of 2. The running
0(|\302\243|log*).

7.107

an undirected
graph G =(V, E) is a set of edgessuch that each vertex in
to
one
is
incident
at
least
an efficient algorithm to find
edge from the set. Design
graph
minimum-size
edge cover for a given bipartite graph.

cover of

An edge
the
a

7.108

7.109

of an undirected graph G =(V, E) is a set of vertices

(J such that each edge
is incident to at least one vertex from U. Design
an efficient algorithm to find a
minimum-size
vertex cover for a given tree. (Vertex coversin general graphs are discussed

cover

vertex

in

the graph

in

Chapter

11.)

Let G =(V, E) be a
each vertex is equal

efficient

Design

an

graph.

(Hint:

7.111 Let G = (V,

of which

to

the

associated with
degree of that vertex.

weights

vertex cover of G, i.e.,a

minimum-weight
7.110

with

tree

Find
E) be an

to
algorithm
a relationship

find

set

in a given bipartite
graph.
11.) (Hint: Find a relationship

An

an efficient

Design

(Independent

in the

cuts

minimum

the

of

weight

to

the

find

weight.
for

a given

bipartite

graph.)
set in G

independent

that

an algorithm

vertex cover

a minimum-size

to minimum

undirected graph.

are connected.

Design

cover with

vertex

such

vertices

the

is a set of vertices no

two

independent
algorithm to find a maximum-size
sets in general graphs are discussedin Chapter

to Exercise 7.110.)

7.112 Design

an algorithm
to find a maximal
in a given
independent set (see Exercise7.111)
undirected graph G = (V, E). The set need not have the maximum size over all independent
sets. It is only
be extended by the
to be maximal in the sense that it cannot
required
addition of more vertices to it and still remain independent.

7.113 Let G = (V,

linear-time

E)

be a tree

algorithm

such

that

to find an

each

vertex

v has

independent set

in

an

weight w(v).
Exercise 7.111) with

associate

G (see

Design a
maximum

weight.

7.114 Let G =(V, E) be a connected undirected graph.

whether
an algorithm to determine
Design
G contains a vertex cover (see Exercise
with at most k vertices, all of which are
7.108)
independent (i.e., no two vertices from the cover are adjacent).

Creative Exercises 263

7.115 Design

an

to determine

algorithm

vertices U,

U is a

that

such

same time. The algorithm

cover

vertex

find such a

should

undirected

an

whether

minimum

set if

and a

graph

E)

G=(V,

maximum

has

a set

of

set at the

independent

it exists.

is an undirected graph whose vertices

7.116 An interval graph
to intervals on the real
correspond
line and two vertices are connected
if the corresponding intervals intersect. Let G =(V, E)
be an interval graph such that the corresponding intervals are known.
an efficient
Design

algorithm to find

independent set in

a maximum

G.

if its vertex set can be partitioned

G = (V, E) is a split
into two
graph
disjoint subsets U and W such that the graph induced by U has no edges and the graph
induced by W is a complete graph (i.e., all the edges are present). Design
a linear-time
whether
a given graph is a split graph.
algorithm to determine

7.117

An undirected graph

7.118

a. Design

an

Can

b.

7.119 a.

to determine whether a given undirected graph

G = (V, E) contains
The running
time of the algorithm
should
be O (| V \\ \342\200\242
\\ E |).

algorithm

your algorithm

Design

square

as a subgraph.

triangle

find

triangles contained as subgraphs

all the

in G?

to determine whether
an algorithm
a given undirected graph G = (V, E) contains
a
as a subgraph (i.e., a cycle of length
should
4). The running time of the algorithm

beO(|V|3).

b.

Improve

algorithm

your

You can use

the

to

run

in time

matrix

adjacency

O (|

V |

\342\200\242
| E |).

representation

or

the

list

adjacency

representation,

whichever is more convenient.

7.120

Prove that there is no algorithm

undirected graph G = (V, E) whose

7.121 Let
each

T be a rooted
vertex,

finds

all squares

time in the

that

are

is a weight
There
tree, not necessarily binary.
of a vertex is greater
than the weight of the
with the minimal
weights satisfy the heap condition

the
be

associated
vertex's

designated

same as its

as either a

regular

vertex

or a

weight

pivot vertex. The

with

parent
on

top).

cost of a

\342\200\224
however
their cost
vertices get discounts,
is
vertex.
ancestor
that
a
Thus,
pivot
weight
selecting a vertex as a pivot vertex may increase its cost, but it will also decrease the costs
of some of its descendants.
vertices.
There is no limit on the number of pivot
Design an
or a pivot vertex, such
efficient algorithm to designate
every vertex as either a regular vertex
is minimized.
that the total cost of all vertices
pivot

vertex

is the

\342\200\242
| E |).

the weight

that

of a given

subgraphs

worst caseis O(| V

directed

such

(in other words,

Each vertex can

that
running

is their

7.122 Let T be a complete

minus

binary

of

the

closest

tree of height h, and n=2h -\\ vertices. We

Each
vertex corresponds to a unique
way.

want

to embed

T in

point (i.e., a
point with integral coordinates), adjacent vertices are connected
by straight line segments,
and
no two line segments intersect.
Embedding
graphs in the plane in this way is an
and especially VLSI design.
in
Our objective
important
problem in integrated chip design
the area enclosing the layout.
this exercise is to minimize
We define this area to be the
lattice
minimum-area rectangle along
that contains the
points (which are not occupied)
with k vertices would be enclosedin a rectangle
of
So, for example, a straight chain
layout.
area is of
area 2(k + 1). It is clear that for any graph with n vertices the minimal
possible
the

plane in the following

Regular

weight.

the weight

size Cl(n).

lattice

264

Algorithms

Graph

a. Describea

layout

completebinary
new

shared

root.

layout of a tree
b.

Design

obtained

T that

for
tree

consists

Assume

of height

an algorithm
in part a.

that

requires O(n) area. (Hint: Use divide and conquer; each

of two smaller completebinary
both connected to a
trees,
know
how
to
embedtreesof
h - 1, and find the
you
height

h.)

to

compute,

for

each

vertex

in

7,

its coordinates

in

the

layout

CHAPTER

ALGORITHMS

GEOMETRIC

paint

objects
Pablo

8.1

as I

think

Picasso

them,

not as I

see them.

(1881-1973)

Introduction
role

of computer science,
areas
VLSI
and
design, robotics,
computer-aided
design,
including
computer
graphics,
circles
or even millions of points, lines, squares,and
There may be thousands
databases.
in a computer-generated
picture; a robot may have to make thousands of moves;a design
involve the
All
these
of
a computer
problems
chip may involve millions of items.
of geometric
manipulation
objects. Since the size of the input for these problemsmay be
for them.
quite large, it is essential to developefficient
algorithms
areas in which
There are two somewhat
arise;
algorithms
geometric
separate
of
them
deals
are
both
One
called
computational geometry.
mainly
unfortunately,
they
with
deals
discrete
with
of
the
other
one
continuous
mainly
aspects
geometric objects;
similar
is not strong, and there are many
of geometric objects. The distinction
properties
will
be
on
and
Our
discrete
problems
computational geometry. In
techniques.
emphasis
this
As in other chapters, the
we discuss several basic geometric
chapter,
algorithms.
that
of this chapter is necessarily limited.
We include
some of the basic algorithms
scope
in the design of more complicated algorithms,
and
that
blocks
appear as building
illustrate interesting techniques. We will limit the discussion to two-dimensional objects.
in this chapters are points,
The
and
line segments,
lines,
objects
appearing
these objects and compute certain propertiesof
manipulate
polygons. The algorithms
and a discussion of data structures
definitions
We start with
them.
basic
used to represent
A point p is representedas a pair of coordinates
the
different
(jc, y) (we assume
objects.
this chapter).
A line is represented by a pair of
a fixed coordinate system
throughout
Geometrical

algorithms

play

an important

in

many

266

Geometric

points

p and

-p-q-.

line

by

p-q.

can

(which

segment

we assumethat
A

Algorithms

the

be any two distinct

is also represented

are the

points

P is a

path

by

points

on the

a pair

of points

segment's endpoints,

sequence

of

we

and

We

denoted

it is

and

and

q, but

denote

the

the
and
Pi,P2\302\273\342\200\224\302\273P/i\302\273

points

them.

line),

by

in this

case
line segment

line

segments

the line

segments
connecting
Pk-\\~Pk
P\\-Pi- Pi~P?>i\342\200\242-\">
last point is the same
as its first point. A
in a path
edges. A closed path is a path whose
The
are also called the
closed
points defining the polygon
path is also called a polygon.
is a polygon
with
three
For example,
a triangle
vertices.
A
vertices of the polygon.
a
set
of
the
rather
than
as
because
as
a
order
is
in
points
sequence
represented
polygon
call

sometimes

will

the order, even without

are given is very important.
Changing
changing
is one whose
result
in
a
different
the points themselves, may
polygon. A simple
polygon
intersect
correspondingpath does not intersect itself; that is, no edges of the polygon
at their common vertex. A simple
encloses a
edges
polygon
except for neighboring
this region the inside of the
A convex
We will
call
polygon.
region in the plane.
line
two
a
such
that
inside
is
the
any
segment
connecting
points
polygon
polygon
A
the
is
a
of
is
itself
inside
convex
polygon.
path
entirely
path
points
polygon
the points

which

p!, p2,...,

that connecting

such

pn

We assume

that

\\

with

pn results

is familiar

reader

the

in

basic

with

a convex

polygon.

analytic

geometry.

For example,

to compute the intersection point

of two line segments, determine whether
need
a
between
two
point lies on a certain side of a given line, and compute the distance
can be done in constant
time with basic arithmetic
given points. All these operations
for
now
roots
assume
that
can
be
time; we
(We
operations.
computed in constant
square

we will

given

discuss

in Section

issue

this

One

8.3.)

numerous \"special cases.\" For

the
unique
point, except when

perform a

of many

characteristic

inconvenient

on

computation

two

example,

two

geometric
lines

in

the

algorithms

plane

is the

usually

intersect

or when they are the same. When we

lines, we need to consider all three possibilities.
special
many other types of specialcases,requiring

given

objects can lead to

care.
most
of these special casescan be handled in a straightforward
Usually,
but the need to consider them
the design
makes
and the description
tedious
sometimes.
We
details that are not
algorithms
occasionally
ignore
the
main ideas of the algorithm.
understanding

We

Whether

Determining
start

with

a simple

This

problem

the

Is Inside a

a Point

manner,

of

geometric

essential

for

Polygon

problem.

The Problem
whether

at

one

are parallel

lines

More complicated

8.2

of

existence

point

may seem

Given

a simple

polygon

is inside or outside the

trivial

at

first,

and

a point

q, determine

polygon.

but when

complicated nonconvex polygons are

8.2
as is

considered,

a problem

the

by

in Fig.

case

hand

8.1,

is always

first

a Point

Whether

Determining

the

is definitely

problem

a good idea.

The

Is Inside

not simple.
intuitive

first

a Polygon

Trying

approach

267

to solve

is to

try

the outside boundary

the given point. When we try this approach
from
of intersections with edges of the polygon
it
is
to
see
that
sufficient
count
the
number
we
the outside
is reached. For example, in Fig. 8.1, going northeast from the given
until
the dashed line in the figure) results in two intersections
with the
(following
point
is
the
first
before
the
outside
reached.
Since
intersection
from
the
outside
brings
polygon
intersection
the polygon and the second
us inside
brings us back outside, the point is
the point is
outside the polygon. In general
(ignoring
special cases for the moment),
of intersections
if and only if the number
described
inside the polygon
(as
above) is odd.
is
in
8.2.
now
have
of
which
a sketch
an algorithm,
We
Fig.
presented
As we mentioned in Section
several
cases that need
8.1, there are usually
special
the polygon, and let L be the line segment
Let s be a point
outside
attention.
connecting q
of
number
to s. We are trying to determine whether
q is inside P according to the
of L with edges of P. The line L, however, may overlap
some edges of P.
intersections
Two intersections?
another one, do we call this an intersection?
When
one edge overlaps
to reach

somehow

do not want to count overlaps as intersections.Another

special
of P. Figure 8.3(a) gives an example
in which
a vertex
case is the intersection of L with
not
and Fig. 8.3(b) gives an
the intersection of L with a vertex of P should
count,
We leave it to the reader to
in which it should
count
as an intersection.
example
characterize these casesand to find how to handle them (Exercise 8.1).
at
we implicitly
assumed that we are looking
In the
of this algorithm,
development
In

this

case,

pictures.

we clearly

When

the

as a
given
is different.
task
is

input

computer application, the

and

we see

intersections)

the

with

polygon

from the

Figure

point

8.1

of

list

For example,

our eyes, it is easy to

This is
to the outside.

Determining

as is

coordinates,

whether

find
not

a point

when

a good
an

easy

we

the

usually

do the

work

path (i.e., one

task, however,

is inside a

polygon.

case
by

in a
hand,

with

few

when

the

268 Geometric Algorithms

(P, q);

Point_in_Polygon_l

Algorithm

Input: P (a simple

en), and q
Inside (a Boolean

e!, e2,...,

Output:

with

polygon

{first attempt}
and
.,/?\342\200\236,

verticesP\\,Pi,

edges

(a point).
that

variable

is set to

if q

true

is inside

and

false

otherwise).

begin

Pick an

q -s ;

segment

0 ;

:=

count

line

the polygon ;

s outside

point

arbitrary

LetL be the
for all

edgeset of the polygon do

{ We assume that
if e{ intersectsL then
is the

nor

vertex

line L overlapping

the
with

intersection

is not

at a

the text}

e{; see

increment count;

if

count

else

is odd

Inside

:=

then Inside

true

:=false

end

Figure 8.2

8.3

Figure

is stored

polygon
to

do

visually,
of

polygon

Fig.

Point_in_Polygon_I.

Algorithm

Special cases for determining

as a

whether

series of coordinates.

but again, it
8.1 is given

at | 368 308 | .)
8.4. Clearly,

is

not

as a

of

when

coordinates

of intersections is easy
coordinates
are given. The

the number

Counting

as easy
list

is inside a polygon.

a point

only

the

in Fig.

8.4. (The given

point

is

now by looking
to solve the problem
This
work is computing all the intersections.
work can be substantially
for
if the line q-s is parallel to one of the axes \342\200\224
simplified
the vertical axis. The number
of intersections
with this special line may be
example,
much more than that with the optimal line, but we do not need to find the optimal
line
\342\200\224
more difficult
a much
(which is actually
see Exercise
8.3), and computing
problem
The modified
each intersection is much easier.
algorithm is presented in Fig. 8.5.

centered
only

at

Fig.

The

the

reader
bulk

is encouraged

of the

8.2

Determining

Whether a Point

Is Inside

a Polygon

269

320.00368.00

380.00
384.00 | 288.00 384.00 | 288.00380.00| 308.00
| 320.00
|
376.00 | 280.00 376.00| 280.00392.00
392.00 | 332.00 364.00 |
| 332.00
364.00364.00| 364.00 352.00 | 256.00 352.00 | 256.00 404.00| 224.00404.00 |

308.00

288.00
332.00 | 352.00 332.00 | 352.00288.00| 224.00
| 224.00 312.00 |
320.00 312.00| 320.00300.00 | 256.00 300.00 | 256.00 296.00 | 328.00296.00|
280.00
328.00
320.00
340.00 |
| 384.00
| 208.00 320.00 | 208.00 280.00| 384.00
240.00
396.00
248.00
248.00
416.00
240.00 340.00 |
396.00
348.00 |
348.00|
|
|
416.00272.00 | 320.00 272.00 | 320.00 256.00 | 448.00256.00|448.00 320.00 |
224.00 | 256.00 224.00 |
432.00 320.00 | 432.00 340.00| 452.00340.00
| 452.00
256.00244.00| 320.00 244.00 | 320.00 248.00 | 248.00 248.00| 248.00216.00|
256.00
224.00
216.00
256.00 |
| 288.00
| 224.00 240.00 | 232.00 240.00| 232.00
272.00
288.00 264.00 | 224.00264.00| 224.00
| 192.00 272.00 | 192.00 416.00|
428.00416.00
| 428.00 384.00 | 416.00 384.00 | 416.00 400.00| 424.00400.00 |
384.00
424.00
408.00 | 384.00 408.00 | 384.00384.00| 400.00
| 400.00 396.00 |
388.00 396.00| 388.00404.00 | 408.00 404.00 | 408.00 372.00 | 352.00372.00|
404.00
352.00
| 264.00 404.00 | 264.00 368.00|
224.00

Figure 8.4 The

P (a

simple

en), and

e!, e2,...,

coordinates.

{ secondattempt}

p {, p2,...,
y0) (a point).
variable
that is set to true
vertices

with

polygon

8.1, given as a sequenceof

(P, q);

Point_in_Polygon_2

Algorithm
Input:

of Fig.

polygon

and edges
p\342\200\236,

= (x0,

Output: Inside (a Boolean

if q

is inside

and

false

otherwise).

begin

count :=

0;

for all

polygon do
intersects e{ then
line x=x0
that the intersection
assume

edges

if the
{ We

e{

of the

Let y{

be the y coordinates
the linex=x0 and et ;

ifyi

if count
else

<y0 then {the

increment count;

is odd

Inside

then Inside

is not

at a

vertex nor

is the

with et }

overlapping

linex=x0

of the

intersectionis below

:=

between

intersection

q }

true

:=false

end

Figure 8.5

As an example, let's try

(368, 308) is inside or outsideof

to
the

Algorithm

determine
polygon

Point_in_Polygon_2.

whether
given

in

Fig.

the
8.4.

point

with

We count

the

coordinates
number

of

Geometric

270

starting at q and going straight down. We need to look

are below 308, whether
edges whose y coordinates
cross 368. Thereare four edges that cross the line:

intersections with
the edges,

a line

at all

280)-(384,

(208,

segment

for those

check,

and

x coordinate

the

Algorithms

280);

(416,272)-(320,272);

(320,256)-(448,256);
Hence, the

and

Comments
hand

by

such

other
is O

algorithm

In

(or eye)

an

The algorithm
performs

this

many

is not

that take constant

operations

two

between

n is

(where

the

time. Hence,the

line
size
total

(n).

a simple

cases,

calculations is

approach

to perform an intersection
computes n such intersections

time

constant

takes

It

plane.

polygon),
time of

running

the polygon.

is outside

point

Complexity
segments in the
of the

224).

224)-(256,

(452,

not

approach

efficient

only simple,

but

for

large inputs.
efficient.

also

In

obtained

a solution

from

originating

some

Starting

however,

cases,
an

with

\"easy-tothat
can be

a good
idea. There are several observations
visualize\" method is always
we could
In this case, by looking
at the picture, we observed that
achieved this way.
to the outside, disregarding
some path from the point
solve the problem by following
that led to the algorithm.
else. This was really
the main observation
everything

8.3
A

set

of

the

of points
points.

in

the

In this

Polygons

Simple

Constructing

on the order
plane defines many different polygons, each depending
a simple
section, we concentrate on finding
polygon defined by a set

of points.

The Problem
simple

There are

a set

Given

of n points

in

the

plane,

connect

them

in

closed path.

methods

several

corresponding to

to

construct

we would

simple polygons.

We

present

if we

method

had to solve it
of C by a
area

probably approach
a large circle C that contains
all the points. Scan the
line originating
Let's
assume for now that
from the center of C (seeFig. 8.6).
rotating
the rotating line never touchesmore
that,
than
one point from the set at a time. It seems
if we connect the points
in the scan, we get a simple
in the order they
are encountered
Let's try to prove this claim. Denote the points,
as they appear in the order
polygon.
the
is chosen arbitrarily). For
first
line,
imposed by
by P\\,p2* >,pn (the
rotating
point
is included
all /, 1 < / <n, the edge p,\342\200\224p,-+i
in a distinct
of
the circle; hence, it does
region
not intersect with
other
this
is
not
to
edge.
However,
any
prove that the resulting
enough
by

hand.

Consider

the

way

this

problem

8.3

Figure8.6

Scanning

Constructing

the

Simple Polygons

271

points.

is simple \342\200\224
in fact, it may not be. The angle
between
some p, and p/+1 may be
to the edge P/-p,+i
in
which
case
the
180
degrees,
region
corresponding
greater
consistsof more than one-half of the circle. Thus, the edge Pj-pi+i cuts into the other
other edges. (To see that it may, we can consider a circle
regions, and it may intersect
that
is centered
somewhere
of
outside of the circle of Fig. 8.6.) This is a good example
in geometrical problems. We have to be
the kind of ''special cases\" that
often
arise
careful to make sure that all cases are considered. (Of course, we must do that for any
kind
is more prevalent in geometric
of algorithm, but this problem
algorithms.)
For example, we can take any three
We can overcome this obstacle
quite easily.
formed
points from the set and choose, as a center of the circle, a point inside the triangle
a segment
that the circle does not contain
by these three points. This choice will ensure
of more than 180 degrees without any points from the set. Another
which is the
solution,
one we will use, is to choose one of the points from the set as the center of the circle. We
will
if
the point z with
the largest
x coordinate (and the smallest
choose
y coordinate,
there is more than one point with the largest x coordinate). We now use basically the
same algorithm.
We sort the points according to their
in the circle centered at z.
position
These
can
be
a fixed line (e.g., the x
the
between
positions
angles
computed by sorting
lines
z
and
the
from
If
to the other points.
two (or more) points
the same
have
axis)
z to the
from z. We then
connect
angle, they are further sorted according to their distance
with
the smallest angle and to the point
with the largest angle, and connect
the other
point
lie to the left of z, the
Since all other points
bad
case we mentioned
points in order.
is still one more special case that
earlier cannot occur. (There
occurs
when all the points
the points
lie on a line; in that case,
will have overlapping edges.)
any polygon through
in Fig. 8.6 is given in Fig.
obtained by this method for the points
The
simple
polygon
polygon

than

8.7.

Geometric

272

Algorithms

Figure 8.7

we

do

this
We can improve
not have to compute

connecting the points.

by the ratios of the
than

It is

the

computing

same

exact

to

share

same principle. First,

only to find the order for

the

angles

same order is imposedby the slopes of the lines (that is,

is easier
to the x differences). Computing
the slopes
y differences
is
no
need
to
the
Second,
(there
compute arctangents).
angles
using

compute

-iPti

Output: P (a simple

(points

the

in

(p {,p2,..., pn
in the plane).
whose

polygon

have the

two points

distances!

is presented

algorithm

Simple_Polygon

Pi,P2\302\273

when

distances

of

the square

compute squareroots. The

Algorithm

polygon.

But the

we can avoid computing

argument,
sufficient

Input:

in two ways, which

We use the
angles.

method
the

a simple

Constructing

there

Therefore,
Fig

same

slope.

is no need to

8.8.

)\302\273'

vertices

are p

\342\200\242\342\200\242\342\200\242\302\273
Pn m some order).
{,p2\302\273

begin

for i := 2 to

n do

and the x axis ;

\\ -ptake an extremepoint instead of
Pi,e.g.,a
point from the set with the largest x coordinate
with the
smallest
(and
y coordinate
if there are severalpoints
same largest x coordinate)}
Sort the points according to the angles a2,..., oc\342\200\236
;
{ break ties according to distancesfrom
p { }
compute

{it

P is

the angle

is sometimes

a,

the

between

line -p

more desirable to

the polygon defined

the

by

list of

points

in

sorted

order

end

Figure 8.8

The running time

Complexity
requires

0(n

logn) time.

Algorithm

of

this

Simple_Polygon.

algorithm

is dominated

by the

sorting,

which

8.4
Hulls

Convex

8.4

set of points

of a

hull

convex

The

273

Hulls

Convex

the points. We would

the vertices should
namely,

listed

be

The Problem

smallest convexpolygon
enclosing
as
a
represented
regular
polygon,
cyclic order.
as the

is defined

convex

the

like

in

hull of

convex

the

Compute

all

be

to

hull

given

in the

points

plane.

Dealingwith
serves,

hull

convex

The

belongs

to the
as

and

many

consequently,

hull

of

if it

as all

numerous

the

the

usual,

we

hull.

as

vertices.

points

been

have

region

are points from the

A convex hull

of the

The convex

polygons.

arbitrary

handling

\"convenient\"

hull

convex

algorithms

start with a

than

smallest

the

is a vertex

A Straightforward

8.4.1
As

sense, as

vertices

points.

is easier

polygons

in some

set.
can

encompassing
We
contain

say that a point

as little as three

hulls have many

to compute them.

Convex
developed

a set of

uses,

and

Approach
inductive

straightforward

approach.

We can

easily

find

the

hull of
hull of three points. We assumethat we know how to compute the convex
< n points, and we try to find the convex hull of n points. How can the Arth point change
extra
the convex hull formed
are two cases: Either the
by the first n -1 points? There
the
is
outside
in
which
the
hull
or
case
is
is
inside
the
convex
hull,
unchanged,
point
point
to reach that point (see Fig. 8.9). So, we
the hull, in which case the hull is \"stretched\"
whether
a point is inside the hull,
need to solve two subproblems. We have to determine

convex

\\pi

Pi>\\
p5

Figure 8.9

-_

Stretching

Pa

a convex

polygon.

Geometric

274

Algorithms

to be able to

are
point is outside of it. These
problems
with
the
now
inductive
straightforward
not easy.
enough
experience
by
is to choose a
right away. The first improvement
approach to try some improvements
is
to
choose
a
rather
an
It
inside
than
one.
the
nth
tempting
point
arbitrary
special point
of
some
will
to
the
hull.
work
be
extend
must
But,
course,
points
hull, so no
required
in some cases all points
to the hull. Another
and
to the hull,
belong
possible
belong
\342\200\224
is an extreme point
which worked well for the previous
choice,
problem,
namely,
some sort of maximal or minimal
point.
with the maximal x coordinate (and
the
minimal
We choose again the point
y
are
with
the
same
maximal
x
Denote
there
several
if
this
coordinate).
coordinate,
points
to be a vertex
of the convex
hull. Thus, the
q is guaranteed
by q. It is clear that
point
to
the
to
We
first
is how
hull
include q.
need
to find the
modify (stretch)
problem
only
vertices of the old hull that are now inside the new hull (p3 and p4 in Fig. 8.9) and to
remove them; then, we must insert the new point as a new vertex between two existing
is a line that
line of a convex polygon
vertices (p2 and p5 in Fig. 8.9). A supporting
at exactly
one vertex of the polygon.
The polygon
thus lies
intersects the polygon
of
a
one
side
line.
Consider
now
the
lines
on
and
-q-pi~
supporting
supporting
entirely
two
vertices
of
the
have
lines
to
(see Fig. 8.9). Usually,
q which
only
-q-p$polygon
lines. (We will ignore
the special case of two
or more
are supporting
points that are on
same
line with q.) The polygon
lies
the two supporting
the
between
and that is
lines,
lines have the maximal
and
exactly the way we want to modify it. The supporting
minimal angles, with,
in
the
x
all
other
lines
from
the
axis,
say,
among
points
polygon to
To
find
these
two
we
to
the
need
consider
lines
from
to
all
the
vertices
vertices,
q.
q
t0
the
the
maximal
and
to
and
minimal
Pi\302\273P2\302\273
(see also
\342\200\242\342\200\242\342\200\242\302\273P/i\302\273
compute
angles,
pick
Exercise 8.4). Once the identity
of the two extreme verticesis known,
the modified
hull
can be constructed. (There
are several
other approaches to modifying
the hull, and this is
not
the best one; we chose it for its simplicity.)
We omit the details
necessarily
we have

and

We

this algorithm

concerning

For each

Complexity
find

the

the

work

solution
running

running

maximal
involved

of the

the hull

stretch

if

the

have

because we will

present

a faster

point, we need to compute

and minimal

angles,

in processing

recurrence

time of this algorithm

time is dominated by

the

is

to delete

point is O
= T(nT(n)

fah

relation

the

and

0(n2).

other

algorithm

anglesto

and insert

(/:), and
\\)

+ 0(n)

The algorithm

we

all
points

have

shortly.

the previous points, to

from
the list. Thus,

already

is 0(n2).

seen

that

Therefore,

also requires sorting,

but

the
the

the

operations.

Gift Wrapping

8.4.2

point by point, we
algorithm? When we extend the polygon
to the
be internal
that
convex polygons containing
may
points
final convex hull.
of
Can
we avoid doing that?
convex hulls
of considering
Instead
subsets
of the set of points,
we can start with
the hull directly.
the whole
set and build
That is, we can start with an extreme point (which
find its neighbors
must
be on the hull),
in the
hull by finding the supporting
and continue from these neighbors in the same
lines,
as the gift wrapping
is known
for obvious reasons. We
way. This algorithm
algorithm
How

can

spend a lot

we improve this
of time building

vertex of

one

start

with

after

neighbor.

The

the

wrap the

and

\"gift,\"

is given

algorithm

in

Fig

the gift by finding neighbor

to work for higher

around

hull

be modified

It can

8.10.

275

Hulls

Convex

8.4

dimensions as well.
The
induction

Induction

With

this

hull.

Instead

the following

of

application

straightforward

(on k):

hypothesis

convex

is a

algorithm

gift-wrapping

hypothesis:

path of

hypothesis,

of

finding

length

of n points in
is part of the convex

a set

Given

that

k<n

the emphasis is on extending

convex
hulls of smaller

hull of

we can
this set.

rather

than on

the

a path

sets, we

plane,

a part

find

find

the

extending

of the

final

convex

hull.

\342\200\242
\342\200\242,
Gift_W'rapping (p x, p 2, \342\200\242
pn) ;
in
set
of
the plane).
(a
Input:
p!,p2\302\273 \342\200\242\342\200\242\342\200\242\302\273P/i
points
P
hull
of
convex
(the
Output:
P\\,pi,
..,p\342\200\236).

Algorithm

begin

set P

to be the

be the
(and the

Let p

empty

set

the

with

x coordinate

largest

if there

y coordinate,

smallest

the same

with

set;

in the

point

are severalpoints

largest x coordinate) ;

AddptoP ;
L be

Let

while P

the line

is not

is parallel to

p which

containing

the

x axis

do

complete

and L
be the point such that the angle betweenthe line -p-qall points ;
(in counterclockwisefashion) is minimal
among

let q

add qtoP;

L := line

-p-q-

p:=q

end

Figure 8.10 Algorithm

Gift

Wrapping.

To add the kih point to the hull, we find the minimum

Complexity
time of the gift-wrapping
among n-k lines. Therefore,the running
0(n2), which is not better than the stretching algorithm.

and

angles

8.4.3
We

now

algorithm

Grahams

maximum

is

algorithm

Scan

show

an algorithm

starts

by ordering

the

to compute

the

convex

hull

in time

to angles, similarly
according
8.3.
Section
Let p x be the point

of simple polygons described in

coordinate (and the minimal
y coordinate,

if

there

are several

other

O{n logn).
to

points

with

points

The

the construction
the
with

maximal

the same

Geometric Algorithms

276

point ph we compute the angle of the line -p\\-p,\342\200\224with the x axis,

according to these angles (seeFig.8.11).We now scan the points in
order
and, as before, try to find the vertices of the convex
the
they appear in the polygon
a path consisting of a subsetof
the
in
As
hull.
algorithm, we will maintain
gift-wrapping
This
will
a
whose
so
far.
be
convex
scanned
convex
the points
path
path
corresponding
is
polygon
polygon encloses all the points scanned so far. (The correspondingconvex
the first and last points
of the path.)
all the
Hence, when
the one formed by connecting
points are scanned, we find the convex hull. The main difference betweenthis algorithm
is that the convex path
we maintain
is not necessarily
and the gift-wrapping algorithm
It is only part of the
convex
hull
convex
hull.
of
the
final
that
the
were
of
points
part
contain
that
are
not
The
on
the
final
convex
far.
so
these
scanned
hull;
may
points
path
will be eliminated later. For example, the path from px to qm in Fig. 8.11 is
points
convex, but qm and qm_\\ clearly do not belong to the convex hull. This discussion leads
based on the following
induction
to an algorithm,
hypothesis.
For each

coordinate).
and

the points

sort

Induction

hypothesis:
to algorithm

according

the first

among

path

Given a set

of

in the

points

Simple Polygon (Section 8.3), we

k points whose correspondingconvex

plane, ordered

can

find

a convex
encloses

polygon

the first k points.

The caseof
points

then
and

can

pk+l

are

by removing

becausethe

points

have

now

be added to

the

obtained

path

to extend

-qm-\\-qm180 degrees (where the angle

than
we

We

the convex

the lines

between

angle

is less

polygon),
convex),
obtained

the

Denote

...,qm.

by P=q\\,q2,

Consider
angle

k = 1 is trivial.

and

existing

qm

the

to

hypothesis

-qm-pk+\\-

is measured

from

&+1

Fig. 8.11).

(see

the

path (since the path

that
qm is inside the

inside
with

first

points.
If the
of

the

it is still

convex polygon
Otherwise, we claim
from
to
and
This is so
/>, addingpk+l
P,
connectingpk+l
top{.
ordered according to their angles.
The line -p\\-pk+\\- is on the

done.

were

from the

(inductively)

Figure

8.11

Graham's

scan.

8.4 Convex Hulls

above, qm

can

qm

Although

8.11shows
qm_x

180

be removed

as well.
For example,
points may have to be eliminated
the
modified
We
defined
must
continue
by
polygon
path.
last two edges of the
until we find two that form an angle of less than
path
The path is then convex,
the hypothesis
has been extendedto k + 1 points,

the

degrees.

be inside

and we are done. The

the

detailed

is presented

algorithm

Graham's_Scan

Algorithm
Input:

other

that

clearly

now

may

checking

first k points. Hence, qm must be inside the convex polygon defined

from />, and pk+\\ can be added. Are we done? Not quite.
can be eliminated, the modified
path is not necessarily convex. Indeed,Fig.

of the

side

\"left\"

277

(p \\,

p2,...,

in

8.12.

Fig

pn);

of points in the plane).

<7i,<72>\342\200\242\342\200\242
><7m (the convex \\\\\\x\\\\oipup2,

(a set
Pi,P2\302\273 \342\200\224<>Pn

Output:

..,pw).

begin

Let p

the point

be

(and

the same

with

Usealgorithm

order

largest x coordinate)
; let

the

the points around

to arrange
order

be P\\,p2,...,

px

;
p\342\200\236

<7i :=P\\
<72

:=Pi

<73

'=P3

{P

the largest x coordinate

there
are severalpoints
if

set with

the

Simple_Polygon

in sorted

m :=

in

y coordinate

smallest

consists

initially

ofp\\,p2,

andp3

3;

fork :=4
while
m

m :=
qm

ton

do

-qm-\\-qm- an^ ~qm~Pk~

the angle between

:= m -1 ;
m

is >

180 degrees

do

+ 1 ;

\342\200\242\"=
Pk

end

Figure

8.12

Algorithm

Graham

s Scan.

The complexity of the

is dominated
algorithm
by the sorting. All the
in
Each
the
set
is
considered
O(n)
point
exactly once in
steps require only
the induction
At
that
the
is
as
to
the
convex
added
time,
always
step
Pk+\\.
point
path.
later (possibly more than once) to verify its inclusion
The same point
will be considered
in the current
We call this phase a backwardtest. The
convex path.
number
of points

Complexity

other

time.

but all these points except for two

test may be high,
(the
still
to
to the convex path)
found
is
are eliminated!
belong
point
spend only a constant time to eliminate each point, and a constant time to add it.
The total running time of the algorithm
O(n)
steps are required for this phase.
O (n \\ogn)
due to the sorting.

involved in a
point and the

backward

that

current

So, we
Overall,

is thus

278

Geometric Algorithms

8.5

Closest

Pair

Supposethat we are given the locations of n objects and we want to check that no two of
for example, to parts
the objects are too close to each other. The objectsmay correspond,
in
a
In this section, we
to
stars
or
to
in a computer
galaxy,
irrigation
systems.
chip,
is
which
of
this
an
of
a
a
variation
discuss
problem,
example
large set of proximity
problems.

of n

Given a set

Problem

The

in the

points

plane,

a pair

find

of

closest points.

Other proximity
problems
closest points to

include

or the k
A

A straightforward
minimal one.

This solution

comparisons.

The

solving

distance,

then

for

the distances from

and

points,

it

to

point

the

additional

efficient

In fact,

these two
for

algorithm

A Divide-and-Conquer

would

of

to

point

of distance
(7(2)=

other

n -1

is the

are identical.

We

must

want

be
the

satisfies

we have already

and

1),

minimum

points

computations T(n)

solutions

straightforward

all

if

However,

point.

case

n-\\

the

the

- l)/2- 1
removing a

by

proceed

the extra

considering

take

to

and

and n(n

computations

induction

the solution

from

obtained

distance

using

n-\\,

= 0(n2).

a more

solution

As a result, the total

number
relation T(n) = T(n-\\) +

recurrence
T(n)

requires

- l)/2

all pairs

between

distances

the

n (n

straightforward

the problem

information

check

is to

solution

the only
checked.

set, the closest

new given point.

in the
to a

Approach

Straightforward

point,

for each point

the closest point

finding,

it, and finding

seen

that

to find

n.

large

Algorithm

two equal parts. The

the problem of n points to
the problem of n - 1
nil points. We assume,
for
that a? is a power of 2, so that it is always possible to divide the set into two
simplicity,
to divide a set of points
into two equal parts. We are
equal parts. There are many
ways
free
to choose
the best division for our purposes. We would
useful
like to get as much
information
as we can from
much
as
the solution
we
want
of the smaller problems;thus,
of that information
seems
to be still valid when
the complete
problem is considered. It
reasonable
to divide the set by dividing the plane into two disjoint parts, each containing
one-half
of the set. After
find the minimal
we
distance in each part, we have to be
concernedonly with the distances between points close to the boundaries of the sets. The
easiest way of dividing
for
the set is to sort all the points according to their x coordinates,
example, and then to divide the plane by the vertical line that bisects the set (see Fig.
lie on the vertical line, then
we divide
8.13). (If several points
them arbitrarily.) We
Instead

induction

of

considering

hypothesis

one point

at

a time,

we divide

stay the same, but instead

we reduce it to two
points,

can

the

set

of reducing
with
problems

into

8.5 ClosestPair

279

d2

\342\200\242
+

choosethis
For

subsets,

the

Identifying

If the

algorithm.

set of

of

work

needs to be performed
simplicity, we concentrate on

The sorting

problems.

points.

the

to minimize

division

The closest pair problem.

8.13

Figure

n points,

\\

and

actual

set has only

and assume
P2,

two

two
that

points,
n is

as described

a power

above.

We

of 2.
find

the

will

points

then we

of

the

smaller

once.
only

finding

closest

the solutions

combining

only

find
We

minimal

be

first

straightforward

distance

their

distance among

divide

directly.
P into two

distance
P2 be d2, and

the closest

in

each

the

from

the

Let P

be a

equal-sized

subset

by

in Px be d\\, and in
without
induction. Let the minimal
distance
assume,
loss of generality,
that
We need to find the closest
distance in the whole set;
d\\<d2.
in
there
is a point in Px with a distance <d\\ to a point
namely, we have to see whether
a
of
that
it
is
to
that
lie
in
we
notice
sufficient
consider
the
First,
P2.
points
strip
only
width 2d i centered
around
the vertical separator of the two subsets (see Fig. 8.13). No
other point can be of distance less than d { from points in the other subset. Using
this
in
worst
can
from
the
we
eliminate
observation,
consideration, but,
usually
many points
can still reside in the strip,
and we cannot \"afford\"
to use the
case, all the points

straightforward

algorithm

for

them.

less obvious observation is that, for any point p in the strips, there is only
a small
number of points
on the other side whose distanceto p can be smaller than
d\\.
This is so because all the points
in each strip are at least d \\ apart.
If p is a point
in the
on the other side with
a y coordinate
strip with y coordinate
ypj then only the points
yq
such
that \\yp-yq \\ <d\\
need to be considered. There could
be at most six such points
on
one side of the strip (see Fig. 8.14 for the worst
As a result,
if we sort all points in
case).
and scan the points
in order, we need to check
the strip according
to their y coordinates,
of its neighbors in the order (instead of against
number
each
against
only a Constant
point
of the algorithm is given in Fig 8.15.
all n - 1 points). A sketch
Another

280

Geometric

Algorithms

\302\253

\342\226\272

di

The

8.14

Figure

Closest_Pair_l

Algorithm

of n points

d (the distance betweenthe

Output:

of six

case

>'

(P\\,P2*\342\200\224*Pn)

(a set
Pi,P2\302\273 \342\200\224<>Pn

Input:

worst

plane).

closest

two

apart.

(firstattempt}

the

in

points d x

in

the

part

points

set).

begin

Sort

the

points

{thissorting

set into
Recursively, compute
Let d be the minimal
Eliminate

Scan

if any

of these

lie farther

to its five

distances

beginning

in

each

two minimal distances

;
separation line ;

d apart

from the

to

y coordinates

than

according
points in the y

remaining
point

minimal

points

remaining

each

the

the

parts ;
distance

equal-sized
of the

that

points

the

two

coordinates ;

only once at

is done

Divide the

Sort the

to their x

according

and

order

neighbors ;
is less than d

their

compute the distances

of

then

update d

end

Figure 8.15 Algorithm

Complexity
sorting

is done

It

takes

0(n

only once.

Closest_P

log a?) steps to sort

We

then

solve

two

air

according to

the

J.

x coordinates,

size nil. Eliminating

takes 0(n log a?)
steps to scan the points

of
subproblems
It then
steps.

but this
the

outside of the strips can be done in O(n)

steps to
sort according to the y coordinates.
inside
Finally, it takes O(n)
the strips and to compare
each one to a constant
of its neighbors in the order.
number
of size a?, we solve
two subproblems of size nil and use
Overall, to solve a problem
0{n log a?) steps for combining the solutions
(plus 0{n log a?) steps once at the beginning
We obtain the following
for sorting the x coordinates).
recurrence
relation:
points

Intersections of Horizontal

8.6

T(n) = IT(nil)
T(n) =

0(n

An

0(n

The

key

0(n log
know

to

how

other

Induction

find

to include

but we

algorithm,

still

takes

so
we

are

the

induction

we have to spend
we

Although

Can we
the closest-pair

too long.

solving

hypothesis

for

the

set of < n

distance and how

to

know

how

sorted according to

the set

output

plane, we

in the

points

the

y coordinates.

to

according

step

is

relation

recurrence

sorting.

Given a

hypothesis:

We have already seen how

this

this

a quadratic

than

step is the sorting of

problem directly, doing
at the same time
problem
we would like to strengthen

words,

the closest

points

of

solution

The reason
hypothesis.
the y coordinates.

induction

the

strengthen

sorting

problem

closest-pair

the

better

in the combining
the sorting
solve

somehow solve the

problem? In

that

verify

Algorithm

here is to

a?) time

1.

that.

than

log n)
idea

T(2) =

is asymptotically

This

log2a?).

better

to do

want

logrt),

the reader to

it to

leave

We

+ 0(n

281

Line Segments

Vertical

and

the minimal

y coordinates.

their

is to

hypothesis

find

to

sort

Hence,

of n

set

the

points

distance

the

thing

only

the

when

if

two

the

points

that

subsets

are sorted

in

each

we need to do to extend
(of size nil) are already

is exactly
of this
sorting
advantage
mergesort (Section 6.3.2). The main
\342\200\224
is
that
we
not
time
we
the
solutions
we
do
have
to
sort
combine
every
only
approach
have to merge. Since merging
can
the recurrence relation
be done in O(n)
steps,
becomes
that T(n) = O(n logn). The
T{n) = 2T{nl2) + 0{n\\ 7(2)= 1, which
implies

sorted.

But, this

revised

algorithm

8.6

is given

in

Fig

8.16.

and Vertical Line

of Horizontal

Intersections

Segments
Intersection

are

problems

applications.

objects, and

we

common

in

are sometimes

We
are

computational

interested
interested only

sometimes

in

geometry,
the
computing

in

whether

detecting

have many
of several
the intersection is

and they
intersection

easier,
nonempty. Detection problems are usually
although not always
substantially
easier.
In this section, we present one intersection
that illustrates an important
problem
of computational geometry. The same
can be applied to other
technique
technique
intersection problems (and
to other
are given as
problems as well), some of which
exercises.

The Problem
segments

in the

plane,

a set

Given
find

all

of n horizontal

the intersections

and

vertical

among them.

line

282 Geometric Algorithms

Closest_PairJl

Algorithm

Input: P\\,P2*
Output: d (the

(a set
\342\200\224<>Pn

the two

between

distance

version }

{&n
improved
Pn)\302\273'
(P\\,p2^ \342\200\242\342\200\242\342\200\242>
in the plane).
of A2 points

closest points

in the

set).

begin

Sort the

to their

according

points

{thissorting

is done

set into
Recursively do the

two

Divide the

sort the points

Merge

Notice

that

supply

the

in

sorted

two

the

we must

beginning

parts ;

equal-sized

distance

each

each

in

part;

to

according

part

y coordinates

their

one sorted list;

merge before we eliminate

lists into

to

set sorted

whole

the

following:

the minimal

compute

x coordinates;

only once at

need

we

to

of the recursion }

level

next

the

distances
d be the minimal of the minimal
;
Eliminate points that lie further than d apart from the separation
Scan the points in the y order and compute the distances of each
Let

to its

point

if any

of these

update

five neighbors ;
distances is less than

line ;

d then

end

Figure 8.16 Algorithm

This problem is
contain hundreds

for

important,

of

are no unexpected intersections. It

elimination. (The hidden-line elimination
the
lines
are not only either horizontal
ambiguity, we call the line segments
is given

problem

Finding
is a

lines

in

Fig.

in the

example,

of \"wires,\"

thousands

is

and

also

A circuit
design of VLSI circuits.
the designer
has to make sure that

is usually

simply

lines

more complicated,

For simplicity,

vertical.)

section.

this

in

may
there

of hidden-line

context

the

in

important

problem
or

air J..

Closest_P

when

there

An example

because
is no

of

the

8.17.

all intersections

simple problem,

which

among either all

is left

the

vertical

as an exercise.

lines

or all the

We assume,for

horizontal

simplicity,

that

there are no intersections betweentwo vertical

If
lines.
horizontal
lines or between two
we try to reduce the problem by removing
at a
one line (either vertical or horizontal)
time, then the removed
all other
line will have to be compared against
lines, and the
algorithm
mn intersections,

resulting
as

will involve

and

the

O(mn) comparisons. In

algorithm

may

require

0(mn)

there

general,

time

just

to

may be as many
them. But

report

an
We would like to find
mn.
may be much smaller than
well
when there are few intersections and
that performs very
not too poorly
algorithm
We
achieve
when there are many.
it by combining two
of our favorite
techniques:
choosing a specialorder of induction and strengthening the induction
hypothesis.
the

number

of intersections

Intersections

8.6

Figure 8.17

The order of
segment)
the

order

that

Intersectionsof
is determined

induction

the plane

\"sweeps\"

in which

intersect

they

of Horizontaland Vertical

not a
line,
by an imaginary line (an infinite
to right;
the line segments are consideredin

from

left

with

this imaginary

keep some information

intersections, we also
so far. This information

will

be

about

helpful

283

Segments

lines.

vertical

and

horizontal

Line

line.

the line

for computing

addition

In

segments
the

next

that

to computing
we

intersections

have

seen
more

efficiently. This technique is calledthe line-sweep

technique.
line be a vertical line that
the plane from left to right.
To
Let the imaginary
sweeps
we sort all the endpoints
of the segments according to their
achieve this sweeping
effect,
x coordinates.
The two endpoints of a vertical
line
have the same x coordinates, so we
of each horizontal line.
We
must
the
two
need
one
of
them.
use,
however,
only
endpoints
are sorted, we considerthem one by one in that order. As usual in
After
all the endpoints
an inductive
we assume that we have already computed the intersections
approach,
and
some additional information,
the previous line segments and have maintained
among
we now try to handle
the next line segment and to update the information. The structure
of the algorithm
at a time in the left-to-right
We consider one endpoint
is thus as follows.
have not yet specified)
to
order. We use the information
so
far
we
(which
gathered
to
find
some
the
information
the
intersections
that
it
and
causes,
process
update
endpoint,
be used for the next endpoint.
of the
The main part of the algorithm is the definition
is
information that we maintain.
Let's attempt to run the algorithm
and to discover what
needed.

One

feature of

knowledge of all
sweeping

line.

Is

the
it

the

induction

intersections
better

line is

to check

which seems natural

to have,
is the
hypothesis,
that occurred to the left of the current position of the

for intersections when

considered?

line is

a vertical

to choose

considered or

When
we
lines
that can intersect it are still under consideration
line, the horizontal
On the other hand,
when
we look at
(since we have not yet reached their right endpoint).
of a horizontal line, we either have not yet
or the right
either the left endpoint
endpoint
them.
about
intersect
Assume that the
seen the vertical lines that
it, or we have forgotten
a horizontal

when

look

at

a vertical

It

seems

better

the

former.

Geometric Algorithms

284

line L (see Fig. 8.17). What

currently at the x coordinates of the vertical
intersections
all
is
to
find
LI Since all
kind
involving
required
line
are
assumed
to
be
left
of
the
to
the
current
intersections
already known,
sweeping
is to the left
line any further if its right
is no need to consider a horizontal
there
endpoint
lines
whose
left
line.
those
horizontal
are to the
Hence, only
of the sweeping
endpoints
line and whose right endpoints are to the right of the sweeping line
of the sweeping
left
consideration
be under
should
(there are six such lines in Fig. 8.17). The list of these
should
be maintained. When L is encountered,it should
be checked for
lines
horizontal
all these
horizontal lines. The important
here
is
that we need
intersection against
point
know that all
L\\ We already
the x coordinates to determine intersectionswith
check
not
that of L. We have to check only
lines in the list have x coordinates that match
horizontal
in
lines
the list to see whether they match the y
of the horizontal
the y coordinates
of L. We are now ready to try an induction hypothesis.
coordinates
line is

sweeping

information

of

Induction hypothesis:Given

list of

k sorted x

coordinatesas described

x coordinates),
we know how
xk being the rightmost
lines that occur to the
intersections among the corresponding
lines
that are to the left of xk.
to eliminate those horizontal

to

(with

We call
right

endpoints

structure

be

whose left endpoints

lines

containing

discussed

under

still

are to the

of the

shortly.

The (k

+ l)th

detected

the

The (k + 1 )th
line

the

reached

3.

to

the set

yet, the line

endpoint

of

a horizontal

than

comparisons,
stretch

against

all

even

if no

line,

hypothesis,

case

which

in

of candidates. As we said,
considered, so we lose no
step thus extends the induction

the set

intersections

lines

are

intersections

is a

left

This

of a

endpoint

of candidates. Sincethe
should

not

horizontal line,
line's

so,

be eliminated,

by

right
the

in

endpoint

part

by

we add

case

has

not

the y

been

above, this is a

of the algorithm.

checking

we

are

hypothesis.

which

arguments

we

We

the y coordinates
coordinates of the

line.

The algorithm is

smaller

the

from

line.

horizontal

endpoint

right

To extend

properway to extend the induction.

The (k + 1 )th endpoint is a vertical line. Thisis the main
can find the intersections
involving this vertical line by
of all the horizontal
lines in the set of candidates
against
vertical

lines

line

the

vertical

when

eliminating

is a

endpoint

simply eliminate

2.

and

consideration

left

The base case for this induction

hypothesis is easy.
need to handle
the (k + l)th endpoint. There are three
cases:
1.

of xk,

candidates.
(These are the
current x coordinate, and whose
are to the right or at the current
x coordinate.)
We maintain a data
the set of candidates. The implementation
of this data structure will
that are

lines

horizontal

the

horizontal

all

report

left

now

The

complete.

of comparisons

number

will

usually

be much

Unfortunately,
algorithm still requires 0(mn)
even
if the actual number of intersectionsis small.
If all the horizontal
from
left to right, for example, then
each
line must be checked
vertical
horizontal
lines, resulting in an 0(mn)
This bad case will hold
algorithm.
mn.

in the

vertical line intersectswith

worst

a horizontal

case,

this

line.

285

8.7 Summary

To

y coordinates

candidates.

Let the

and

yR,

that the

\342\200\224\302\273y/cSuppose
y 1\302\273J2\302\273

order according to

can find

one for

also perform
find

one

only

for

yR.

the

with

problem

yL,

the

be yL

considering

currently
in

be

of candidates

set

candidates are

in sorted

given

order). We

increasing

line by performing

vertical

two

binary

The

yi<yL^yi+\\^yj^yR<yj+\\-

exactly ^/+i,^/+2,\342\200\224.)>_/.
We

then

and

of comparisons
in the
set of

number

is in
\342\200\224..y*

line are

involves

are

set of

the

that

Suppose

search, say, for

original

the

the horizontal lines

lines

y\\,y2*

vertical

the

with

binary

though the

Even

yj.

intersect

that

lines

horizontal

one

and

yL

in

(namely,

lines that intersect

horizontal

the

searches,

horizontal lines

y coordinates

their

the horizontal

of

coordinates

need to minimize
line and those of

of the vertical line we

y coordinates

the

let

and

we
algorithm,
of a vertical

the

improve

between the

scan the

until

y coordinates

can

we

two dimensions, finding

yi+l,...,
yj is
in a given one-dimensional range (in

for numbers
a one-dimensional problem. Searching
a one-dimensional range query. If the numbers
is called
are sorted,
this case, yL to yR)
time for a one-dimensional range query
is proportional
to the search
then the running
the number of items that are found.
time
to sort the
But, of course, we cannot afford
plus
a vertical line.
horizontal lines every time we encounter
Let's review
the requirements.
We need a data structure
that
allows
us to insert
a
to
a
new element,
to delete an element,
and
one-dimensionalrange query
perform
\342\200\224
there are several data structures
for example,
balanced trees
efficiently.
Fortunately,
\342\200\224
that

of elementsin

found.

elements

Complexity
the

handling
one-dimensional
the

8.7

of

to x coordinates requires time

0((m
+\302\253)).
according
+n)\\og(m
the running time for
and delete operation requires
0(\\ogn)
steps,
lines is 0(n log a?) overall.
the vertical lines requires a
horizontal
Handling
where r is
which can be performed in time 0(logrt+r),
range
query,
Sorting

intersections

of

number

O ((m + n) log

is thus

algorithm

(n being
number

insert

each

Since

in O (log n) per operation

insertions, deletions, and searches
to the
the set), and linear scans in time
proportional
in Fig 8.18.
The algorithm
is presented

can perform

the number

involving this vertical

(m + n) + R), where R

line.

The

running time

total number

is the

of

the

of intersections.

Summary

some
sense, geometric algorithms seem less abstract
than,
say, graph algorithms, since
we are used to seeing and handling geometric objects. But, appearances
are sometimes
at small
from
misleading. Dealing with huge number of objects is different
looking
pictures, and we must be careful that the picture that we have in the back of our minds
does not lead us to wrong conclusions. We must deal with many special cases, and make
The algorithm for determining
sure that we can cover all of them.
whether
a point is
inside a polygon
do not think of a polygon
8.2) is a good example. We usually
(Section
In

as being

cases

like

the

one given

in

Fig.

8.1.

Furthermore,

Therefore, special caution

that may occur.

it

must

easy to overlook the special

be exercised
when designing

is

geometric algorithms.
The

techniques

techniques
that

we

for designing
studied in

have

(discrete) geometricalgorithms
the

previous

chapters.

are

Induction plays

similar
an

to the

important

Geometric

286

Algorithms

Intersection

Algorithm

hn)) ;

vm), (hx,h2,...,

v2,...,

((vx,

Input: v!, v2,...,vm (a set of vertical line segments),

and h{, h2,...,hn (a set of horizontal line segments),
Output: The set of all pairs of intersecting
segments.
denote

0y(v/))

I yfi(v/)

bottom

the

of line v,

y coordinates

(top)

begin

all x

sort
V

coordinates

{V

set of

is the

and place

order

increasing

in Q

them

is organized

not empty

Q is

remove the

ifp is the

insert
else ifp

ofhk then

from
left

perform

ofhk then

endpoint

into

hk

is the

Q;

p from

endpoint

right

remove hk
else ifp is the

to the

according

do

endpoint

first

as a balancedtree

horizontal lines }

of the

coordinates

candidates for

are currently

that

lines

horizontal

; it

intersection
y
while

in

0 ;

:=

of a vertical line v, then

a one-dimensional range query
for the range
x coordinate

yB(Vi)toyT(Vi)inV

end

Figure 8.18 Algorithm

The line-sweep

role.
geometric

(one

algorithms

approach is also quite

common.

is

which

presented
Geometric

is common to

on induction,

based

several

The

8.5).
divide-and-conquer
(except for simple ones) seem to
and ingenious data structures
sophisticated
have not covered here any of these special
Section

in

algorithms

many

We

structures.

Notes

Bibliographic
We

was

data structures, and

complicated
been developed
for that purpose.

require
have

data

technique,

Intersection.

have

discrete
literature

seen

small sampleof

geometry.
techniques,

Preparata
examples,

and

15 years.

Shamos

and a

Mehlhorn [1984] and Toussaint

for
The gift-wrapping algorithm
Graham's
algorithm is due to Graham
include

new

relatively

last

the

Reading

[1985]

comprehensive

[1984].
convex

in this

algorithms

geometric

is a

computational
geometry
in this area, spanning

computational
numerous

only a

Further

and

hulls is

[1972]. A

field,

there

chapter.
exists

Several books
and Edelsbrunner

concentrate

due to Chand

and

containing

though
extensive
on

[1987] present

Additional

bibliography.

bibliography

Even
an

Kapur

books

[1970].

268 papers on

Drill Exercises

hull

convex

and other

algorithms

The algorithm
O (n logn)

algorithm

Schorn

and

A probabilistic algorithm
is O(n), was developed by

for

[1988].
time

running

problems of convexity

was

the closest pair is due to

finding
that
uses the line-sweep technique

for

finding

Rabin

[1976]

by Ronse

published

Shamos and

[1987].

[1975].

Hoey

287

An

to Hinrichs,

is due

the closest pair,

(see also Fortune

Nievergelt,
whose expected
and

Hopcroft

A general technique
for
involves the construction
of
[1979]).
proximity
problems
A
Voronoi diagrams.
Voronoi
for a given set of points
is a division
of the
diagram
into regions such that each
contains all points that are closest to one of the
region
plane
from the set. Voronoi diagrams can be constructed
in 0(n
points
log a?) time (Shamos
are
useful
for
a
of
and
[1975]).
Hoey
They
variety
proximity
problems.
The algorithm
for reporting
intersections
vertical and horizontal
lines
among
in
Section
is
due
to
and
The
Ottmann
time
of
8.6)
[1979].
(presented
Bentley
running
is the best possible in the worst case (see Preparata and
O (n log n +/?) of this algorithm
An algorithm for determining
Shamos
whether
there
are any
intersections
[1985]).
an arbitrary set of line segments
was developed
It
[1976].
among
by Shamos and Hoey
also uses the line-sweep
This
can
also
be
solved
a
divide-andproblem
by
technique.
with
the induction hypothesis
and Wood
conquer algorithm
strengthening
(Guting
There
a
of
literature
on
is
intersection
and
the
reader
is
[1984]).
large
body
problems,
referred to one of the books
Exercise 8.16 is from
listed above.
and
Faust,
Bentley,
8.17-8.18
are discussed in Preparata
and Shamos
Preparata [1982], and Exercises
[1985].

Drill Exercises

8.1

when the

8.2

is the

Let S be
different

The

an

arbitrary

simple

first

computing
points.

In PolygonJ
a vertex

of the

set of points

are the
polygons

set 5?

Either

in

8.2)

(Fig.

polygon

to determine whether
complexity of your algorithm?

vertices

whose

8.4

L intersects

line

the

points

the

in

Is there only
plane.
the uniqueness,

prove

plane

one

or

are

unique

all

on

arise

one line.

simple polygon
of two

show

an

(Section

8.4) proceeds by
by the rest of the
it is outside the
that

example

with the same set of vertices.

we presented for computing

algorithm
the supporting lines from an extreme
Suppose

by addressing the special casesthat

or overlaps an edge of the polygon.

n given

an algorithm

Design
What

8.3

Point

algorithm

Complete

that the only

above the hull,

thing

we know

the convex

hull

point to the

about the
or anywhere

extra

hull

formed

point q is
We can

still
try to find the
two supporting lines from q to the hull by computing the angles to all other points, but it is
not clear any more how to select the minimum
and maximum angle, becausethese
angles
Find a method to determine
can be in any range.
of the lines from q to points in the
which
hull is a supporting line.
hull.

8.5

It may be

below

the hull,

that are ordered cyclically

Let P\\,P2, ...,pn be a set of points
is somewhere
inside the convex hull of these points.
center
additional
(without
sorting) on this set of points.

else.

to a circle whose
according
Graham's
scan to work

Modify

288

Geometric

8.6

in a certain order. We used

to a set of points
Graham's scan is applied
algorithm
with
an
Polygon (Fig. 8.8) to sort the points in the following way. We started
was guaranteed
to be on the hull) and sorted all other points
\"extreme\" point p (which
according to the angles between a fixed line (e.g., the x axis) and the line segments
that not every point p can
connecting the points to p. Prove (by showing a counterexample)
S and another point p, not in
be used for that purpose. In other words, show a set of points
the points relative to p (using
the angles from p to the points in the
the set, such that sorting
Graham's
scan does not lead to the correct convex hull.
set) and then applying

Algorithms

Simple

Graham's
Scan (Fig. 8.12) to reject p
by an example, that it is possible for algorithm
value of p. (In other words, the loop can be
in a row, one at a time, for
every
the value of m.)
executed forp steps without
changing

Show,

8.7

points

8.8

of n points

an example

Show

Closest Pair 2 (Fig. 8.16)will

8.9

The

algorithm

The

line

line

take

plane

with distinct x

Q(n log

which

for

coordinates,

algorithm

a) steps.

in the plane, find all the intersections among them.

O (n log n) in the worst case.

segments

in time

Exercises

Creative

8.10

horizontal
should run

a set of n

Given

the

in

input
that

is a set

of n

is parallel

equal-sized

subsets

in the plane and a line. Design a linear-time

points
algorithm
to the given line and that separates the set of given
points
(if a point lies on the line, then it can be counted as being

to find a
into two
on

either

side).

8.11

Let P be a simple

(not

be an arbitrary

point

convex) polygon enclosedin a given rectangle /?, and q

an efficient algorithm to find
a line segment
R
outside
of
that the number of edges of P that this line
such
point
is motivated by the algorithm
for determining
(This
question
necessarily
inside

connecting q to any
intersects is minimum.
whether a point is inside

8.12

Let

or outside

Design

see Section 8.2.)

a polygon;

convex polygon given

by an array of its vertices
whether a given point q is inside
P.
should be O (logn) in the worst case.

P be a

in

to determine

algorithm
algorithm

8.13

R.

Many

convex-hull

asked

to

are based on
algorithms
a convex-hull algorithm

or

are similar

to

cyclic

The

order.
Design
running time of

algorithms.

sorting

an
the

You are

sort. In
to the
be
inserted
and
should
iteration,
point
possibly
current convex hull, which should consist of the convex hull of the points seen so far. The
should be done initially).
points should be considered in an arbitrary order (i.e., no sorting
The algorithm
whether a given
should be based on an efficient data structure to determine
of your
time
point is inside a given convex polygon. What is the worst-case running
to
have
do
not
all
for
all
the
the
details
(You
cases.)
algorithm?
supply
special
develop

one more

each

8.14

Considerthe
with

the

computing

should

that

is similar

to

insertion

efficient

an

be considered

idea of computing the convex hull

by stretching
use of supporting lines (see Section 8.4). Design
the convex hull based on this idea.

the
an

hull

one

O(nlogn)

at a time
algorithm for

point

Creative

8.15

289

Exercises

that you have a black box

of the union of two
that finds the convex hull
disjoint
denotes
the number of points
in
polygons P, and P2 in time 0(\\P]\\ +\\P2\\)(\\P,\\
that uses this black box to find the convex hull of a given set of n
P(). Design an algorithm
in the plane. The running
time of the algorithm should be O (n log n).
points

Assume

convex

8.16

convexhull

A {/-approximate

set of points

P is a convex polygon all of whose vertices

are either inside it or within distance d from it. (We
the distance
of a point from a polygon as the minimum
define
over all lengths of line
the
to
in
the
P
be a set of n points such
Let
connecting
segments
point
anywhere
polygon.)
that the maximal
of any two points in P is X. Design
difference
between the x coordinates
an algorithm
to compute a ^-approximate convex hull of P, which runs in time and space
0(n+X/d).

are

from

P,

8.17 Let P be

such that

all

of a

points

in P

We define the depth of a point p in P as the number

of the convex
(removed)
forp to become a vertex
\"peeled\"
an 0(n2) algorithm
in P. (Notice that the
hull.
to find the depths of all points
Design
finds
convex
hulls and removes them
run for
may
straightforward
algorithm that
0(n2 logrt) time, since all hulls may have a constant number of vertices.)
a set of n

of convex

that

hulls

points

in

need to

the plane.

be

8.18 a. A

another point q if both the x coordinate and y

point p in the plane is said to dominate
of p are greater that or equal to those of q. A point p is a maximal point
in a
coordinate
P if no point in P dominates
it. Design an 0(n \\ogn) algorithm to
given set of points

b.

8.19

8.20

points of a given

all maximal

find

set P

Solve the corresponding problem

for
extended to include all dimensions).

with
three

points.

dimensions

(the

of dominance

definition

is

Let S be a set of points in the plane. For each p e S, we define D (p) to be the set of points
an algorithm
to compute the sizes
in S that are dominated by p (see Exercise 8.18). Design
in
of the sets Dip) for all p e S. The running
time of the algorithm should be O(n \\ogn)
the worst case.
n points

Given

has

them

in the plane,

find

the

pair of points such that the line segment connecting

time of the algorithm should be O (n log n) in the

slope. The running

the maximal

worst case.

8.21

of n points in the plane, represented

as an array of linked lists in the
and
Each
in
has
two
fields:
which
the
X,
array
following way.
entry
gives the x coordinates,
x
linked list of all the points in the set whose
Next, which points to a (nonempty)
are equal to X, sorted according to their y coordinates.
The array is sorted
coordinates
to the x coordinates. Design
an algorithm
to find the closest pair of points whose
according
x coordinates
are either equal or consecutive
in the array.
The algorithm should run in time
in this algorithm?
to compute square roots
Does
O(n) in the worst case. Is it necessary
find the closest pair (without
your
algorithm
any restrictions)?

8.22

The input

The

is a set

in the plane such that all segments are horizontal, vertical,

is a set of line segments
the horizon.
a 45-degree angle with
You are asked to extend the algorithm for
all intersections
and horizontal
line segments to this case
among a set of vertical
reporting
worst-case
without increasing the asymptotic
running time.

or

input

have

8.23 Design

an algorithm

line segments

by

to compute all the intersections

among a set of
a divide-and-conquer approach. The running

using

horizontal

time

and

of the

vertical

algorithm

290

Geometric

Algorithms

should be the same

should be reported
intersections

8.24

8.25

as the

in Section

discussed

algorithm

time

in

+n) + R)

0((m +n)log(m

8.6.

is, all intersections

is the number of

That

(where

found).

is a set of
The input
determine whether any

line

the

to
Design an algorithm
The algorithm needs to output
in the worst case.
time of the algorithm should be 0(n \\ogn)
only yes or no. The running
to
the
horizontal
and
method
vertical
the
but
Use
maintain
case,
(Hint:
similarly
line-sweep
different information.)
A

arbitrary

two

of the

line

in

segments

segments

plane.

intersect.

all of whose edges are parallel

to either the x axis or the
is a simple polygon
the intersection
of two given grid polygons
algorithm to compute
are given by their vertices in a cyclic
common to both of them). The polygons

polygon

grid

y axis.

Design

(i.e.,the

area

an efficient

order.

8.26

The

input

set of intervals
to identify

is a

an algorithm

Design

set. The algorithm

8.27 The input

Can

rectangles.

8.28 The input

8.30

The

intervals

run in time O (n

should

are

which

that are
\\ogn)

is a

set of

all of whose

rectangles

intersection of all

by their two

represented

in the

endpoints.

interval from the

in another

contained

worst case.

all of whose edges are parallel

set of n rectangles
in Exercise 8.26 to mark all the rectangles
time of O (n logn)l
you obtain a running

to find the

algorithm

The

a line,

all

obtained

algorithm

8.29

is a

on

edges are

to the

axes. Extend the

in other

contained

are

that

to the

parallel

axes. Design

an

the rectangles.

is a set of n circles
in the plane. Design
an algorithm
to detect whether
there are
input
not need to
does
any two circles in the set with nonempty intersection. The algorithm
the intersection,
time of the algorithm should
compute
only output yes or no. The running
be O (n logn) in the worst case.
input

is a set

of n polygons,
polygons in

are any two

there

need to

the

compute

intersection,

vertices.

an algorithm
to detect whether
does not
intersection. The algorithm
is the worst-case running
only output yes or no. What

each with k
the

Design

set with nonempty

time?

8.31

convex polygons given by their lists of vertices (in

to compute the intersection of these polygons.
also a convex polygon, should
be represented
by a list of vertices in

The

a
8.32

The input
a linear-time
at least

8.33

The
should

extension

The

a cyclic

order). Design

The output,

algorithm

a cyclic

which

is

order.

convex polygons given by their lists of vertices (in a cyclic order). Design
to compute the union of the two polygons (i.e., the area enclosedby
algorithm

is two

one of the

input

algorithm

8.34

is two

input

linear-time

is a

polygons).

set of

edges are

the rectangles. The

list of vertices in counterclockwise

union

be represented by its
to the skyline problem

is a set of
input
algorithm to compute

all of whose

rectangles

to compute the

of all

in

chapter

in the
triangles
their intersection

parallel
union

axes. Design an
a polygon. It
(This problem is an

to the

is obviously

order.

5.)

plane, given by their vertices. Design an

(i.e., the area common to all of them).

0(n

logn)

Creative Exercises

8.35

8.36

The

input
linear-time

The

set of

there exist four

The input

points

by its list

given

polygon

algorithm to find
is a

input

whether
8.37

is a convex

n triangles

in the

points in

vertices

in cyclic
given

order.

Design

polygon.

an 0(n2
Design
\\ogn) algorithm to
that are vertices of a square.

plane.

the set

of

whose intersection is the

291

determine

is a set of n

in the plane. Design

a polynomial-time
points
algorithm to determine
are k points in the set (for some k<n) that are the vertices of a regular
with equal-sized
(A regular polygon is a polygon
edges and angles.)

there

whether
polygon.

8.38 The input

We are interested in
set of n points all of which have integer coordinates.
of parallel
lines such that all the points are contained in at least one of the lines
in the set. The lines must either be parallel to the axes or have a 45-degree angle
with the
axes. Design an 0(n \\ogn) algorithm to find a minimum-size
set of lines satisfying
these
conditions. Again, the lines must all be parallel,
so, in particular, if one of the lines has a
45-degreeangle with the coordinates, then all of them do.

8.39

is a

a set

finding

divides the plane into two half-planes. The intersection

of any number of half-planes
are convex and the intersection of convex objects is always
convex polygon (half-planes
the intersection of n given half-planes
The problem is to compute
and output it as
convex).
the list of the vertices in the cyclic
a convex polygon. That
is, the output should include
are given
order in which they appear in the polygon. The half-planes
by the linear
an O (n \\ogn) algorithm to compute
this intersection.
that defines them. Design
inequality

line

is a

8.40

position in the plane (i.e., no three points lie on a common

general
are coloredred and n points are coloredblue. Design an algorithm
and
to the red points such that (1) each point has a unique
to match
match,
intersect.
The algorithm need not
(2) none of the line segments connecting matched points
It is not evident that such a matching always
make
use of any graph-matching techniques.
The algorithm
should run in polynomial time. (Hint: Use induction:
but it is true.
exists,
line segment poses no problems;
Try to find a red point and a blue point whose connecting
is divided
if that fails, try to divide
the set of points by a straight line such that the problem

The input
line),

into

is 2n

such

two

points

in

that n points
the blue points

smaller

problems.)

CHAPTER

NUMERICAL

AND

ALGEBRAIC

ALGORITHMS

One
Two
Four

one is

plus

two is four.

plus
plus

Eight

four

is eight.

eight is

A child's

9.1

we

We are

ten.

poem

complicatedarithmetic
very

that

operation,

numbers

large

phenomenon

However,

granted.

we

that

are

have

in

fact

we
these operations that
whether it is multiplication,

with

familiar

so

usually

operation, we are

an arithmetic

perform

for

algorithms

algorithms

more than

Introduction

Whenever

when

two.

plus

the

or large
seen in

good for small

straightforward

sequences of
the

input

previous

chapters

become

inefficient

the

corresponding
or a more

division,

algorithm is
numbers

an algorithm.

executing
take

are

not

always

involved.

the

The

best

same

occurs here as well: Some

when the size of the input

grows.

of an
have done in previous
we will
measure
the complexity
chapters,
that
of
the
the
number
executes.
For
the
most
\"operations\"
algorithm
algorithm
part
by
we will assume
that basic arithmetic operations (such as addition,
and
multiplication,
is
a
This
time.
reasonable
of
when
the
take
one
unit
division)
assumption
operands can
be representedby one or two computer words (e.g., integers that are not too large,
real
There are cases, however,when
the
numbers).
single-precision or double-precision
In
we
have
to
such
take
2000
into
account
are
cases,
integers).
digit
operands
huge (e.g.,
As we

Algebraic and

294

Numeric Algorithms

operands, or at least to be aware that the basic operations are not simple.
but are in fact very
that look very efficient \"on
to
paper,\"
design algorithms
It is possible
the
are
of
the
sizes
because
ignored.
operands
inefficient,
sometimes. Given an integer
is confusing
of the \"size of the input\"
The
meaning
it
is
natural
to think of the value
an arithmetic operation,
to perform
n on which we want
to our usual convention of using
this is contrary
the
However,
n as the size of the input.
The
distinction
for
its
size.
is
of
the
important.
defining
very
input
requirements
storage
even
can be done quickly,
numbers
hand,
by hand. On the other
Adding two 100-digit
cannot
done
in
a
be
reasonable
number
value
a
to
time
100-digit
by
represented
counting
n
can
be
Sincea
number
fastest
the
bits, its
even by
represented by riog2Ail
computer.
that
an
For
size is defined as |\"log2A2~|.
O(logAi)
operations
example,
algorithm
requires
for computing
when n is the input (for example, an algorithm
2ri) is considered linear,
of the input, whereas an algorithm
size
is a linear function of the
that
since 0(\\o%n)
the

of the

size

O(V>0 operations

requires

numbers less than

As

We

algorithms.
what

is probably

factoring

n by

trying

for

designing

all

in
how

finding

matrix
beautiful

the oldest

this
to

chapter
compute

on interesting techniques
powers

of a

given

number.

known

We

then

Euclid's

nontrivial

common

the

2200-year

We

(for example,

algorithm:
algorithm for
that
divisor.
It
is
modern
greatest
computers use a
quite amazing
for polynomial
old algorithm.
and
We then discuss algorithms
multiplication
and most
with one of the most
and we end the chapter
multiplication,
important
\342\200\224
the fast Fourier transform.
algorithms

present

9.2

concentrate

first discuss

is the input

is considered exponential.

to

or equal

we

usual,

when

Exponentiation
with

start

The

a basic

Problem

arithmetic

Given

operation.

two

positive

integers

n and k,

compute nk.

nk=n-nk~l.
reduce the problem
since
to that of computing nk~\\
the problem can be solved
the
and
induction
on
k,
straightforward
resulting
by
algorithm is given in Fig. 9.1. We have reduced the value of k, but not its size. The
k iterations.
Since the size of k is \\og2k, the number of
straightforward algorithm
requires
interation is exponential in the size of k (&= 2log2*).
is not bad for very small values
This
of k, but it is unacceptable for large values of k.
this
Another
to reduce the problem is to use the fact that nk = (nk/2)2. With
way
k
to one with
we reduce the problem
n and k/2.
observation,
Reducing the value of by
half corresponds to reducing its size by a constant. Thus, the number
of multiplications
\342\200\224
will
be linear
in the size of k. We now have the skeleton of the algorithm
repeated
=
k
for
2J
is
The simplest case
for some integer j:
squaring.
We

can

Therefore,

easily

9.2

295

Exponentiation

(n, k) ; {first attempt}

k (two positive integers).

Power

Algorithm

n and

Input:

Output: P (the

of nk).

value

begin

P:=n;

fori:= 1 tok-\\
P :=

do

n*P

end

Figure 9.1

nk =

But

n2' =
if k

what

j times.

is

not

with

two parameters

This

reduction

is

a power

of 2?

and

k, and

not

if k/2 is

an

not

Consider again

reduced
valid since

always

problem does not

the reduced

integer,

Power.

Algorithm

then

integer,

the

be

not

may

the conditions

satisfy

is

(k-\\)/2

to a

problem

k/2
an

we just used. We started

smaller one with n and k/2.
an integer. If k/2 is not an
But
of the original
problem.

reduction

the

and

integer,

the following

is

reduction

appropriate:
\342\200\236*=\342\200\236L(*-l)/2

We

we

odd,

square

Complexity
progresses,
more costly.

algorithm

We leave it
of

an application

Encryption

of

this

n.

by

is 0(\\ogk).
the

the

in which

algorithm

the

As

the

become

complexity

of the

multiplications.
the numbers do not grow

cost

of

algorithm

multiplications

(Exercise 9.12) to analyze

measure for

is

9.2.

in Fig.

Therefore,

for k/2. If
The number

of this

We
during

now

the

algorithm.

Application

The study

realistic

a more

larger.

the solution

multiply

is given

multiplications

the reader

to

and

(k-\\)/2

become

numbers

the

square

simply

The algorithm
of

number

The

execution of the
An

most 2\\og2k.

however,

under

for

solution

the

is at

multiplications

present

If k is even, we

have an algorithm.

now

to Cryptography

cryptography

schemes

usually

is beyond
rely on

exchange secret messagesmust

use secret keys known
only

the

of this

book,

on the

two

and

we

discuss

participants

it briefly.
who want to

and must
algorithm
encryption-decryption
We want
this need to exchange
to avoid
of participants. The following
is known
as the RSA
Adleman
who
Rivest, Shamir, and
(after
[1978],
of participants
be used by a group
(e.g., computer users)
agree

to themselves.

secret keys between

pair
every
scheme
public-key encryption
developed it). The schemecan

scope

complete secrecy. Any

296

Numeric

and

Algebraic

Algorithms

Algorithm Power_by_Repeated_Squaring
n and k (two positive integers).
Input:

Output: ^(the value

k)

(n, k

div

(n,

of nk).

begin

ifk = 1 then P := n
else
z := Power by Repeated
if k mod 2 = 0 then
P :=

Squaring

2)

z*z

else

P := n*z*z
end

Figure

want

to communicate

one for

and

who

encryption

9.2 Algorithm

Power_by Repeated Squaring.

by encrypted messages. Each participant

for decryption (independent of

one

has

two keys,
of other

only

number

the

A participant
P in the RSA scheme
participants). These keys are chosen as follows.
selects two very large prime numbers p and q and computes their product n =pq.
He then
chooses another
d and (p-\\)(q-\\)
have no common
very
large integer d> such that
divisor. (See the next section for an algorithm to verify
that
fact; if d is a random
then the condition above is likely
From p, qy and dy it is possible
to occur.)
number,
not easy)
to compute the value of a number e that satisfies
(although

(9.1)

e'd=\\(mod(p-\\)(q-\\)).

As we

be the encryption
key and d the decryption key. The values
can read. (We assume
publicized by P in a central directory that everyone
the
of a trusted directory such that no other person can forge P's keys.) The
availability
value
of d, as well as the values
of p and q, which
are not needed
anymore, are kept

of n

secret

see

shall

and

next,

will

e are

by P.

Let M

message

can

be an
be

integer

translated

that

to a

to a messagethat
of
bits, which can
sequence

corresponds

Assume
that M is smaller than n\\ otherwise
M can be broken into
each smaller than n. The encryption function EP that P uses is very
EP(M)

= Me

= Cd

several

to
small

an

(every

integer).

messages

simple:

(modn).

Since both n and e are made public, everyonecan

The decryption
function
DP is just as simple (but
value of dis secret):
DP(C)

to encrypt

wants

be translated

(mod n).

encrypt
it can

messages

be performed

and send them to

only

by

P> since

P.
the

9.3 Euclid's Algorithm

297

are
valid
DP(EP(M)) = M, hence these
thus
of
consist
one
algorithms
computing
only
and
or
one
division
the
these
are
(Me
Cd)
(for
power
congruence), although
operations
on
numbers.
The
n
modulo
can
be
at
performed
very large
operation
applied
any step of
and
not
the
at
the
end.
This
is
true
because
algorithm,
necessarily

can

One

prove that
and decryption

encryption

for all integers x,

n.

and

y,

is very

need

to

the modulo
way

the

of

step of the
grow

in each

operation

values

the

do not

operands

by Repeated-Squaring

Power

algorithm
to

multiplication

this

of Fig. 9.2, not only do we

and
divisions
for computing
the
0(\\o%d))
multiplications
and
division
involves numbers that are less than n. We

multiplication

modify

Applying
Power

(or

0(\\o%e)

only

but each

power,

Both

important, since

above n. If we use algorithm

require

that

guarantees

functions.

= [^(modA2)-y(modA2)](modA2),

x-y(xnodn)

computation

(9.1)

by

by

Repeated-Squaring

modulo n. Thus,

a multiplication

the

applying

only

changing each
requires

scheme

RSA

that are less than n.

a very large number (e.g., of 1000
in a reasonable
time (e.g., our lifetime). Thus, the knowledge
of the value of n
digits)
does
not imply the knowledge of p and q. It is commonly believed (although
there
is no
known proof of this fact) that it is impossible to compute the function
DP efficiently
without the knowledge
of any one of d, p, or q} Therefore,
d,
by keeping
p, and q secret,
P can receive encrypted messagesfrom
without
the
anyone
secrecy of the
compromising
There
are
of
several
other
this
which
is
called
a publicscheme,
messages.
advantages
O (log

only

n) multiplications and
is no known
algorithm

There

key cryptosy

the

of two positive integers

k
such that (1) k divides
positive
integer
divide
both n and m are smaller than
k.

unique

integers

that

As usual,
or m

we

divides
GCD(a? -m,
1

Find the

m, denoted

and

both

and

greatest common divisor of two

by GCD(az,

m),

(2) all other

m, and

given

integers.

smaller

If k

divisor

Problem

The
positive

of numbers

can factor

Algorithm

The greatestcommon
is

that

stem.

Euclid's

9.3

divisions

try

to

reduce

without
both
m),

changing

the

m, then it
we now have

n and
and

the problem to
problem?
divides

one of smaller
their

difference!

a smaller problem. But,

Can

size.

Euclid noticed

the

If n >
again,

we somehow

obvious

m,

positive
then

GCD(h,

we reduced

make n
answer:
m) =

the values

for computing d from n and e would lead to an efficient

probabilistic
algorithm
that d cannot be compromised
n, which is a strong evidence
(see Bach, Miller, and Shallit [1986]).
the knowledge of d.
there may be another way to compute DP without
Potentially, however,
It

for

is known that an algorithm

factoring

and Numeric

Algebraic

298

Algorithms

to be efficient, we must
and not their sizes. For the algorithm
aw =24,
and
we will
For example, if n is very large (say 1000digits)
This
will
24
from
times.
take
n
az/24
subtract
to
need
computation
O(n)
approximately
in the size of n.
is exponential
steps, which
m from
n and apply
the
Let's look at this algorithm again. We subtract
same
If
we
In other
n
m is still larger
than m,
subtract
m again.
algorithm to n m and m.
the result
m. But this is
m from
n until
becomes less than
words, we keep subtracting
Division can be done
exactly the same as dividing n by m and looking at the remainder.
which
is presented in Fig. 9.3.
leads directly to Euclid's algorithm,
This
quickly.
of the

numbers

in

question,

sizes.

the

reduce

We claim

Complexity

n+m;

its

specifically,

the size of the

is O

operands)

changed

of algorithm
a mod A3. Then,

b and

into

and

A3 mod

But,

since

(a mod

a >

in two

So,

A3).

have

we

A3,

a mod

in a

In

GCD.

in the

next

iterations,

A3

< all,

the

claim,

the first iteration,

iteration,

first

sufficient

they

number

which establishes

the

are

a is

to

show

Let's look at

of iterations.

number

constant

of

independent

step

it is

and

changed

(a>b)

two

are

into a mod A3

changed to a mod

A3.

claim.

Multiplication

Polynomial

n-\\

n-\\

Let P

half

by

this

size of

in the

time

running

each operation as one

(log (n+m)). To prove

iterations

consecutive

9.4

running

has linear

algorithm

time (counting

of a is reduced

that the value

Euclid's

that

represented by

Q =

and

Pix'*

list of

its ordered

GCD

Algorithm

Input:

two polynomials

of degree n

coefficients.

(m, n)
n (two

and

Qi*1

Output: gcd (the

gcd

positive integers).
of m and n).

begin

a :=max(
b :=
r :=

n, m

);

n, m )

min(

1;
{r is

r>0do

while

r :=

a mod b

a := b
A3

:=

gcd :=

the

remainder}

r;

end
Figure 9.3 Algorithm

GCD.

- 1.

A polynomial is

9.4

The Problem

PQ =

We can

then

number

seen

it is

that

algorithms

algorithm

coefficients

the

have

We

better?

not

+ l+P/J-2<7/+2

+ P/ + l<7/i-l

so

surprising

is positive.

the answer

in Section

9.6.

But

we

first,

quadratic

straightforward
A

0(n

complicated

a simple

describe

if we

that,

Can we do

be 0(n2).

will

of

improvements

many

that

to see

is easy

It

+ Po<7o-

\342\226\240*

logAi)

divide-and-

algorithm.

conquer

we assume that
parts. Let P = P, +xnU

For simplicity,
two

+ '\"

(9.2)

of PQ directly from (9.2).

of multiplications and additions

by now

discussed

be

will

P/J-l<7/

the

+ <7o

\\<ln-\\XH-X +

+p0

compute

(9.2),

\342\200\242\342\200\242\342\200\242

pn.xxn

Pn-tfn-\\X2\"-2

follow

polynomials of

of two given

product

- 1.

degree

the

Compute

299

Multiplication

Polynomial

equal-sized

\"'

P\\=P0+P\\*+

a?

is a

power of

2.

We

each

divide

+Pnl2-\\*

and Q

into

polynomial

,n/2t
=Q, +xnu

Q2, where
. P2=Pn/2+Pn/2+\\*+ \"' +Pn-\\*
P2,

/i/2-l

and
/i/2-l
*\342\200\242\342\200\242

<2l=<7o+<7l*

+<lnl2-\\X'

(?2=<7/i/2+<7/i/2+1-* +

/i/2-l
\342\200\242\342\200\242'
+<ln-\\X

We now have

PQ =
The

(/>i+/V/2)(Qi+Q2*\"/2) = PiQ\\+(P\\Q2+P2Q\\)xn,2+P2Q2xn.
for

expression

compute

the

product

PQ now involves products of polynomials

of the smaller polynomials (e.g., P\\Q\\)

of
by

degree

induction,

n/2.

We can

then add

the

The only constraints are

the solution. Can we use induction
directly?
same
as
the
the
be
that
the smaller
original problem, and that we know
exactly
problems
The
total
are clearlysatisfied.
1.
Both
conditions
of degree
how to multiply
polynomials
is given by the following
for
this algorithm
number of operations T(n) required
recurrence relation:
results

to complete

7\302\273

The factor
from

+ <9(A2),

= 47>/2)
4 comes

adding

(see Section

the

ni)=l.

4 products of the smaller

of
polynomials. The solution
which
means that we have not

from the

smaller

3.5.2),

and the 0 (n)

polynomials,
relation is
this recurrence
achieved

any

comes
0(n2)
(see

improvement

Exercise 9.4).
To

get

an improvement

four
solving less than
reason we use such an

next

section).

to

the

subproblems.
table
elaborate

we need to solve the problem

algorithm
Consider the following
table
multiplication
for this simple notation
will become
apparent

by

quadratic

(the
in

the

Algebraic and Numeric

300

Algorithms

We

have
the

A+

to compute

want

three

only

by

relation

anyway.

Qi

we need

and

D,

contribute

which

only

O(n)

to

to

need

we

Hence,

E. All the rest

can be

recurrence

the

is

relation

recurrence

new

only

do not
compute

is that we

their sum! If we

know

to

polynomials: A,

and subtractions,

The

observation

important

then B + C = E-A-D.

of smaller

products

additions

computed

Q\\

+Dxn. The

(B + C)xn

C separately;
to compute B and
= (P { +P2)
E
(Q\\ +Qi),
product

compute

Pi

P\\

T(n) = 3T(n/2) + 0(n),

which

7\302\273

implies

Notice that

0(A2,og23) =

the

0(n L59).

are

and Q\\+Q2

P{+P2

polynomials

coefficients
They are formed by adding
a nonintuitive
This is quite
differ
way to multiply
polynomials,
by n/2.
reduces the number of operations significantly for large n.

polynomials

a strange

in

the

original

whose

way.

yet this

indices

algorithm

9.1

\342\226\241
Example

Let P

to

related

-jt

and Q = 2+jt-jt2 + 2*3. We compute their product

We carry the recursion
algorithm.
only one step.

+ 2*2-jt3,

divide-and-conquer

using

the

= 2-x-x2J

A=(\\-x)-(2+x)

(2-x)-(-\\+2x)

= -2

(3-2x)-(\\+3x)

= 3 + lx-6x2.

+ 5x-2x2,

and

E =
From

\302\243,
A,

and D,

B+C = 3
Now, P

A +

(B + C)xnl2

P Q =2-x -x2 +

Notice
that we used
and 12 additions
and
to

a constant

multiplications

have

+ 3x3

3x2

-3x4 -2x4

+ 5x5

12 multiplications

-A -B:

we

instead
carried

n is

large.

when

factor

the

had

of

is reduced by

that

in

about

the

-2x6

5x5-2x6.

to 16

compared

subtractions

9 if

course, much larger

within

=E

and we

+ Dx\\

+ 2x2 + 3x3-5x4+

= 2-x

multiplications

B +C

+ 3x-3x2.

\342\200\242
=

we can easilycompute

(The

of 9.
number

straightforward

a?04.)

could

(We

recursion

in

the

straightforward
have reduced

one more
of

step.) The

algorithm,

the number of
savings

are,

of

and subtractions remains

of
algorithm, whereas the number

additions

9.5 Matrix

Multiplication 301

9.5 Matrix Multiplication

The

C of

product

two

x n

A and

matrices

B is defined as follows:

(9.3)

5>,**V

Cij=

The Problem
of real numbers.

The

Compute the

way (and

straightforward

C =A

product

seemingly

the

only

xB of

way)

two

n x

to compute

n matrices

matrix

multiplications and (n

product

is to

additions.
Notice
that
follow (9.3), which
\\)n2
using n3
requires
of rows and columns in the matrix,
rather than the size of the
n represents the number
which is n2. We now
two different schemes that show
the possibilities
for
present
input,

improvements.

Winograd s Algorithm

9.5.1

for

Assume,

simplicity,

that n is

even. Denote
nil

nil

Ai =

Z
k

After

ai,ik-\\

'ai,2k>

and

Bj=

= \\

rearranging

\302\243
b2k-\\j'b2kj.
k = \\

terms, we get

nil
\302\243
(ai,2k-\\+b2kj)'(<*i,2k
k=\\

C'J=

+ b2k-\\j)-Ai-Bj.

To compute
and BjS need to be computed
only once for each row or column.
of
The
total
number
n2 multiplications.
multiplications
requires
only
BjS
of additions
has increased by about
has
thus been reduced to Vin3+n2. The number
in cases
where
Vm3.
This algorithm is thus
than the straightforward algorithm
better
more
than
additions can be performed
multiplications.
quickly
the /4,s

But

all

the

Afi

and

This algorithm
Comments
make a difference, even for
simple form. The next algorithm
Strassen

9.5.2
use

that

expressions,

carries

order of the

the
rearranging
such as matrix

this idea much

multiplication,

can

computation
which

have

farther.

s Algorithm

the divide-and-conquer
9.4. For
algorithm in Section
We

shows

method

in

a way

similar

simplicity, we assume that

to
a?

the

polynomial

is a

power of

multiplication

2. Let

302

\"^

A\\2

1.1

the

#1,1 #1,2
and

#2,1

C =

#2,2

\302\243/>7s,

^2,1 ^2,2

matrices.

C/jS are n/2xn/2

the problem to

and

AjjS,

Algorithms

B=

^2,2

^2,1
where

Numeric

and

Algebraic

We

can

use

the divide-and-

computing the Ctjs from the Ajjs and the

as elements and consider
nllxnll
submatrices
the whole
2x2
matrices of elements. (We have to
a product of two
substitute
elements for submatrices; this is the subject of Exercise
to an n x n product
for the 2 x 2 product can be converted
algorithm

conquer approachand reduce

That is, we can treat the
BjjS.
problem as one of computing
be careful

9.23.)

we

when

The

algorithm

by substituting a recursive call

for multiplying
two
algorithm

a product

time

each

of elements appears.The

2x2 matrices uses8 multiplications.

regular

Substituting

each

by a recursive call, we get the recurrence relation T(n) = $T(n/2)+ 0(n2)y

og2
= 0(n3).
This is not surprising
since we are using
which
)
implies that T(n) = 0(n
If
we
2x2
could
the
of
two
matrices
with
the regular
algorithm.
only
compute
product
multiplication

less

we would get

8 multiplications,

than

an

algorithm

that

is asymptotically

faster

than

cubic.

is how many multiplications

are required
2x2 matrices. The number
of additions
is not as
since
which is not a
relation,
they always contribute 0(n2) to the recurrence
important
in determining the asymptotic
factor
(It does affect the constant
factor,
complexity.
Strassen
found
that
7
are
sufficient
to compute
the product of
however.)
multiplications
2x2
two
matrices.
Instead of simply writing
down
the equations
leading to Strassen's
we sketch
a method that could
have been used by Strassen
to find it. This
algorithm,
The

most important

to compute

part

product

of

method can be used for

similar

the

eg

c d

c d

00a
write

the

product

r
\342\200\224

OOrrf

We

00

b 0

'a

problems.

P s
=

to computing

is equivalent

the recursion

the product

Computing

'a b

of

two

(9.4)

multiplications

as

AX

required

Y.

are

We

to evaluate

Y.

easy to compute.
two

of

which

Let's

look

As
it turns
out, we need four
are very similar). They are as follows:

Product

Type

the number of
to minimize
ways
that are
for special matrix
products
such
of
special products (the last
types

for

looking

No. of

Multiplications

a a
a)

a a

l
a(e+f)\\

9.5

a
P)

a-b

-a(e+f)
ae

ae+b(f-e)

a b-a
5)
We

+ bf

a(e-f)

bf

for ways to divide the general

listed above.
Since these
types

matrix

look

now

steps of

the

we may be

of multiplications,

number
trial

to reach

error

and

even clear,

~b

B =

it is

but

the

less

somewhat

able to

0 0 0

use

in

less

at the

something

This process is

hardly

(9.4)

into

than the

end.

It

takes

several
nominal

a lot

of
or

straightforward

Let

0 0

a-b

0 d-b

c-b 0 0 c-b
=

given

0000 , c 00cc
00c c
0000
0

save

magic.

0 0

than

product

types of products

combinations.

right

0 0

b b 0

303

Multiplication

a(e+f)

-a -a

Y)

Matrix

b-c 0 0 b-c

E =

and

0 b-c

c-b 0 a-c 0

0 0 0 d-c

0 00 0

All
the products
BX + CX +DX+ EX.
above, except for EX, can be computed with one multiplication
using types a or p. The
= F + G, such
\302\243
two matrices
is to compute EX.
But E can be divided into
only
problem
F is of type y and G is of type
that
8:
A=(B

Then,

+ C

+ D+E)

0 0
0 0 0 0

a-b
F =

c-b 0 a-c 0

0 0 0 0
So,

9.10).

0 d-b 0 b-c
G

0 0 0 0
0 0 0 d-c

(B +C +D +F + G)X, and we need two products

of 7 multiplications
each of types p, y, and 8, with a total

overall,

product

AX =

therefore

and

AX =

a,
(see also

of type

and

one

Exercise

Numeric

and

Algebraic

304

Algorithms

the original
7 products of matrices of half
size, and a constant
less important
are
than the products,
of matrices. The additions
in
time
which
is basically a
two
nxn
be
done
of
matrices
can
addition
0(n2),
because
the
is
not
The
term
dominant
factor in the
size
of
the
matrices.
O
the
in
time
(n2)
linear
=
+
The
this
solution
of
which
is
recurrence
relation,
T(n) lT(n/2) 0(n2).
recurrence
If we use the derivation
is approximately
is T(n) = 0(nlog2 ), which
0(n2S[).
relation
to reduce the
18 additions
above, we obtain
(see Exercise 9.10). It is possible
described
to 15 (Winograd
but this reduction does not change
of additions
the
number
[1973]),
use

We

Complexity

of additions

number

running time.

asymptotic

Comments
1.

are

There

Empirical

studies

algorithm

faster

major drawbacks to Strassen's

that
n needs
to be at least

three

algorithm:

indicate

the

than

straightforward

0(n3)

stable than the

straightforward

100

to make

Strassen's
Roth

and

(Cohen

algorithm

[1976]).

Strassen's algorithm
similar errors in

2.

the

is less
the

algorithm

will

algorithm.
probably

create

much more

is obviously

complicated and
Strassen's

Furthermore,

algorithm.

harder

and

inversion

determinant

computation.

We

will

see

to implement

algorithm cannot be

whereas
the regular algorithm can.
easilyparallelized,
is important.
It is faster
Nevertheless,Strassen'salgorithm
algorithm for large az, and it can be used for other problemsinvolving

matrix

That is, for

larger errors in

output.

Strassen's algorithm
than the straightforward

3.

Strassen's

input,

the

than

in Chapter

10

regular

as

such

matrices,
that

several

are equivalent to matrix

can be
Strassen's
multiplication.
algorithm
in
when
it
for
matrices
and
the
recursion
improved
practice
by using
only
large
stopping
the
size
of the matrices become smaller than
of
idea
This
is
about
100.
similar to the
with care, which we discussed in Section
6.4.4 and
selecting the base of the induction
Section 6.11.3. Strassen'salgorithm
also
and raised
the door to other algorithms
opened
similar
that seemed unsolvable.
many questions about
problems
other

problems

Boolean Matrices

9.5.3
In

this

we consider

section,

Boolean matrices. All

following

These
and

case

special

are 0

or 1, and

correspond to or and

rules (which

definitions
product;

of sum
hence,

defined

and

(both 0+1

with
and

and

of computing the product

the sum and product are

of

two nxn

defined by

the

respectively):

are of course different

from
the usual integer sum
be used for
cannot
designed for integers normally
the definition of a Booleansum
is not
is that subtraction
1 + 1 are defined as 1; hence,
1 and 0).
1-1
can be both

product

algorithms

Booleans. One problem

well

the

elements

Multiplication 305

9.5 Matrix

Strassen's
cannot be used for Booleanmatrices,
it requires
because
algorithm
subtraction. However, there is a trick that allows us to use Strassen's algorithm.
We
an
+
bit
as
n
where
n
is
consider
modulo
the
size
of
the
and
we
1,
matrices,
every
integer
of such
use the rules of addition and multiplication
for
if
So,
integers.
example, az=4,
and 1 + 1 + 1 + 1 + 1 =0.
It turns out that,
if we compute
then 1 + 1 =2, 1 + 1 + 1 =3,
the
matrix
to these rules and if we substitute every nonzero entry
in the
according
product
final result by a 1, then we get the Boolean
This is so, essentially,because we
product.
will
the number n + 1 (we omit the proof). (More precisely,the integers
not \"overflow\"
k form a ring, which
is an algebraic
structure with definitions
modulo
of sums and
that
certain
can
Strassen's
be
satisfy
algorithm
properties;
applied to any ring;
products
Ullman
for
more
see Aho, Hopcroft, and
of
[1974]
details.) Thus, the complexity
Boolean
matrix multiplication
is also
The use of Strassen'salgorithm,
0(n2Sl).
than
Boolean
Next, we present
however,
operations.
requires
integer operations rather
that utilize the properties of Boolean
to improve the running
two
operations
algorithms
matrix multiplication.
These
are more practical in most
of Boolean
time
algorithms
than Strassen's algorithm
for Boolean
matrix multiplication.
situations
k operands
Since Booleanoperands
in
only one bit of storage, we can store
require
n is stored
in
of size k. In particular, since we assume that
one computer word
one
The regular algorithm
for
word, we can storek bits for k <log2Ai in one word.
computer
consists of n2 row-by-column
matrix
(or inner products), as
products
multiplication
Therefore,

each of

which

Boolean

vectors

assume
of

time.

the

and

of k-bit

how

idea

(For example,

to

Boolean

store
of

represented
on,

we

the
by
will

in

time

all the

algorithm

the

four

k bits
not

a
of a

make

time; thus,

computed
the

whether

they involve
(k22k) (we can
in

and
and

two

is stored

ih is

a distinction

Boolean

between

do

and

with

bits

k. There
k. We can
that; see Exercise
of size

of

where
by
ih

another
illustrate

of size

than

better

a (or

idea

Both ideas

products

represented
ia

Nevertheless, we

this

vectors

table of
at entry (iaJh),

the integer

takes one unit

first, we compute

0s.)

further.

a two-dimensional
b

two

not unreasonable to

steps:

combine

Booleaninner

actually

of

assume

special assumptions concerning

Next, we show
operations).

even

two

We

of size k

is all

result

on
dependent
arithmetic
basic

multiplication

in

products,

product

integers.

it is

vectors

assumption. Then, we

matrix

results

two vectors
the

check

since

products,

all of them

9.24),and
product

idea

possible

compute

we

two k-bit

Boolean

for algorithm design.

is to precompute all possible

techniques

The first
are 22k

one unit of

of nlk

sum

for

Assume,

'bmj.

inner

the

Finding

multiplying

can be

product

the need for such

avoid

takes

(besides

primitives

to improve

interesting

inner

an

say,

a\\m
\302\243
m = \\

into a

product

of size k.

product of two

two vectors, then

want to make the

not

do
computer

simpler than,
integers

inner

each

vectors

an inner

computing

of the

usually
the

that

We can divide

of size k is

of computing

consists

product

Boolean

involves

a multiplication

that

inner

//th

k divides n.

that

simplicity,

now

The

in (9.3).

defined

the
and

ia

The

2*x2*.

size
is

k bits

the

integer

of b.

/?), since

From

they are

of size k, we can
vectors
represented in exactly the same way. Thus, given two Boolean
compute their product by simply looking at the table. If we can access a table of size 22k
time (once
in 0(1) time, then each inner product of size k can be computed in constant

Algebraic and Numeric

306

the

Algorithms

constructed). For example, let

table is

table is O(n), and constructing

access a table of size O(n) in
(implicitly)

assumption

we can

then

input,

it requires
constant

k =

many times before.

store a

number

with

[\\og2n/2\\.

0(n \\ogn) time.

time is not unusual.
We

usually

In that case,
We

assume

have

that,

if

size

the

The assumption

that

of the
we can

already made this

the size of the

n is

word (or
log2Ai bits in one computer
we can compute
is constructed,
table

constant

a Boolean
words). Once the
=
Notice
that
the
table
n in time
size
O
O
of
(n/k)
inner
(n/\\ogn).
depends
only
product
the product
of two Boolean
not on the matrices.
So, computing
on the value of k and
and extra storage of O(n). We can also
matrices can be done in time 0(n3/\\ogn)
in
the
table
size is 0(n2), but we save an extra factor
which
case
be
to
k
choose
[log2\302\253J,
However, if we can afford an extra space of size
of 2 in the multiplication
algorithm.
0(n2), we can find a faster algorithm.
Consider two n x n Boolean matrices A and B. The usual way to view matrix
each involves a row
multiplication is as defined in (9.3): We perform n2 inner products,
the two matrices
can also multiply
columns
of A and a column of B. We
by multiplying
of A with rows of B in the following way. Denote the rth column of A by Act^l, and the
and ^[rjasa
lx/i matrix.
Consider
rth row of B by BR[r],
Ac[r] as an n x 1 matrix,
x
The product of Ac[r ] with BR[r ] is an n n matrix, whose ijth entry is the product of the
the 7th entry of BR[r ] (see Fig. 9.4). It is easy to see that
of Ac[r ] with
/th entry
of computer

number

A'B

(9.5)

itAc[r]-BR[r].
r

= \\

is equivalent to (9.3) in the sense that the same products and

in a different
but they are performed
order.
We now partition the columns
of B into n/k equal-sized groups.
of A and the rows
assume
for simplicity that n/k is an integer;
there will be an extra smaller
(We
otherwise,
group.) In other words, we divide A into A\\,A2, \342\200\224,An/k,such that each At is an nxk
and we divide B into BUB2,...,
is an kxn
matrix. It is
matrix,
Bn/k, such that each \302\243,
to
see
that
easy
The

(9.5)

expression

are performed,

additions

n/k

A-B

(9.6)

YtAi-Bi.
/

=!

BR[k]
Ac[k]

ABC
Figure

9.4

Multiplying

matrices

columns

by

rows.

307

Matrix Multiplication

9.5

how to compute Ci=Ai-Bi efficiently.

We
describe
this
9.5).
(see
computation by
Fig.
The first row of C, is exactly the same as the third row of Bh because the first row
of Aj has a 1 only in column 3. Similarly, the second
row of C, is the Boolean
sum of the
second and third rows of Br It is easy to see that the yth row of C, is a Boolean
sum
of
rows of Bj according to the yth row of Aj. Instead of computing
each
row of C, in a
similar
to the algorithm we described earlier, for
way, we use a method,
straightforward
all possibilities.
in each row of Ah so there are 2k
There are k entries
precomputing
=
of
k
and
combinations
rows
of
Let
assume
Bj.
log2\302\253,
again that k is an integer.
possible
all 2k = 2og2\" = n combinations,
We precompute
and store the results
in a table. In
to
the
first
this
n
table
contains
rows rather
than
n bits; thus, the
contrast
algorithm,
The

is

now

problem

an example

Also,

is 0(n2).

requirement

storage

for each Bj. To

of Bj that need

find

this

table

depends

of C,, we look at row

be added. This combination

row j

to

j of Aj

on Bh and
and see the

of rows

as an
represented
the first row of Aj

be

can

be constructed

must

combination

integer

in Fig.
j of Aj (e.g.,
to 3, the third row corresponds to 4,
9.5 corresponds to 1, the second
row corresponds
This integer is the address
in the table where row j of C, is stored. It takes
and
so on).
time to find a row of C, in the table, and O(n) time to copy this row to the
0(\\)

to the

corresponding

appropriate row
We now
O (n

time

induction,

Thus, computing C, can be done

all the combinations of sums

in C,.

that

show

\342\200\242

2k).

Each

that

we

Assume

in time
of

0(n2).
of Bj can

rows

of rows corresponds to a k-b\\i

sums
of
how to compute the

combination
know

corresponding to integers
trivial.

of row

representation

binary

that

are

less

than /'. Computing

the binary representation

followedby j Is. The sum of
that

significant 0 is
of rows corresponding to

xxxx

000000

of / -

is xxxx

We assume,

integer.

combinations
sum

corresponding

011111

\342\200\224

rows corresponding

the row

plus

the

be computed

to

/ is

in

by

of rows
to 0 is

namely,

its least

equal to

the

sum

corresponding to 0000100000.Since

is less than /, we know

its corresponding
sum by induction, and we need
row
to
it.
It
n
Boolean
additions
to add a row, and we have 2k
to
add
one
takes
only
If
with
all
be
done
combinations.
the
can
Hence,
0(n-2k)
operations.
precomputing
=
k log2fl,
is known
as the four-Russians
then
the running time is 0(n2). This algorithm

xxxx

000000

0
0

I
0
0

1
0

1
0

1
1
0

I
0

1
1
1

0 1 1 1 0 1
1 1 1 0 0 0 0

1
0

B,

1
0

0
0

1
1

1
0
0

C^Afr

Ai

Figure

9.5 Boolean matrix

multiplication.

1
0

1
0
0

is given

algorithm

in

Algorithms

[1970]), after

et al.

(Arlazarov

algorithm

The

and Numeric

Algebraic

308

the

and

nationality

number

of

inventors.

its

9.6.

Fig.

(A, B, n, k) ;
Boolean_Matrix_Multiplication
Boolean
and
B
A,
matrices),
k(an integer).
(twonxn
Input:
of
A
and
C
B).
(the
Output:
product
| we assume, for simplicity, that k divides n )

Algorithm

begin

C toO ;

the matrix

Initialize

fori :=Otonlk- do
1

Construct

Table,

{ Tablej is an 2k array of Boolean vectorsof size n which contains

all possiblecombinations
of sums ofk rows ofBj; see the text}
m := i* k ;

forj :-

do

ton

be

Let Addr

k-bit

the

A[j, w +

number
\342\200\242
\342\200\242
\342\200\242

l]y4[/', m+2]

add Tablej [Addr

] to

row j

A\\J,

in

m+k]

end

Figure

9.6

Boolean

Algorithm

JAultiplication.

JAatrix

To compute A B we have to compute the nlk products AjBj.

Since
Complexity
each such product takes 0(n2) time and constructing
the
the table takes 0(n -2k) time,
=
total running time of the algorithm
is O (n3/k + n2 \342\200\242
If
k
then
the
2kIk).
running
log2fl,
time isO(n3/\\ogn).
we show how to combine the
ideas
of the first algorithm with
the ideas of the
Next,
second algorithm
to improve
the running time by another
O (log a?) factor. The main
step
in algorithm Boolean JAatrix JAultiplication
9.6) involves additions of a row from a
(Fig
table to C. We can perform
trick of
this addition in time 0(n/m)
by using the same
is
all
if
addition
additions.
be
a
Boolean
not
(This
precomputing
possible
may
necessary
a primitive
not
does
can
that
be
the
however,
operation
performed quickly;
algorithm,
on this assumption.)
We first construct
table AddJTable of
a two-dimensional
depend
size 2mx2m
that
includes
all possible additions
vectors of size m. In
of two Boolean
other
in AddJTable
is the Boolean sum of / and j. (Again, / and j
words, the (/, j)th entry
are used both as integers and as Boolean
can be
It is easy to see that AddJTable
vectors.)
constructed in time and space 0(m-22m).
Notice that, unlike the tables we used in
of A and
Boolean JAatrix JAultiplication
is independent
algorithm
(Fig. 9.6), AddJTable
B\\

as

group
All

only on

it depends

each of
the

size m
a m-bit

steps

(we

integer;
of the algorithm

thus,

We now divide
for simplicity, that

of m.

value

the

assume

again,
each
will

row

of

is
\302\243,

be performed

each
m

represented

row of

divides

n).

Bt

into

by an rt/m-tuple

on these tuples.

nlm

groups,

We consider

each

of integers.

9.6 TheFast FourierTransform

To

vectors of size n, we use

steps. Each step consistsof

add two
in nlm

rt/m-tuples

corresponding
entry
in constant
performed
use

this trick

for

adding

both for
the rows

approximately
represent n in

one

instead of the

usual

that

the

changing

and, since we

0(n)

word, we can representa 2m-bit

of the improved algorithm

running

an

order of

the

As

an

to the fast

introduction

Fourier transform,

we

the hardware.

by

Lipson's excellent

from John

quote

aesthetic

grounds
it

Fourier
is strong

all

on

algorithms

of these

The

FFT algorithm is

We concentrateon

no

product

p(x)-

and

simple,

of

Given two

polynomials

a^x

its

+an_2xn~2

This linear

But, since the

function

sixties.

p (x)

straightforward.

multiplication.

polynomial

is

above,

not

and q (jc),compute

We

polynomials.

'\342\200\242'

+a{x

well defined.

+ a0

by the

function

corresponds

is

their

to a

line

of

its

of

in the

degree

by the
plane, it

coefficients
it is not

but

1, which is
two

can

the

polynomial
in

increasing

the

only

a linear

coefficients

also

specified

represent

adequate,

specified

usually

list

not

have

We

usually

of degrees.

+ a0.

is not

development
\342\200\224

FFT

the

This representation is definitely

possible. Consider, for example, a polynomial
order

it

\"super\"

the mid

in

discovery

the

of

q (x).

The problem, as stated

of the
representation
P =an_{xn~[

means

one application

Problem

The

because

perhaps

\"FFT\,")

293.)

by

only

its

grounds
The fast

applications.

departments, has emerged as one

of Computer Science since

(Lipson[1981],page

dramatic

on

on

problem,

practical

or even
the

on technological

grounds;

important

to as

referred

(popularly

transform

of

unexpected areas of

new and

up

opens

an

is elegant,

it

because

a number

solves

it efficiently

because

because

be appreciated on

may

algorithm

grounds

of

manner to other

same

book:
An

be
can

Fourier Transform

Fast

The

trick

The

algorithm.

all possible combinations can be applied

in the
on bit strings that cannot be performeddirectly
algebraic functions

computing

9.6

and

all possibilities
We also demonstrated

a better

lead to

can

be
We

word. For

in one

number

we

the

is 0(A23/log2Ai).

is needed.

what

only

computation

assume that

two

of computing

method

interesting

of computing

wisdom

Such a step can

their sum).

time

We presented

Comments

22m =

then

[log2\302\253/2j,

computer

the

execution

the

add
the corresponding
m-bit numbers and fetching

of the computer
word is at most 2m.
for the regular four-Russians algorithm,
of the algorithm.
If we select m to

tables

the

during

taking

two

the size

as

constructing

J able to

contains

(which

time, as long

equal to

of m,

choice

this

AddTable

in

Add

309

ax

be specified

one

function

and a0.
by

any

Algebraic

310

and Numeric

Algorithms

of degree n is
on that line.
In the same way,
any
polynomial
the
n
+1
For
example,
second-degree
polynomial
by
points.
uniquely
and it is the only
p(jt)=Jt2 + 3jt+l is defined by the points (1,5), (2,11), and (3,19),
are not the
all those points. These three points
second
degree polynomial that includes
on
the
three
that
define
this
points
corresponding curve
polynomial; any
only three points
two

points

(nonequal)

defined

will

do.

This

representation
of points is

is

for

attractive

multiplication

polynomial

because

multiplying

q(x) = 2x2 -x + 3 can be

easy. For example, the polynomial
We right away know that the product
p(x)-q (x)
represented by (1,4), (2,9), and (3,18).
has the values (1,20), (2,99), and (3,342). These three points are not enough to represent
this problem
by requiring five points
p(x)q (x) since it has degree 4. We can overcome
smaller
for
we
can
add
the
and
each of the
from
(0,1)
points
polynomials;
example,
and (0, 3) and (-1, 6) to q (x). We can then easily obtain five points
that
top(jc),
(\342\200\2241,-1)
\342\200\224
to the product \342\200\224
(1, 20), (2, 99), (3, 342),(0, 3), and
(-1,-6)
by making
only
belong
the

values

of two
we can compute the
scalar multiplications!
Using this idea,
product
in
this
with
of
O
az,
(n)
degree given
only
representation,
multiplications.
polynomials
with
this approach is that
we
cannot
The main
problem
simply change the
to
fit
one
We
must
be
for
able,
only
application.
example, to evaluate the
representation
at given points. This is much
than it is
harder
to do for this
representation
polynomial

five

coefficients
are given. However, if we could convert
from one
when
the
efficiently
we would have a very
then
representation to another,
good
polynomial
multiplication
algorithm. This is what the FFT achieves.
can be done by polynomial evaluation. We
Converting from coefficients to points
can compute
the value of a polynomial
at any given
p (jc), given by its list of coefficients,
We need to evaluate
5.2) using n multiplications.
point by Horner's rule (Section
p (jc) at
n arbitrary points, so we require n2 multiplications.
from
points to
Converting

coefficients is called interpolation,

key

here (as

idea

so

in

many

arbitrary points; we are

chooses

transform

Fourier

can be

interpolation,

free to

choose

a very special

it also

generally requires 0(n2) operations. The

this book)
is that we do not have to use n
set of n distinct
we want. The fast
points

in

any

set of points

that both

such

steps,

and

evaluation

done quickly.
Transform

Fourier

Forward

The

and

other examples

problem. We need to evaluate two n - 1 degree

each at 2n-\\
which is a 2n - 2 degree
polynomials,
points, so that their product,
can be interpolated.
an n -1 degree
polynomial,
However, we can always
represent
as a 2n - 2 degree polynomial
coefficients
1
the
n
first
polynomial
(leading)
by setting
to zero. So, without
loss
of generality,
is to evaluate an
we assume that the problem
We

first

the evaluation

consider

polynomial

arbitrary

points for which

is a powerof 2.
We

polynomial

the

P =

n-\\

a{x{
\302\243

polynomials

matrix terminology
P above for the n

of degree

n-\\

are easy to

use

to

points

n distinct

evaluate.

simplify

x0,xu

at

the

..., xn_x

We

assume,

notation.
can

We

points.

want

to

for simplicity,

The

be represented

evaluation

as

the

find

that n
of

following

the

9.6

matrix

vector

by

311

Fourier Transform

multiplication:
n-\\

(x0Y

The question is
this

can

we

P(x{)

(x{y

n-\\

P(*n-\\)

C*\342\200\236_i)J

choose

the values

of

jc0,

two arbitrary rows i

Consider

multiplication.

P(x0)

<*0

n-\\

(Xn_\\)

Xn_i

whether

(x0y

Ui)2

*i

simplifies

The Fast

...,

jcj,
and

j.

in

xn_\\

would

We

that

a way

like to

as similar
as possible to save multiplications.
We
make them
cannot
make *,=*,-,
because the values must be different, but we can make (jc,)2= (jc7)2 by letting Xj = -xh
This is a good choice, because every even power of x{ will be equal to the same even
of the multiplications
of
We may be able to save one-half
involved with row j.
power
Xj.
of
do
same
for
other
rows.
Our
is to have n special
we
can
the
Furthermore,
pairs
goal
above
nil
rows for which the computation
vector products. If we can do
requires only

that,

we

then

efficient

may

algorithm.

of half

the

We

Let's

try

cut

to

the

size by half,

problem

this problem

pose

in

terms

which

of two

will

lead

to a

very

separate subproblems

size.

want
to the

according

be able to

into two subproblems of size nil,

to divide the original
problem
in the following
described above. Thisis illustrated
expression.

scheme

*i

n-\\

(xoY

(x0y

C*i)2

(x{y

n-\\

P(x0)

<*0

P(x{)
1

xni2-\\

(Xnl2-\\)

i.Xnl2-\\)

n-\\

(9.7)
-x0

i-x0f

~X\\

(\"*l)2

(-x0y

n-\\

P(-Xn,2-\\)

0*i-l
1

The nxn

~Xn/2-\\

(-Xn/2-\\)

* *

n-\\
(-Xn/2-\\)

each of size n/2 x n. These two

is divided into two submatrices,
For
each
such
that
we have JC,=-Jcn/2+/.The
similar.
i,
0</<az/2,
very
to
same
in both submatrices, so they
of the even powers are exactlythe
need
coefficients
The coefficients
of the odd powers are not the same, but they
be computed only
once.
of each other! We would
like to write the expressions for P(Xj)
are exactly the negation
of the even and odd coefficients:
P (-jc,) for 0< i < nil in terms
and
n/2-1
n/2-1
P(x) = E + 0= X a2ixl1 + X fla+i*2\"1.
/=o
/=o
matrices

The

\"even\"

matrix

in (9.7)

are

polynomial

can
(\302\243)

be

written

as a regular

polynomial

of

degree

nil-

Algebraic and Numeric

312

the even

with

Algorithms

coefficients of P:

/i/2-1
\302\243=

The

a2i(x2)l=Pe(x2).

\"odd\"

(O) can

polynomial

be

same way:

in the

written

n/2-1

0=x

a2M(x2)l=xP0(x2).

2
/=o

we have

overall,

So,

the

expression:

following

P(x) = Pe(x2)+xP0(x2\\
where

powers
To

are the

(P0)

Pe

of P.

-jc for x

in

(9.8),

need to compute P(Xj)

coefficients of the even (odd)

we get P(-x) = Pe(x2)+ (-x)P0(x2).
for 0</ < n/2. To do that, we
P(-Xj),
with the

polynomials

degree

substitute

we

When

(9.7), we

evaluate

n/2-

(9.8)

and

need to compute only n/2 values of Pe(x2) and n/2 values of P0(x2), and to perform az/2
and n multiplications.
of size
So, we have two subproblems
additions, n/2 subtractions,
additional
O
and
nil,
(n)
computations.
with
the same scheme recursively? If we could, then
Can we continue
we would
=
+ 0(n), resulting
in an 0(n\\ogn)
T(n)
lT(nll)
get the familiar recurrence relation
of computing P(x) (a
the problem
algorithm. But this is not so easy. We reduced
of degree
n- 1) at n points to that of computing Pe(x2) and P0(x2)
(both
polynomial
of
n/2at
n/2
is
This
a
valid
one
for
reduction,
1)
polynomials
degree
points.
except
small thing.
The
of jc in P(x) can be chosen arbitrarily,
values
but
the values of jc2,
which
are needed,
for example, in Pe(x2),
this
can only be positive. Since we obtained
reduction by using negative numbers, this poses
a problem.
Let's extract from (9.7) the
matrix
that corresponds
to the computation
of/^((jc,)2):
ri
1

If

we

squares

try the
numbers

let

= V-1

xj+n/4

c*o)4

c*0r2

*0

t*i)2

C*i)4

C*i)\"-2

a2
a4

C*/i/2-l)

C*/i/2-l)

same

on

trick

are always positive,

complex
first

c*o)2

partition.

xj9

0<y

O (n)
to

V-1.

We

solve the

a number
=
and z2 V-1).

z such
In

again

general,

partition

problem

of size

1, and

we need

Pe(xn/2-\\)\\

set

the

n/2
we

zj * 1 for

a number

not

impossible

that

into

two

same properties

by

need

solving

two

a number

0 <j
satisfies

Since

(jc\342\200\236/4)2
=-(jc0)2.

the problem

divide
satisfies

that z8 =

But it is

impossible.

This

Pe(*\\)

we need to

subproblem,
seems

PeiXo)

an-i

(xn/2-\\)n

additional computation.
carry this process one step further,

that is,
z4=-l,

< n/4.

Hence, we can

of size n/4 and

If we want

this

include

which
for

this

' '

\"

<8

(which

that

if we

use

parts

and

as did

the

subproblems
is equal
implies

the condition

to
that

above

9.6 The

for

n rather

denoteit

n throughout
co\"

do

= 1,

this

co
1

we

simplicity;

will

313

of unity.
use

conditions:

(9.9)

are 1, co,

or

co

2-2

CO

co\"-1.

co2,...,

Therefore,

*0

P(D 1

CO\"\"1

a\\

/>(co)

CO2

We

the same

for0<j<n.

The n points that we choose as x0,x{,...,xn_{

the following product:
to compute
1

root

nth

primitive

for

include

not

section.)

and o7^

a number is called
n in the notation
satisfies the following

Such

8.

for

than

by co. (We

Fast FourierTransform

^\342\200\242(n-l)

we

want

/>(co2)
=

This product is

co\"-1

called

co(\"-1)2

CO'|(n-l)-(n-l)

/Mco\"-1)

an-\\

of (a0,ah ...,an_{). First,

transform
we notice
that
=
=
we
have
So
the
reduction
that
we
<n/2,
any
y, 0<j
Xj+n/2
co\"/2Xj
-Xj.
of
n
is
still
to
size
valid.
the
Furthermore, the subproblems
applied initially
problem
from that reduction have n/2 points,
which
are 1, CO2, co4,...,
co\"-2. But this is
resulting
of size n/2 in which
we substitute CO2 for co. The conditions in (9.9)
exactly the problem
root
of unity.
that co2 is a primitive
we can continue
Therefore,
(n/2)th
imply
A
of
the
is
and
the
O
view of the
(n
complexity
algorithm
logAi).
high-level
recursively,
the

Fourier

indeed for

algorithm

is presented

depends
Output:

(az, a0,

Fast_Fourier_Transform

Algorithm

Input:

9.7.

in Fig.

(an integer),
aQj a \\,..., an_x
on the application), and

V (an array

in

the

| we assumethat

a x,...,

(a sequence
CO

mh root

(a primitive

- 1 ] of
range [0..n
is a power of 2 )

co, var

an_x,

of elements

of unity).

elements

output

).

begin

ifn=

then

V[0]:=a0;

else

a0a2,...,an-2>to2>U)>'

Fast_Fourier_Transform(n/2,

ax
a3,...,
Fast_Fourier_Transform(nl2,
(9.8)
{follow
forj:=0ton/2-\\do
for

V[j]:=U[j]

an_x,
Jt =

co2, W
co7

+ rtW[j];

V[j + n/2]:=UV]-rtWlj]

end
Figure

9.7 Algorithm

Fast Fourier

Transform.

V);

whose type

);

and Numeric

Algebraic

314

Algorithms

D Example 9.2
We

to compute
we

how

show

denote

(0, 1,2, 3,4, 5,6, 7).

for the polynomial

transform

Fourier

the

where
jk(x0,xj,...,xk),
and jc0,x{, ...,xk denote the
the coefficients
of the polynomials,
denote
JoJi'\342\200\224'Jk
for which we need to evaluate the polynomials.
So, in particular, this
values
example
notation
is quite awkward,
but
co,co2,\342\226\240\342\200\242\342\226\240\302\273co7).
(This
involves
computing /*(),1,2,3,4,5,6,7(1*
we need.)
The main recurrence we use is (9.8).
it contains all the information
To

confusion,

avoid

first

The

co2,co4, co6).

Px 3,5,7(1,

and

P0,40\302\273co4)

4.

We

Pjojxt

co7)

t0

co2, co4,

^0,2,4,60.

and

CO6)

reduce /^^O'co2,
co4,co6) to
recursively
tnen
to
reduced
which
is
P0U).
clearly 0, and
^0,4(1. co4)is
can now combine
the results to get
and

continue

We

co4).

P2,6(U

is clearly

which

P4(\\),

by

subproblems

^0,1,2,3,4,5,6,7(^ co,co2, -.

reduces

step

the

/>o,4(l) = />o(D+l-/>4(l)= 0+l-4

4,

and

P0 4(co4)
Since

co4

manner,
We

= />0(co4)+ co4/>4(co4)

= 0

+ co4\342\200\242
4.

= -1,
we get P04(to4) = -4, and,
overall,
co4) = (4, -4).
P0,40.
we get /^(l* co4)= (8, -4).
now
combine
the two vectors aboveto compute
co2, co4,
Po,2aA^

PoiAfii\302\2562)

/>0,2,4,6(C04)

Po,2aA<\302\2736)

^o,4(co4)

+ co2\342\200\242
/>2,6(co4)

/>o,4(C08)

+ C04-/>2,6(C08)=
+

^0,4(C012)

= -4

\342\200\242

CO6

same

co6):

= 4 + 8=12.

= />o,4(l)+l-/>2,6(l)

/>o,2,4,6(l)

the

In

/>2,6(C012)

+ co2(-4).

/>0,4(l)-l-/>2,6(l) = 4-8 = -4.

=

/>0,4(C04)

CO2

\342\200\242

= ~*

/>2,6(C04)

~ ^HO-

So, overall

same

In the

CO4, CO6) = (

CO2,

^0,2,4,6(1,

P 1,3,5,7(1,
co2,co4,co6)= (
To

compute

= 12-

co, co2,...,

1-16=-4; P0t

in

the

+co2)) +

= (-4(1

1,2,3,4,5,6,7(co)

we need to

co7),

12+ 1 \342\200\242
16 = 28, and,

= (-^4(1+co2))- co-(-4(l+
manner, ^0,1,2,3,4,5,6,7(co5)
to the reader.
The

Fourier

Inverse

The algorithm

for

the

evaluate

the

multiply

the resulting

two

).

16,-4(l+co2),-4, -4(1-co2)).

/>o,i,2,3,4,5,6,7(l\302\273

P 0,1,2,3,4,5,6,7(1)=

example,

-4, -4(1 -CO2)

find that

we

way,

12, -4(1+C02),

given

same

compute 8

manner,

P0,i,2,3,4,5,6,7(co4)

co-(-4(l +co2)),and,
co2)),

and

so on.

For

values.

in

the

We leave the

same
rest

\342\226\2

Transform

fast Fourier

values, and

solves

transform

polynomials
find

p{x)

and

the values

q{x)

of

only half of
at the points
the

product

our

problem.

1, co,...,co\"\"1

polynomial

p(x)-q

We

can

quickly,

(x)

at

we still need to

But

those

points.

from

the evaluation points.

similar to the evaluation

Consider again

vector

the

Fortunately,

the

and we

polynomial,

points 1, co,co2

at the n

algorithm

to

want

, we

co\"

can

we are

When

compute

out

to

solve

it.

values of

the

matrix

be very

coefficients

the

given

compute

the

polynomial

product

turns

problem

interpolation

notation.

matrix

of the

coefficients

and an almost identical

problem,
the

...,0\342\200\236_i)of

(a0,ai,
polynomial

the

interpolate

315

Transform

Fast Fourier

The

9.6

by

the

product:

co

CO\"

2
CO

oy

On

the

(v0,

v j,...,

other

system

following

2-2

co2'\"\"\"

oy((n-l

and we

are given,
v\342\200\236_!)

co

CO\"

of

want

/>(0)2)

)\342\200\242(/.-!)

/>((0\"_l)

an-\\

the

(fO),
f^co), ...,/> (cow_1))=
the coefficients, we need to solvethe

polynomial

to compute

a0, a \\,...,0\342\200\236_i:

of equations for

\"

v0\"

\"tfo\"

o)\"-1

2-2

CO\"

P(l)

a)\"\"1

the values

when

hand,

co(\"_1)*2

co\"\"1

Vl

tfl

v2

a)2c-|>

CO

1 CO\"\"1

Solving
case),

(vo.

where V(co) is

Tne
\342\200\224\302\273
vi\302\273
vn-\\)-

V(co) has
inverse

an

has a

simple

very

of

the

matrix

for a

form

out

that

(we omit

011-1
.

time

quite

in

the

left

can be written
V(co)
the

v\342\200\236-i
-

consuming
Let's write

equations.

always

(9.10)

(0(az3) for
this

matrix

the

general

equation

as

side, a =(a0,a\\, ...,an_i),

that
a = [V(co)]_1 \342\200\242
v, provided
its
has an inverse; furthermore,
and

as

proof):

9.1

\342\226\241
Theorem

[V(co)]\"1 =

usually

solution

It turns

inverse.

oy((n-l)(n-l)

CO*\"\"02

systems of equations is
but this is a special system

V(co)-a=v,
v =

the

following

- V(-).
co

\342\226\2

to solve

Therefore,

by vector product.

the system of equations

we need to compute
(9.10),
This task is greatly simplified by the following
theorem.

Theorem
\342\226\241

If (d is a primitive
unity.

only

matrix

one

9.2
nth

root

of

unity,

then

1/co is

also a

primitive

nth

root

of

\342\226\2

and Numeric

Algebraic

316

Therefore,we

the product

V(l/(fl)v by
substituting 1/(0 for co. This transform

can

compute

transform,

Fourier

Algorithms

the

using

for
the inverse

algorithm

is called

fast

the

Fourier

transform.

Notice

operations.

0(n\\ogn)

product of two
that we need to

the

Overall,

Complexity

be

computed with
complex

and multiply

add

to

able

be

can

polynomials

numbers.

9.7
The

Summary

numerical

We have

algorithms.

Strassen's

best.

the

lecessarily

in this

presented

algorithms

nonintuitive

algorithm

examples of

the

for

chapter are a
seen again that

a seemingly

and,

of induction,

use

is

algorithm

small

one

known

straightforward
of the most striking

simple problem.
in

of

sample

the

particular,

of the

use of

of

examples

seen

have

We

algebraic and
algorithms are not
a

more

several

divide-and-conquer

algorithms.

which is not based

algorithm suggests an interesting
technique,
The main idea is to compute all possible combinations
of certain
terms,
in
them
are
This
is
useful
cases
where
needed.
even if not all of
technique
computing all
combinations
costs much less than
each one separately.
(or many)
together
computing
Another
for problems
which is common particularly
technique,
involving matrices, is the
use
of reductions
is described,
between problems. This method
with examples, in
four-Russians

The

on

induction.

10.

Chapter

Notes

Bibliographic
The
include

Aho,

[1980],

for arithmetic

source

best

and

and

Hopcroft,

and
and

probably
algorithm

is Knuth
Munro

and

[1981]. Other books

Winograd

[1975],

[1981].

Lipson

schemes

encryption

algorithm

algorithms

algebraic

powers by
writings circa 200 B.C. (seeKnuth
scheme
is due to Rivest, Shamir,

encryption

Reading

Ullman [1974], Borodin

The algorithm for computing

in Hindu

key

Further

repeated

squaring

is very

old;

it

was introduced
in Euclid's Elements,

Book 7 (circa 300 B.C.),but

known even beforethen (see Knuth [1981], page 318). The divide-and-conquer
for multiplying
and
two polynomials was developed by Karatsuba
appeared

first

appeared

The RSA public-key

[1981]
page
and
Adleman
[1978]. The idea of publicHellman [1976]. Euclid's
Diffie
and
by
441).

it

was

Ofman

two large numbers).

in Winograd
[1970]).
[1968] (see also Winograd
Winograd's algorithm
appeared
Strassen's
in
in
the
Strassen
The
c
constant
[1969].
algorithm
asymptotic
appeared
1969
since
has been reduced several times
multiplication
running time 0(nc) for matrix
of asymptotic
in terms
at this time \342\200\224
(first by Pan [1978]). The best-known
algorithm
\342\200\224
is by Coppersmith
times
and Winograd [1987], and its running
time is
running
as the CO notation indicates, this
is
not
(D(n2316). Unfortunately,
practical.
algorithm
of matrix multiplication and related
For more on the complexity
topics see Pan [1984]. A
[1962] (in

the

context

of multiplying

discussion on
Roth [1976].

the

The four-Russians

of

the

Myers

in

and

Cohen

to Arlazarov, Dinic, Kronrod, and Faradzev

tables
has
algorithm
by using addition
it
is
without
in
mentioned,
details,
people;
Rytter
is used for context-free language recognition. The

been observed by many

[1985], where a similar
technique
same
idea was also used to improye
[1983],

be found

317

four-Russians

probably

Paterson

algorithm can

Strassen's

Exercises

is due

algorithm

The improvement

[1970].

of

implementation

Drill

Exercise

of

solution

(Masek

algorithms
9.26 appears in

comparisons

sequence

[1988]). The

and

and

Atkinson

showed a reduction
between
Boolean
matrix
[1988]. Fischer and Meyer
[1971]
and
the
transitive-closure
multiplication
problem.
The algorithm
for the fast Fourier transform
was introduced
by Cooley and Tuckey
can be traced to Runge
and Konig
[1965],
[1924]. For
although the origins of the method
more information
on the fast Fourier transform,
see Brigham
and Rao
[1974] and Elliott
Santoro

[1982].

Exercises

Drill

9.1

the

Discuss

computing

9.2

9.3

Algorithm Power_by_Repeated_Squaring (Fig. 9.2)for

lead to the minimal
number
of multiplications.
Show an
of multiplications.
with fewer number

Let x be a positive
Design an algorithm

(a,
then

9.4

between algorithm Power_by_Repeated_Squaring

relationship
nk and the binary
oik.
representation

not

of computing

example

necessarily
nk

(k>

10)

= a/b.
number that is represented
by the pair (a, b) such that x
of x\\ that is, the representation
to compute the smallest representation
b) with the smallest possible values of a and b. For example, if x = 24/84 = 6/21 =2/7,
of x.
(2, 7) is the smallest representation

Prove
that

nk does

computing

(Fig. 9.2) for

that

the

computes
as

operations

rational

divide-and-conquer algorithm for

of the smaller polynomials
products

straightforward
all

does

four

the straightforward

algorithm

follows

that

polynomial

makes

(9.1).

multiplication

exactly

Assume

the same
that

n is a

power of 2.

9.5

the

Find

P(x)=x

algorithm

+ 2x2 + 3x* +

\302\243)(.*:)=16+15*

How

9.6

P(x)-Q(x\\
presented

product

multiplication

+14*2

many operations

hand,
Section

using
9.4.

the divide-and-conquer

polynomial

\342\200\242\342\200\242\342\226\240
+

15jcl\\

\342\226\240\342\200\242\342\200\242
+ 2x,4
1jc'\\

are required

A divide-and-conquer technique
and discuss
an algorithm,
such
algorithm.

by
in

overall?
can
the

be used

differences

to

multiply

between

two binary numbers.

Describe
it and the polynomial multiplication

318

and Numeric

Algebraic

9.7

Use the algorithm

9.8

way to

all the

Design

an

Derive the
in

9.11

two numbers
4679x7114.

base b (not
the
recursion
Carry
in

any

two complex numbers

with

only three

scheme

described

(a+bi)(c+di)

of -1.)

root

2x2

for Strassen's

matrix

multiplication

9.5.2.

Section

Suppose that

you

be the

would

square

expressions

explicit

by 10111010.

10011011

multiply

can be used to multiply

the decimal multiplication

to multiply

(/ is the

9.6 to

Exercise

numbers.

1 -digit

algorithm

multiplications.

9.10

discussed in

divide-and-conquer
technique
b = 2).
Use it to perform
only
down

9.9

Algorithms

complexity of a

What

algorithm?

find an algorithm

value

matrices

4x4

multiply

matrix

general

maximal

is the

to

multiplication

will lead

of k that

to

with k multiplications. What

algorithm based on the this
an asymptotic
improvement

over Strassen'salgorithm?

Exercises

Creative

9.12

Consider

the

two

for computing

algorithms

Let n

be an

nk

9.2 (simple iteration,

in Section

given

Assume

d digits.

and

repeated squaring).
integer
multiplications
integer
are performed by the regular algorithm, which requires J, \342\200\242
two integers
d2 steps to multiply
with dx and d2 digits.
What
is the number of steps required
to compute
nk by the two
algorithms? (You can assume that k is a power of 2, and that a product of two integers with
with d]-\\-d2 digits.)
dx and d2 digits is another
integer

9.13
9.14
9.15

9.16

to

an algorithm

Design

is a multiple

The Fibonaccinumbers

a.

Prove

of m and

integer

LCM of k given
of all of them.)

> 2

is the

integers.

is a

that

multiple of

integers.

given

(The LCM of

integers

is the

recurrence relation:

by the following

can be

smallest integer

LCM of two

F{n) = F{n-\\) + F{n-2)

every

that

of k integers.

to find the

are defined

F(2)=l,
that

(LCM)
algorithm

to find the

Design an algorithm
smallest integer that

F(\\)=\\%

the GCD

find

The least common multiple

both
n and m. Design an

with

(n>2).

written

as

a sum

of

at

most

\\og2n Fibonacci

numbers.

b.
9.17

an algorithm

Design

Let P(x)

to

find

such

a representation

for a given

number

n.

be two polynomials. We say that a polynomial D(x) divides P(x) if

Q(x)
another polynomial S(x) such that P(x) = D(x)-S(x). Similarly,
we say that
and there exists a
Q(x) = R(x)modP(x) if R(x) has a smaller degree than P(x),
such that Q(x) = D(x) P(x)+ R(x).
P(x)
The GCD of two polynomials
D(x)
polynomial
and Q (x) is a polynomial
R (x) such that R (x) is the highest-degree polynomial
that divides
bothP(.v)and(2(.v).
there

and

exists

a.

Show

b.

Extend

that

the GCD

Euclid's

of two

algorithm

polynomials

to

find

is uniquely

the GCD

defined.

of two given

polynomials.

319

Creative Exercises

9.18

the

Modify

multiplications involving
of 3. What is the

smaller
complexity

power

9.19

the

Modify

polynomial
multiplication
into four equal parts, and

polynomial

parts. You can assume

of the algorithm?
complexity
smaller

9.20

Hamilton's
real

and

i, j,

componentwise, and

= -kj

ki

= -ik
/,

symbols

j\\

a. Prove that
b.

= j

P[i]

Considerthe

by

required

a, b, c, and d are
subtract
quaternions

where
and

is an

x n

with

themselves).

the ordinary

procedure

and

the number

reduces

matrix such

each

of two permutation

to
The outcome should

multiply

also

many

quaternion

to 12.

each column
be represented

row and

is equal to 1. A permutation
matrix can
the ith row contains a 1 in the jth column.

if

How
for

only five multiplications.

with
that

of multiplications

that

the product

representation.

matrices is another

permutation

has

exactly

by an array

matrix.

two permutation
matrices given
be given in an array representation.

by

the

array

We can use Winograd's

to modify
Strassen's
algorithm.
with approximately k312
the product of two kxk
matrices
We can then use this
as the basis for the divide-and-conquer
multiplications.
product
strategy instead of the one using 2x2 matrices. If k is large enough we get a better
time than Strassen's algorithm.
this suggestion?
What is wrong with
asymptotic
suggestion

following

to

algorithm

9.24

that

a linear-time algorithm

Design

+ cj + dky

a+bi

form

the

with real numbers

numbers are

matrix

one nonzero entry

such that

of

the square of a 2 x 2 matrix

to compute

permutation

of

problem is a

size of the

algorithm described in Section 9.4 by dividing each

the number of multiplications
minimizing
involving
the size of the problem is a power
of 4. What is the

are

k commute

and

Give an algorithm

each

= i
= j

multiplication?

9.22

the

9.4 by dividing
the number

= -l

of real

Show how

minimizing

special symbols. We add

them by using the following rules:
k

multiplications

9.21

9.23

and

multiply

/' = y2 = k2
ij = -ji = k

jk

that

Section

(instead

are vectors

quaternions

numbers,

(the

algorithm described in
of two),
and
can
You
assume
that
parts.
of the algorithm?

polynomial
multiplication
into
three equal parts

polynomial

compute

to compute all possible

vectors
Boolean inner products of two Boolean
algorithm
create a table of size 22*. The product of
should
(see Section 9.5.3). The algorithm
the two vectors a and b should be stored at entry i, where i is an integer respresented
by 2k
bits such that the k most significant
of a and the k least significant
bits are
bits of i are those
those of b. The running
time of the algorithm should be O (22*).

Design an
of size k

9.25

the program

Complete

tables

explicitly,

significant
9.26

Design
submatrices
such

should

and

for

matrix multiplication
to handle the case where

Boolean

how

(Fig. 9.6).

Show

is not an

n/k

how

to build the

integer

without

loss of efficiency.
an algorithm for Boolean matrix
multiplication
of size kxk, and uses the idea of precomputing

submatrices.

The running time

require extra

operations on

numbers

of

the

space of 0(n\\ogn).
with up to

bits
log2\302\253

in one

can

assume

step.

divides the matrices

all possible

should

algorithm
You

that

into

products between

be 0(w3/(logw)15),
that you can perform

and

it

basic

320

Algebraic and Numeric

9.27

Let A and B
chosen

be two

(independently)
the product of A and

9.28

nxn

Algorithms

random

to be
By such

Boolean matrices; each

either

that the

0 or 1 with

expected

probability

number

entry

in each

matrix is

randomly

Vi. Design an algorithm

of operations

to

find

will be 0(n2).

B be two 2n x In Boolean matrices

that represent
open Gray codes (seeSection
to the strings in the Gray
in the following way.
The rows of each matrix
correspond
code, such that two consecutive rows differ by exactly one bit (the first row and the last row
to find the product of the two
differ
by more than one bit). Design an O (n2) algorithm
may

Let

A and

2.9)

matrices.
be n matrices of real numbers.
The dimensions of A/, are fl,Xfl/+1, so
9.29 Let Af M A/2> \342\200\242\342\200\242\342\200\242\302\273A/\342\200\236
is defined
for each 1 <i <n.
We want to compute the product
the product of A/,-A//+1
Let's
assume
that it takes af-a/+1 a/+2 operations
to multiply an
MlxM2'x '\" xMn.
matrix
The problem is to find the right order in which to
by an ai+l xai+2 matrix.
fl,-Xfl/+1
For example, let n = 3, and let the matrices be of dimensions
carry out the multiplications.
and 5x3. Finding
the product
of the first
two
matrices
takes 10-2-5
2x5,
10x2,
in a matrix of dimensions
10x5.
resulting
Finding the product of this matrix
operations
with the third one takes 10-5-3 operations \342\200\224
250 operations. On the other hand,
if
overall,
we first find the product of the last two matrices and multiply
the first matrix with
that
an algorithm
to find the optimal order
product, we end up with only 90 operations. Design
out the matrix product above.
of carrying

10

CHAPTER

REDUCTIONS

is of two kinds.
a subject ourselves,
know where we can find

Knowledge

know

We

or

we

upon it.

information

Samuel Johnson,

Introduction

10.1

We

1775

this

start

chapter

with an old

joke.

mathematician

and

her husband

are asked the

to boil water,
and you want
\"Suppose that you are in the basement
following
question:
and boil
The mathematician
what do you do?\"
says that she will go up to the kitchen
asked
the following
Now they are both
answers
water there; her husband
similarly.
what
do you
and you want to boil water,
question: \"Suppose that you are in the kitchen
\342\200\224
the water.\"
boil
I'll just fill the kettle and
do now?\"
husband
The
says \"it's easier
basement
I'll go down to the
The mathematician answers \"it's even easier than that \342\200\224
to solve that problem.\"
and I already know how
will
show that
we will concentrate
on the idea of reduction. We
In this
chapter,
is
another
useful.
Here
reductions
can be extremely
besides being funny
sometimes,
a package
send
time
a real one. When you
of a reduction, this
by Federal
example
New
York City, the package will be
New York City to downtown
Express from uptown

all packages
through Memphis, so
of delivering packages across town
they
In this case, the solution
makes
sense. It
know how to solve the problem.\"
\"already
a special situation and
to identify
to build
a mechanism
to
may be much more difficult
more efficiently. It may be easier, and overall cheaper, to handle
situation
that
handle
true
in algorithm
This is also often
design. When we encounter a
equally.
everything

routed
when

through

they

are

Memphis.

faced

with

Federal Express routes

the

special

situation

i^

322 Reductions
whose solution is already
problem,
posed as a specialcaseof another
a solution
solution can be used. Such
may sometimes be too
a
solution
is the the easiest,
But
in many
or too expensive.
cases,
general
using
general
use
this
We
solution.
most
to
a
and
the
fastest,
the
principle
every day.
elegant way
get
\342\200\224
\342\200\224
it is usually
a database query
for
not
For some computing
problems
example,
it
is
to
this
sufficient
a program
that solves only
use
problem;
necessary to write
more
The
that
handles
software
general
problems.
general-purpose
general-purpose
but it is much easier to use.
not be the most efficient solution,
solution
may
P
that
seems
but that also seems
we
are
a
t
hat
complicated,
problem
given
Suppose
to
solve
P
from
or we can try to
We
can
to a known
scratch,
similar
problem Q.
try
them
to
P.
to
solve
and
There
of
the
methods
used
a
some
is, however,
borrow
Q
apply
the
two
to
between
We
can
find
a
reduction
third way.
(or transformation)
problems.
try
of one problem using
a \"black
box\" that
is a solution
Loosely speaking, a reduction
can achieve one of two goals depending on the
Reductions
solves the other problem.
black
box is used to solve which
in which they are done (i.e., which
direction
problem).
into an algorithm
for P if we
A solution of P that uses a black box for Q can be translated
for Q. On the other hand, if P is known
to be a hard problem, or, in
an algorithm
know
if
a
same
lower
bound
be applied to
we
know
lower
bound
for
then
the
f\\
may
particular,
is used to obtain information
about
in
f\\ whereas,
Q. In the former case, the reduction
about
the latter case, it is used to obtain information
Q.
For example, in Section
we discuss the problems
of matrix multiplication
10.4.2,
and
matrix
the
matrix
with
itself).
(i.e.,
squaring
multiplying
Clearly, we can square a
matrix
with
a matrix multiplication algorithm;
the problem of matrix
therefore,
squaring
can be reduced to the problem
of matrix multiplication.
We show in Section
10.4.2 that
it is possible to multiply
two
with the use of a matrix
matrices
algorithm;
squaring
is reduced
to matrix squaring.
The
of the latter
therefore, matrix multiplication
purpose
reduction
is to show that
the square of a matrix
cannot
be done faster (by
computing
can be

that

problem

more

that are

We

see

will

between

reduction

lower bounds

on

problems.

problem

than computing

a constant)

than

conditions

both

known

the

then

known,

discussed in
several

the

An effective

examples

is useful
of the

The reduction

may be

of

For

it.

enough

have

arbitrary matrices

some

(under

example,

the

of reductions

use
even

does

if it

used to

find

new

this

Finding
to new upper

helps us to understand
for attacking

techniques

the reduction may

chapter.

directly

reduction

The

problem.

in

lead

not

be used to

or
the

a parallel

design

problem.
way

to

use

problems can be reduced.

should be generalenough
to
to

of two

product

10.4.2)

of

two problems
the complexity

or variations

algorithm for

the

Section

reductions

Finding

10.3.

is to define a
a general

a wide variety
solution. We

cover

an efficient

such

of

general problem

problem is

easy.

but it must

problems,

discuss one

not

such

problem,

to

which
This

many

problem

also be simple
called

linear

programming,
We have already seen severalexamplesof reductions
for example,
in this book \342\200\224
the reduction of the transitive-closure
to
the
problem
all-pairs shortest-paths problem
are important
(Section7.8). Reductions
enough, however, to deserve a special chapter.
are also the cornerstone of the next chapter.
Reductions
in

Section

Examples of

10.2

10.2
In

of Reductions

Examples

this

we

section,

Reductions 323

present

four

of

examples

reductions

using

to

efficient

obtain

algorithms.

A Simple

10.2.1

with

start

We

The
two

a simple

ai=b(k+i)modn
and

matching),

of the

variation

determine

0</</?-l.
'\302\273

we call

the

solve

and

an_x

B is

whether

We

call

string-matching
the

modifying

a cyclic

this

\342\200\242\342\200\242\342\200\242

B=b0bi

be

bn_x

of A.

shift

an index

exists

there

original

CSM, for example, by

in Section 6.7. But
described

\342\200\242\342\200\242\342\200\242

whether

f\302\260r
aU

problem.

string-matching

Let
Problem
A=a0a{
of
sizen.
Determine
strings

The problem is to

Problem

String-Matching

such

0<k<n-\\,

ky

CSM

problem

(for

cyclic

that
string

problem (Section 6.7) SM. We

Knuth-Morris-Pratt
algorithm that

can
was

at a solution. The idea is to

we
for a certain
text T and a
look
words,
pose
B
is a cyclic
to
whether
that
P
in
T
is
certain pattern P such
finding
finding
equivalent
T and P can be appliedto
shift
of A. If we can do this,
then
a solution to SM involving
the problem in these
terms it is easy to
about
solve
CSM involving A and B. If one thinks
to itself). Clearly,
T as AA (namely, A concatenated
see the solution:
We define the text
B is a cyclic shift of A if and only if B is a substring of AA. Since we already know how
for CSM.
to solve SM in linear
time, we have a linear-time
algorithm
as a

CSM

10.2.2
Let

S i,

a set R
R

to

regular instanceof

S2,

way

to arrive

a collection

of sets. A system
of distinct
(SDR) is
representatives
that, since we require
r, e 5,, for all /, 1 </ <ky (notice
R includes exactly one
be distinct). In other
words,

\342\200\242\342\200\242\342\200\242>
Sk be

r2,...,
a set, the
{r j,

be

better

In other

SM.

of Distinct Representatives

Systems
=

is a

there

representative

such

must

/*,s

each

from

collection of

rk

sets. For

{2,3,4},
53 = {1,3},

set.

that

It is

example,

an

not

to

possible

always

SDR for

the

an

find

of

collection

= {1,2}, S2 =

= {1,2,3} is {1,4,3,2}, but there

is no SDR for
S4
=
=
=
=
and
Sx {1,2},S2 {2,3,4},
53 {1,3},54{ 1,2,3},
55 = {2,3}.
and

of sets

The Problem
for

There is

the

collection

very

conditions for

the

elegant
existence

Given

(any SDR

a finite
will

collection
do),

of

or determine

theorem, due to P. Hall, that

of SDRs. Let card(S) be the

finite

find

sets,

that none

gives
number

of a given

SDR

sets Sx

the

collection

an SDR

exists.

and sufficient
necessary
of elements of S.

324 Reductions
Theorem

Hall's
\342\226\241

Let S i,

S2,..., S*

be

only if the following

card

clear that the

It is

are

bipartite

vertex v,

in V

each

for each

Consider

the

there
of

the

if

and

size k.

Involving

\342\200\242
\342\226\240
\342\200\242

bm are

it becomes
\342\200\224

insert,

an

three

different

Another

Each

entry

elements,

apply

the proof

way

is also

condition

be

cannot

they

unfortunately,

subcollections, and
pose this problem

and

network

discussed

the algorithm

of Hall's theorem

there

as

in

Section

be obtained

can

flows.

Sequence

problem discussed
two strings of characters,and

Comparisons
in

we

6.8:

Section

want

A -ax

to edit

A,

a2

'''

an

character

by

equal to B. We allow three types of edit steps, each involving

and replace. The cost of each of these steps is given, and
the
cost
of the edit. The solution
in Section 6.8 was to
given

delete,

edit steps.
to look at this

in the table

less than

altogether
the

exercise.

our
goal is to minimize
construct a table of size n by my where each entry corresponds
contains
the cost of editing
the first / characters of A into
entry
The
is
thus
to
the
goal
compute
\"bottom-right\"
entry (nm) of
each entry can be computed from only three other \"previous\"
the

with

That

all possible
The idea is to

can

We

matching

sequence-comparison

until

character

every

distinct

such that there is a

U, E) be a bipartite
graph
is a vertex Uj in U for each possible element(i.e.,
sets). Each element is connectedto all the sets
to see that an SDR
only if Uj e 5,. It is now
easy

Furthermore,

of bipartite

A Reduction

character,

union

G of

in

10.2.3

b2

G=(V,

sets

to check

approach.

Let

problem.

are m

there

representatives.

it as

have

will

uj) e E

is, (v,,

if

simple conditions but,

set Sh and

in the

element

B=b{

leave

we

We

problem.

containing it;
is simply a matching
7.10 to solve this
from the properties

one

since,
have m distinct

is necessary

cannot

We need another

that

and

other
m

\342\226\

efficiently.

them.

2k of

contain

must

sets

to prove, and
theorem provides

matching

for

they

checked

directly

2, 3,..., k).
altogether at least

In
of {1,
\342\200\242\342\200\242\342\200\242\302\273'wl
{\302\253i,/2\302\273

condition

then

Hall's

words,

>m

us,

is harder

sufficient

if and

\\<m<k.

for every

m elements,

SDR

is satisfied:

\342\200\242\342\200\242\342\200\242

sf us,-u

for every subset

subcollection of m

of sets. This collectionhas an

a collection
condition

problem is by considering
in the graph.
corresponds to a vertex

to a

partial edit. The //th

j characters of B.

the first
the

We

table.

showed

that

entries corresponding to

the table

as a directedgraph.

A vertex

thus

corresponds

to

edit. There is an edge (v, w) if the partial edit corresponding to w has one more
edit step than the partial edit corresponding to v. An example
of such a graph is given in
the
10.1, where A =caa and B =aba.
The horizontal edges correspondto insertions,
Fig.
vertical edges to deletions, and the diagonal
edges to replacements. For example, the
shaded
of b,
of c\\ a match of a, an insertion
path in Fig. 10.1 correspondsto a deletion
and another match of a. In the basic problem, the cost of each
for
is 1 except
edge
to equal characters (i.e., no
diagonal edges that correspond
replacementis necessary)
a partial

10.2 Examples

Figure 10.1 The

to the

corresponding

graph

325

Reductions

of

sequences A

and B

-caa

-aba.

now becomes
a regular
0. The problem
single-source
shortest-paths
Each edge is associatedwith a cost (which is the cost of the corresponding edit
We
m].
path from vertex [0,0] to vertex [\302\253,
step), and we are looking for the shortest
to
the
have reduced the string-edit
problem
single-source sortest-pathsproblem.
than solving this problem directly.
shortest
paths in general is not easier
Finding
of
is
variations
useful.
for
this
reduction
Consider,
Nevertheless,
example,the following
the sequence-comparison problem. The cost of editing is not necessarily
per character.
of characters in the middle of another string
not be the
The cost of inserting
a block
may
same as that of inserting the same number of characters,one by one, in different places.
The same may be true for deletions. In other words, instead of assigning a cost per
we may want
to assign
a cost per blocks of
and replacement,
deletion,
insertion,
of
their
sizes.
we
or
deletions, regardless
insertions,
Alternately,
may want to assign a
/
is
the
where
cost of say, I+ck, for inserting a block of k characters,
cost,
\"start-up\"
metrics.
We can
character.
There are many other useful
and c is a cost per subsequent
the
formulation than by modifying
model them more easily by using the shortest-path
original problem. We can add edges anywhere we want and assign any cost to them,

whose cost is

problem.

without

the problem.

changing

10.2.4

a Triangle in Undirected Graphs

Finding

strong correlation
can be represented by

is a

There

vertices

the ijth entry is 1 if

and

is a weighted

graph,

to

of edge

to

the

weight

associate

G =(V, E)
and

only

ith

vertex

and

matrices.

matrix

A,

adjacency
if (v,, v7) e E. If G
we define A as an n x

(v,, v;) or to
with

n vertices

with

if the

graphs

its

only

then

a matrix

between

0 if

graph.

to

is undirected,
n

matrix

this edge is not

For example,

and m edges is
is incident

the

an

jth

A graph G=(V,
is an n xn matrix

which

n xm

edge.

then

such

in the
the

matrix

that the

graph.

incidence
in

A is

which

E)

symmetric.
ijth

entry

with

in which

If G

is equal

There are other

ways

of a graph
the ijth entry is 1 if
matrix

Reductions

326

The correlationgoesbeyond

be

understood

better

Here is one

n vertices

designan

use of

making

by

this

analogy.

running

graph

with

determine

whether

subsets of

vertices,

of the

time

whose

Can we

is sparse.

three

of

subsets

algorithm

an algorithm to

undirected

connected to one another.

is to check all

1)(a?-2)/6
so the

be a connected

(V, E)

Design

edges.

solution

straightforward

constant time,
graph

and

G =

Let

three vertices all

contains

(^) = n(n-

be resolved

example.

The Problem

The

by

problems can

algorithmic

many

surprisingly,

Many properties of graphs can

corresponding matrices. Similarly,
many
at the corresponding graphs. Not
looking

representation.

at the

can be discovered

of matrices

properties

mere

looking

by

resulting

time is 0(mn)

running

do better than

and each

subset can

algorithm

is 0(n3).

are

be

checked

in

It is

(Exercise7.118),which
to show

We proceed

that?

There

vertices.

three

an

possible to
if the

is better

is

which

algorithm,

The main purpose of this

discussion
asymptotically faster, but is far from being intuitive.
is to illustrate the relationships
between
graph algorithms and matrix
algorithms.
Let A be the adjacency
matrix
of G. Since G is undirected,
A is symmetric.
Denote by A2 the square of the matrix A, namely, A2 =A xA (the product
is the usual
matrix
We
want
to
the
between
the
A2
entries
of
and
the
product).
study
relationships
G.
definition
of
matrix multiplication,
graph
By

A2[iJ]=idA[iAYA[kJ].

Therefore, A2[i, j]>0

if

of the

[ky

and

j ] are

1.
and

k*j,

terms

In

both / and

and

if there

only

graph,

j are

[/,

exists an index k
j ] > 0 if there exists
to

connected

k.

such that both

(We assume

a vertex

ky such

the

graph

that

and

[/, k]

* /,
not
does

that k

that
means
that there exists a
/]=0 for all /.) However,
if / is connected to j and A2[iy j] >0. Thus, there
triangle
involving
only
2
exists a triangle
in G if and only if there
are i and j such that A [/, j ] = 1, and A [/', j ] > 0.
The
discussion
above
A2 and then check
implies an algorithm. We first compute
the
condition
above for each pair / and j. It costs O (n2) to check all pairs,
so the running
time of the algorithm
is dominated
of
matrix
the
time
multiplication. We
running
by
have thus reduced
the problem of finding
to that of Boolean matrix
a triangle
in a graph
contain

self

loops;

two

problems
and
multiplication

0(az281).

We

multiplication,

of

precisely to matrix
are equivalent).

(more

multiplication
these

hence, A [/,
/ and j if and

obtain

also

can

and

an

obtain a

0(n3/(\\ogn)2).

We

use

for finding

algorithm

in

practical algorithm

for

have

multiplication,

so, in

the complexity

of Booleanmatrix

general,

the

reduced
complexity
multiplication.

this
of

will

see

in Section

use Strassen's

now

can

We

algorithm

the

but we

squaring,

triangle

Section
finding

9.5.3
a triangle

for

algorithm

whose

10.4.2

running

for Boolean
with

a running

graph problem to Boolean

this graph problem is 0(M), where

that

matrix

time

is

matrix

time
matrix

is

10.3

Reductions

10.3

Reductions

Involving Linear

Programming

327

Involving Linear Programming

areas of algorithm
examples of reductionsfrom different
one
to
that
we
could
another
so
use
a
known
design.
problem
algorithm.
This section also presents reductions,but with a slightly different approach. Instead of
for a candidate for a reduction whenever a new
we explore some
arises,
looking
problem
4'super-problems,\"to which many problems can be reduced. One such super-problem,
the most important
is linear programming.
There are efficient
one,
perhaps
algorithms
for solving linear programming, although
are not simple. A thorough
discussion
of
they
linear programming is beyond the scope of this book. In this section, we only define
and show several examplesof reductions
some variations
of the problem,
to it.
The

section

previous

tried

We

included

to map

Introduction

10.3.1

and Definitions

a certain
are
function
many
problems that involve maximizing or minimizing
involves maximizing
subject to certain criteria. For example,the network-flow
problem
flow
constraints
and to the conservation
the
function
constraints.
subject to the capacity
in
Linear programming is a general formulation
of such
cases
where
the
problems
and the constraints
can also be written
linear
functions
function is a linear function
using
of variables. An objective
in
the
be a vector
following
way. Let x = (x\\,x2, \342\200\242\342\200\242\342\200\242,-*\342\200\236)
is defined as a linear
function
the variables of x;
function
involving
There

c(x)

^CjXh

where thee,

s are

constants.

(10.1)

some constraints
is to find the values of x that
of linear programming
satisfy
function.
We shall see later that, if
the value of the objective
(listed below) and maximize
with a similar minimization
it is easy to replace the maximization
objective
necessary,
with
of
three
we
a
form
linear
define
First,
types of
general
programming
objective.
will
show
that the
we
all
of
which
needed
for
all
not
are
Later,
constraints,
problems.
with
two
constraints.
to
a
of
can
itself
be
reduced
general problem
problem
only
types
Let a!, a2y...,ak be vectors of real numbers, each of length ny and let b i, b2,.., bk
be real numbers.
The inequality constraints are as follows:

The goal

ax

-x<bx

a2 'X <b2

(10.2)

ak-x<bk.
(Exceptfor
The

jc, all
equality

other symbols
constraints

are constants.)
are similar:

Reductions

328

-x=dx

ex

e2 'X

=d2

(10.3)

em'X=dmy

1 \\, e2>

where

also vectors

\342\200\242\342\200\242\342\200\242,
em are

We also

Xj

> 0,

for all j

a given

function

(10.3),

and the

used in

all

of {1,

nonnegative
the

real numbers.

\342\200\242\342\200\242\342\200\242\302\273
^m are

constraints separately (even

constraints).

previous

2,..., n).
can be

problem

to

subject

the

inequality

constraints

as

formulated

follows:

(10.4).

constraints

problem.
we can get rid

must be

that

show

a loss

of either

of generality.

or the

the equality

inequality

constraints,

Let

(10.5)

erx=d;
be

an

the

maximize

(10.2), the equality

Of course, not all constraints

constraints

of the

without

both,

dx, d2,

(10.4)

nonnegative

instances
first

We

but not

following

and

/>,

programming

(10.1)

c(x)

subset

linear

The

of size ny

represented as a specialcaseof

can be

P is

where

usually

they

though

the

add

We can

constraint.

equality

arbitrary

for

substitute

(10.5)

the following

two

constraints:

inequality

(10.6)

erx<dh
and

(10.7)

-erx<-dh

we can

Alternately,

Given a general

replace the

with equality constraints.

constraints

inequality

constraint

inequality

(10.8)

arx<bh

we can

a new

introduce

Such a

yh

and

replace

is

variable

is said

called

to be in

a slack
standard

variable.

will

following:

not
existing

describe

linear

program

with

only

equality

form.

of these cases, replacing

one
of constraints
constraints may cause the number
these transformations.
good idea to perform
We

the

with

(10.9)

In both

here that the

(10.8)

and^>0.

ai-x+yi=bh

constraints

variable,

set

of constraints

to increase.

with

Therefore,it

is not

always

any algorithm for solving linear programming. We only

for linear programming are quite
fast
in practice,

algorithms

of

set

another

note

and

10.3 Reductions Involving

thus a reduction to linear programming

Linear

just an exercise

is not

329

Programming

a good

but

way to

solve the

problem.

10.3.2

in

introduce

seldom

life are

real

Problems

has to

to Linear Programming

of Reductions

Examples

the

to make

definitions

right

in linear

directly

given

the

problem

One
programming formulation.
fit this formulation. Here is one

example.

The Network-Flow Problem

(This

in
in Section 7.11.) Let the
detail
variables
A*i,jt2, ...,*\342\200\236
of the flow for all the edges (n is the number of edges here). The
is the value of the total flow in the network

is discussed

problem

values

represent

the

objective

function

c(x)=

\302\243*,,

ieS

S is

where
the

<

where

(10.4),

that

inequality

constraints correspond to

edge /.

The

constraints

equality

to

correspond

the

constraints:

\302\243 x,r
leaves v

whole set

is the

the

maximize

maximum

A Static

for alive

= 0

\302\243 Xj
enters v

.v,

the nonnegative

Finally,

The

source.

the

< a?,

capacity of

is the

conservation

x,

/, 1 < /

for all

Cj

c,

leaving

constraints:

capacity
Xj

of edges

set

the

constraints

{1,2,..., /?}).

function

objective

to all

apply
We

V-[s, t).

leave
under

variables (i.e., the

the reader to verify

set

it to

constraints

these

f\\ as
that

correspond

defined

the values

in

of

indeed to a

flow.

Problem

Routing

Let G =(V, E) be
that each

node

messages

in one

undirected

an

v,
unit

of

size). Supposefurther

graph representing

network

the

in

time
that

has a

limited

buffer

corrmunication

space,

network.

and can

receive

Suppose
only

\302\243,

have the same

(we assume, for simplicity, that all messages
of messages that can be
there
is no limit
on the number

of messages. The
infinite
supply
in one unit
of time in
edge should
carry
on the network.
(This is a static routing
all nodes
since we assume that
transmission
problem,
always want to transmit;
usually,
In a graph-theoretic formulation,
needs are dynamically
the problem
is to
changing.)
that
the
sum
such
of
to
the
the
of
all
incident
to
node
edges
assign weights
weights
edges
and the total sum of weights is maximized.
v, is <Bh
can be easily formulated as a linear programming
This graph-theoretic
problem
variable
with
a
each
can
associate
We
the
xf
edge e,=(v, w), indicating
problem.
The objective
function is c(3c) = \302\243\342\200\242*/\342
The
of messages passing through
number
er
transmitted

through

any

link,

problem is to decide how

order to maximize the total

and

that each

node has an

each
messages
of messages
number

many

Reductions

330

constraints are as follows:

forallveV,

Xi<Bh

\302\243

is incident to v,

e,

and

for all/.

jc/>0,

Problem

The Philanthropist
there

that

Suppose

are n organizations

sciencedepartments.
year, as well as a limit
be 0 for

may

a,j

of

limit

a^

/ has

organization

amount

on the

to contributed

want

that

it

of

limit

s, on its

money to k computerfor the

contribution

total

to contribute to department

is willing

j (e.g.,

some departments).

In

general

s, is

smaller

than

each

\302\243fl,;; therefore,

that each department j has

has to make somechoices.Furthermore,
suppose
total
amount
of money
it
can
receive
on
the
constraint
(this
may be
tj

organization

Each

an algorithm
that
nevertheless). The goal is to design
no
to
fairness).
(with
regard
is a generalization
of the matching
in
This problem
introduced
Section
problem
It can be solved by matching
but it also has a simple linear
7.10.
techniques,
There are nk variables xijy 1 </<\302\253, 1 <j<k,
formulation.
the
programming
representing
amount
of money
/
is
to
contribute
to
The
organization
willing
department j.
objective

unrealistic,
maximizes

it

but

total

the

is interesting
contributions

is

function

c(x)

^Xij.

'j

are the

constraints

The

Xjj

<

for all /,

a,j

following:
y,

/t
\302\243
Xjj

<

Sj

for

all /,

and
n

2>,7<0
In

addition,

The

Let's

for all j.
of course,

change

the

must

be

nonnegative.

Problem

Assignment

donate money

all variables

problem
one department and

philanthropist

to only

slightly
that

that

insisting

by

In other words, we make

it
organization.
Each possible match has a dollar amount
attached
weights.
but also one that maximizes the
only a perfectmatching,

total

usually

one

bipartite

problem.

weighted

matching

problem,

each

organization

department accepts money from

a standard
matching problem, but

each

or, as

it

is

to it,

and we

donations.

called,

want

to

only
with

find not

This problem is
an assignment

10.4
The variables for

problem.

one edge is

exactly

each edge (/, j)

problem

must

be

the

notion

to capture

connectedto

the

so

previous

and of

jt/; for

0 otherwise.

The

becomes

c(x) = ^aijxij.

(10.10)

are the following:

constraints

The

that

insist

must

a variable

assigning

by

is selected,

edge

of the

matching. We

of a

331

Bounds

Lower

those

from

different

We do

node.

each

of 1 when

a value

with

function

objective

this

have

somehow

We

for

Reductions

= 1 for all

\302\243
Xjj

/,

and

= l

2>//

for all/

These constraints

that no more than

must be nonnegative.

one

guarantee

all variables

addition,

is selected for

edge

each node.

In

a yes or no
deficiency. The variablesrepresent
real
We
have
that
be
numbers!
to
add
constraints
optimal
may
limit
the values of the variables
to either 0 or 1. This
is generally
hard
to
do.
Linear
very
must be integers are called integer linear programs. Solving
whose
variables
programs
of the problems
discussed in the previous
them
involves
integer programming.
Many
as integer linear programming problems. However,
formulated
chapters can be naturally
linear programs are usually
linear
can be efficiently
solved,
integer
although
programs
We
issue
in
the
next
discuss
not always)
difficult.
this
(The
(but
chapter.
very
can
linear
the
be
solved
see,
programming;
assignment problem, by
way,
efficiently
by
for example, Papadimitriou
and Steiglitz
[1982].)

This formulation

choice,

much to

the

applies to
lower bound
common

for Lower Bounds

A as

problem
proofs.

errors

in the

A Lower

Consider

the

problem

well. We

Another

that,

under

be modified

can

problem

then
\302\243,

present three

examples

example

is presented

in

\342\200\224
without

too

adding

bound for problem B

the use of reductions for
section, which deals with

a lower
of
next

the

use of reductions.

for Finding Simple

Bound

of connecting

polygon (see Section8.3). We

true

problem A

\342\200\224
to solve

time

running

10.4.1

also

major

show that an algorithm for

can

we

one

values

Reductions

10.4
If

has

their

but

certain

have

a set
seen

assumptions,

of

points

how to
this

in the

solve this

problem

Polygons
plane
problem

cannot

by

using

be solved

closed

simple

sorting.

more

It is

quickly

we
the algorithm
for the simple closed polygon
Therefore,
presented
sorting.
without
be
cannot
(When we say \"improvement,\"
improving
sorting.
improved
problem
a
than
constant
more
we mean an improvement
factor.)
by
than

Reductions

332

10.1

Theorem
\342\226\241

It is possibleto
simple

Proof: Consider

0(T + n), given

in time

sort

that

problem

polygon

in time

runs

a (black-box)

algorithm

for

the

O(T).

on a circle (see Fig. 10.2). The

only
way to connect
polygon is to connect each point to its neighbor on the circle.
if two points
that
are not neighbors are connected, the
line
Otherwise,
connecting
two
of
the
into
that
cannot
be
connected
rest
the
without
points
groups
separates
to the sorting problem.
Consider now an input xx,x2,
If we
this line.
...,*\342\200\236
intersecting
we
could
use
it
in
for
the
to
sort
the
box
a black
had
simple
polygon
problem,
following
to y x, y2,..., yn, such that the >>/S are angles
The inputx {, x2,..., xn is first converted
way:
in the range -180 to 180degrees,with the same relative order as the jc/S. The angles are
on the unit circle. The point
to jc, is the
to points all lying
converted
then
corresponding
some
line
the
circle
with
to
fixed
circle.
These
the
on
conversions
angle y(
crossing
point
a simple
can be done in linear time. We can now use the black box for constructing
a
of points
in
As
we
from
set
time
O
this
mentioned,
(T).
simple polygon must
polygon
But that means that
we can scan the
connect each point to its neighbor on the circle.
the sorted
in time O (T + n).
in order and find
order of the original
\342\226\2
sequence
points

these points

the model

that was

proved

bound for
must

the

include

assumption.

the decision-tree
bound

decision

for

way

sorting

tree.

for

the

simple

that

we

assume.

the decision-tree model.

we must use the same model.

problem,

black box that solves

that is consistent with
We

model. In

have

then

this

polygon problem, we have to be

The Q(n log n) lower bound
for

assumed

6.4.6

Section

simple-polygon

in a

this

in

that the

assume

comparisons

points

a lower bound
of computation

obtain

To
about

a simple

into

case,

the
the

simple

the

on

the

the

model.

from

sorting

lower

reduction

The theorem

is also

first

or

coordinates

numbers

to points.

must

with

consistent

since the proof of

type of queries allowed

\\/yi

conversion

this

That is, we

the reduction is valid

did not make any restrictions

Thus, a comparison involving

Figure 10.2 The

use

polygon problem uses 0(T)

decision-tree

to show that

To

careful

of

the lower
in

the

the

point

10.4 Reductions for

to

corresponding

the

tree that solves

the

the decision-tree

connecting a set of
in the

comparisons

model, the

tree.

the decision

a decision

into

height.

the

in

a simple polygon

of finding

problem

points

given

requires Q(n log

plane

n)

case.

worst

the fact

establishes

reduction

This

in

change

significant

in

can be transformed

10.2

\342\226\241
Corollary

Under

as one comparison
problem

simple-polygon

without

sorting,

still counted

y,- is

angle

that solves

tree

decision

333

Bounds

Lower

\342\226\

that

at the

is really

sorting

of

heart

the simple

solving

polygon problem.

Simple Reductions Involving Matrices

10.4.2
In

Section

in

we saw

9.5,

matrices
practice.

to multiply
is
entry
equal to the
it is easier to multiply

nonintuitive

very

matrices

in which

It is natural

to ask

(i.e.,

the

two matrices.

ways

ijth

whether

entry)

y/th

symmetric

Symmetric

occur commonly
It is
matrices.

3
helps in finding better expressions for multiplying,
say,
for multiplying
symmetric
by 3 matrices. This may lead to a better asymptotic algorithm
that this is not
the
case.
We prove that
two
matrices. We now
show
multiplying
a constant factor, as is multiplying
two arbitrary
to within
symmetric matrices is as hard,
that

possible

entirely

symmetry

matrices.

Let's denote
ArbM,

the

the product of

of computing

problem

the product of

of computing

that

and

two

two

symmetric

matrices

arbitrary

matrices

by SymM.

by
is

It

than ArhM (since SymM is a special case of ArbM).

that SymM is not harder
that solves SymM. We show that we can use this
now that we have an algorithm
as a black box to solve the
more
general
problem ArbM. Let A and B be two
from A by
of A (i.e., the matrix
obtained
matrices.
Denote
by AT the transpose

obvious
Suppose
algorithm
arbitrary

every

exchanging
involving

ij

entry

of two

a product

the entry

with

In x In matrices,

ji).

utilize

We

is easy

which

the

following

expression,

to verify:

\"

(The 0s
the

fact

using

stand

two

algorithm

AB

n xn

for

that the
the

Br

\"0

AT 0

ATBT

whoseentries

matrices all of

for

side are

on the left

matrices
the

problem

SymM.

are

// there is an
real matrices
timeO(J(n)

can

the

following

follows

to

algorithm

that

time

0(T(n)),

in

compute

+ n2).

the product of two

such that T(2n) = 0(T(n)),

computes

the product

of

two

arbitrary

nxn

from

find their product

their

algorithm

theorem.

symmetric
then

real

there

nxn
is an

matrices

by

product

10.3

Theorem
\342\226\241

algorithm

The reduction

But the upper-left side of

can
solve ArbM by using the

exactly the product AB. Hence, we

of twice the size. Thisleadsto
SymM on two matrices
contains

0.)

We

symmetric.

in

for

Reductions

334

Proof:

two

Given

arbitrary
as shown in

product
compute
and to construct the
their

two

symmetric

nxn
to
matrices, we use the assumed
algorithm
(10.11). It takes 0(n2)
steps to compute AT and BT
them.
The theorem
matrices, and T(2n) to multiply

follows.

\342\226\

The
polynomial

do not

to

suggest
the

utilize

is faster

it. This
it in

using

symmetric

0(T(n)),
the
compute

Proof:
following

such

that

the
computes
= 0(T(n)),

T(2n)

of

product

the

in

As

enough
A

two

proof of

a matrix

multiplication

that

algorithm

reduction.

arbitrary

real matrix
square ofannxn
then there is an algorithm
nxn
real matrices in

Theorem 10.3, we need

the product of two

to obtain

information

in
to
time

~AB

0\"

BA

arbitrary

matrix

matrices.

whose

square

This is

done

\342\226\

Errors

Common

Reductionsshould

to find

expression:

The theorem followsimmediately.

10.5

for

similar

overly

+ n2).

0(T(n)

that

algorithm

time

by

to multiply.
properties of a matrix
practice

for example, any

restrictive;
for
a
lower
bound.
We
establishing
only
it is impossible
Theorem
10.3 tells us that
not

10.4

If there is an

the

is good

reduction

Here is another

asymptotically.

Theorem
\342\226\241

contains

= O(T(n)) is

that T(2n)

assumption
satisfies

The following
are examples
of common errors one
a reduction. The most common
error is to apply
the reduction
in the wrong
order.
This mistake is more prevalent
in reductions
for lower bounds. The
reduction
should
in this case that one problem
establish
P is at least as hard as another
whose
we
with an arbitrary
know.
We
need
to
start
problem
Q
complexity
already
instance of Q and to show that it can be solved with
a black-box
solution for P.

can

make

when

Consider,for

be

used

with care.

attempting

the following
of data compression
example,
attempt to reduce the problem
via Huffman's encoding(Section6.6)to the problem of sorting. The goal is to prove a
lower bound of Q(n log a?) for the complexity of Huffman's
encoding.
The main observation is that,
are wide apart,
of the characters
if the frequencies
then
the tree becomes
so unbalanced that it can be used for sorting
(see
Fig. 10.3). In that
the
case, the characters
will appear in the tree in decreasing order of frequencies
(with
at
the
Huffman's
of
character
means
that
But
that
the
tree).
highest-frequency
top
the tree is at least as
encoding can be used to sort these frequencies. Therefore, building
of \302\243l(n
hard
as sorting,
and a lower bound
be
seems
to
log a?)
implied.
The error in this argument comes from the fact that we started with a special case
We considered only
of the sorting
those
that that are wide apart.
problem.
frequencies

10.5 Common Errors

Figure10.3 A
To prove

a lower

After

the proof

all,

sorting.
by

the

We

must

should

start

Huffman's

we

for sorting,

bound

with

show

for frequencies

tree

Huffman

arbitrary

numbers

and

We

will

encoding algorithm.

wide

apart.

with an arbitrary instance of sorting.

encoding can be used to perform
any
show
that these numbers can be sorted
this error further in the next
discuss

start

must

Huffman's

that

are

that

335

chapter.
The trick
As it turns out, we can modify the arguments
above and save the proof.
is to spend some time
the input of the
(which must be
sorting
problem
changing
of distinct
Let the input
be a sequence
so that it conform with
our
arbitrary)
goals.
=
are
X
We
can
assume
that
the
numbers
distinct,
(jcj, jc2,...,jc\342\200\236).
integers
positive
the lower bound for sorting
to distinct numbers as well (in fact, the lower
because
applies
The Huffman's
bound was proved for distinct
encoding corresponding to
integers).
in
X
can
be
tree;
thus, the
any
general
frequencies that are equal to the numbers
above
cannot
be used. However, we can replace each
jt, with,
say, y; = 2X'.
arguments
tree will have the
for any positive integer m, we have 2m > \302\243
2', the Huffman
Since,
i <m
form

sort

in Fig.

shown
the

reduction

y,-s.

We

10.3. So, it is possible

now must make sure

(computing

the yp from

the

jt,s,

to use
that

the

in this

the

Huffman's

extra

encoding

computation

case) is

not

prohibitive.

algorithm

involved

in

to
the

Computing

be quite expensive,
but that is irrelevant
case, because the lower bound
the number
of
involves only comparisons.
We made no assumptions about
other operations (see again
the
we established that building
Section
Therefore,
6.4.6).
Huffman's
in the worst
case
the
under
encoding requires \302\243l(n
comparisons
logn)
decision-tree model. (It may be possible to build the tree more quickly with an algorithm
that does not conform
to the decision-tree model.)
have
to be careful that
the
We
also
reduction
does not impose significant
the
Consider
in
discussed
Section
and the
5.11,
knapsack
problem
inefficiency.
5.17. (The extension was to solve the knapsack
in Exercise
extension to it addressed
an unbounded number of
problem where each item can be included in the knapsack
the
of
extended
to
reduction
A
(in
times.) straightforward
problem the original
problem
Let the size of the knapsack
be
at most once) is the following.
which each item appears
powers can
for sorting

in this

Reductions

336

of size Sj cannot be included more than

So, we can replaceeach
K/s,- times.
items of the same size in the original
extended problem with [K/Sj]
problem.
it is not very efficient, since we have increasedthe
is
size
reduction
this
correct,
Although
solved
more
This
can
be
the
of
problem
efficiently.
problem considerably.

K.

item

An

in the

item

10.6
It

is

Summary
idea to look for

a good

always

differences and similarities

problems. Given a new
problem similar to a
become

problems

between

similarities

problem,
known

only

apparent

reductions betweenmatrix
several examples of reductions

and

after

all

\"Is this

cases),

similarities between two

reductions
are exhibited.
The

complicated

graph algorithms
in this chapter,

into both

insight

the

Sometimes,

problem?\"

studying

By

problems.

two problems, one usually

gains
be (in almost
the first thought should

between

are especially interesting.

We
and we will see more examples

have

seen

the

next

in

chapter.

Linear

are very

and

important

and should

be

Notes

Bibliographic

Hall'sTheorem

were described

programming

integer

studied

in detail

and

Further

too

by anyone

in

briefly

this

interested in

They

chapter.
algorithms.

Reading

of Hall's theorem can be

discussion
[1935]. A detailed
for
Brualdi
[1977] or M. Hall
any combinatorics book; see,
example,
\342\200\224
\342\200\224
in particular,
between
and
[1986]. The relationships
finding small cycles
triangles
matrix
are discussed by Itai and Rodeh [1978]. The two reductions
multiplications
matrices
Similar, but more
(Section 10.4.2) are due to Munro
[1971].
involving
reductions can be obtained betweenBoolean
matrix
and the
complicated,
multiplication
transitive-closure problem (Fischer and
matrix
between
and
[1971]),
Meyer
and
matrix
inversion
multiplication
(Winograd [1970b]).
Linear

detailed
algorithm,

is due

to P. Hall

almost

in

found

discussion and
has been

Nevertheless, Klee and

numerous
in

However,

ellipsoid

by

Dantzig

proved that the

Khachian [1979] was

[1972]

Minty

time

running

algorithm

in 1947

(see Dantzig
algorithm, called

examples).
Dantzig's
use since the 1950s.

extensive

is exponential.
simplex algorithm
for linear programming whose
also
algorithm, which is known
the

first solved

was

programming

is

It

worst-case
the

is polynomial

as the

ellipsoid algorithm,
has other applications

first

in the
works

fast

worst

very

(see, for

running

case.

poorly
example,

the

and

to exhibit

for

[1963]
simplex

practical.

time

of the

an algorithm
Khachian's

in practice.
Grotschel,

linear
a
received
programming was introduced
[1984]. Karmarkar's algorithm
by Karmarkar
lot of attention,
for being
and
extensive
research, because of its potential
triggered
in certain cases. For more on linear and
superior to the simplex
integer
algorithm
and Steiglitz
see, for example, Papadimitriou
programming,
[1982], and Schrijver [1986].
10.7 can be found
A solution
to Exercise
in van Leeuwen
[1986]. Exercise 10.8is
from
Exercise 10.11 is from Maggs and Plotkin
Even
Exercise 10.22 is
[1979].
[1988].
from Aho, Hopcroft,
and Ullman [1974].
Lovasz,

and

Schrijver [1981]).

Another

polynomial-time

algorithm

for

337

Exercises

Exercises

10.1

and

10.2

10.3

by using the

theorem

Hall's

Prove

techniques developedin

the

sections

on network

flows

matching.

bipartite

Solve the

variation
of the sequence-comparison problem. The input
is two
following
the
is
B
so
it
becomes
to
A.
The
A
and
and
to
edit
that
edit
\302\243,
equal
goal
steps are
sequences
the
usual ones: insert, delete and replace
(or match). The cost of a step, however,
depends
of the corresponding characters.
The cost of inserting a
on the position in the sequence
the yth
character at the z'th position in B is d, where c is a constant, and the cost of deleting
character of B is cj. The cost of replacing a character with another character is still 1. The
edit sequence.
algorithm should find the minimum-cost

(in some

a reduction

Find

8.18) and

plane (Exercise

between

direction)

of marking

that

the problem of finding

maximal
on the line for containment

in the

points

intervals

(Exercise

8.26).

10.4

Department D at
The

for its Ph.D. students.

X administers a qualifying
examination
University
divided
into n areas such that there are qt questions
consists of Q questions

examination
n

in area

= Q).

(\302\243<7,
/=i

questions for

write

who

There are

that can

questions

be

used,

P \\,P2

professors

examination.

the

>

that

\302\243/?,
/=i

Q. A

(these

are not their real

that each professor

Suppose

and

Pk

/>,

committee is responsible

has

names)

overall

for selecting

/?,

the

We assume
from the questions supplied
by the professors.
and that they are all good.
Assume, furthermore, that each
questions are unique,
of
r is a constant independent of the professor)
professor insists that no more than r (where
in later
will be used (so that he or she can use the remaining
his or her questions
questions
for the examination under these
years). Design an efficient algorithm to selectthe questions
so.
is
to
do
or
to
determine
that
it
constraints,
impossible
for the examination

questions
that

10.5

all the

variation of

matching problem. Suppose that there are

bipartite
Consider the bipartite graph formed
to n universities.
an edge
the students and the universities as the two sets of vertices and including
by having
Find an
to admit that student.
between a student and a university if the university
agrees
such that no more than two
the number of students that are admitted,
algorithm to maximize
Solve the problem by
students are admitted
to each university (there are no preferences).
a reduction to the regular bipartite matching problem.
exhibiting

Consider

the

following

In students

who

10.6 Hereis another

the

to be admitted

want

variation

of the bipartite

matching

problem.

Suppose

now

that

there

are n

the graph in which

the courses
and
training courses and n trainees. As usual, we consider
the trainees are the vertices and there is an edge between a trainee and a course if the trainee
for the course. Each course can have
at most two trainees, and each trainee can
is qualified
to
take at most two courses. Design an algorithm
(by a reduction to a known
problem)
no preferences
are given, and there are no scheduling
maximize the registration. (Again,

problems.)

10.7

Let

G =(V,

E) be

an

undirected

graph

such

that

each

vertex

v is

associated

with

an integer

338

Reductions

b (v) < degree(v).

more

than

in G

A ^-matching

incident to

edges

b(v)

it.

matching problem.)

Reduce

of finding a

the

problem

is a set of edges of E such

b(v) =

(If

maximum

^-matching
maximum

for

is one

is

of

to that

the

exactly

number

maximum

with

^-matching

each vertex

that

all v, then this

has

no

regular

of edges.

a maximum

finding

matching.

10.8 Let G=(V, E) be an

number of vertex-disjoint
10.9

directed

acyclic

graph.

that include

paths

all

Design

an

to find a minimum

algorithm

of G.

vertices

Let G = (V, E) be a network with source s and sink t. Assume that G is planar,
it
namely,
that such a
can be laid out in the plane such that no edges intersect. Assume furthermore
and that both the source and the sink
layout is given to you (in a reasonable representation),
an algorithm to find a minimum-cost
cut in G
of the layout.
lie on the outside
Design
using the

without

maximum-flow

algorithm.

10.10

Exercise 8.40 can be solved by reducing the problem to that of minimum-weight matching.
leads to an inferior
is
(In this case, the reduction
algorithm since minimum-weight
matching
and prove its validity
harder than a direct solution.) Show the reduction
(i.e., prove that the
satisfies the conditions of the problem).
matching
corresponding minimum-weight

10.11

an MCST in an undirected graph to a bottleneck shortestReduce the problem of finding

in which we try to
bottleneck
(A
problem.
path
problem is a minimization
problem
minimize the maximum
rather than the sum of values;
value,
so, a bottleneck shortest-path
involves
rather
than the cost of
paths whose maximal-cost edges are minimized,
problem
the whole
that the MCST problem can be solved by shortest-paths
path.) As a result, show
are usually
more
than MCST
techniques. (Although
shortest-paths
algorithms
expensive
the reduction
for parallel algorithms.)
can be helpful
algorithms,

10.12 The
v, x],

x2, .., jc*,

e

u is

that

variables.

allowed

10.16

vertex v, such that there

The cost of a directed

costs of the two

The

endpoints

u are

and

is a
path

ignored,

you

have
that

The first type

relations (of course,with

to the

a linear
in our

one

Section

in

programming
definition

Show

how to

a bad example, that

linear programming formulation.

exhibiting

in a

programming than the one given in Section 10.3

imposes the \"<\" relations, and the

of linear
constraints:

can be reduced

(Recall

by

\302\243<:(*,).
i=i

formulation
inequality

restricted to be nonnegative.)

10-15 Show,

w.

defined as

the \">\"

imposes

type

formulation

10.14 Suppose

vertex

the
\302\243,

An even more general

allows two types of
this

a distinguished

with

each

cost of getting from v to u is 0. Design

an efficient algorithm to find the
to Exercise
7.47,
paths from v to all other vertices. (This exerciseis identical
insist on a solution by reduction.)

but here we

second

E)

associatedwith

cost c(w)

so if (v, u)
minimal-cost

10.13

directed graph G =(V,

is a

input

positive

of

algorithm

linear

reduce the
constraints

different

Show

coefficients).

that

10.3.
that

can only handle

programming
general

of the

not

problem
type

all

nonnegative

variables

were

to this one.

ax*b

should not be

that there are n people

in a scientific conference whose
is to maximize
goal
the
Not
can
of
ideas.
We
else.
ideas
with
everyone
exchange
represent
exchange
everyone
conference by an undirected graph, with the vertices associatedwith the people such that / is
connected to j if /' can exchange ideas with j. (One can also define
a directed
version.)
Suppose

Exercises

339

for talking is limited. For simplicity, we assume

hours
hours. That is, every person can spend at most h hours
here with scheduling. We assume that time is also spent on
talking. We are not concerned
other
to arrange any
so there is sufficient
For
activities,
flexibility
possible
meeting.
to the others, and let h = 1.
example, suppose that there are three people, each \"connected\"
If two of them
talk
there
is only 1 hour of
to each other for the whole hour, then
conversation. If, on the other hand, each one talks to each other for half an hour, then
his or her time and there is 1.5 hours
of conversation.
We want to
everyone exhausts
maximize
the total conversation time. Formulate
this
in
terms
of linear
problem
or reduce it to another problem that we have already discussed.
programming,

Supposefurther

10.17 Consideragain

global

the

of h

bound

philanthropist

amount of

on the

limits

of

the number

that

is one

there

that

each

money

problem of Section
is willing
department

10.3.2. Supposethat
to accept.

there

Solve this

variation

are no
of

the problem.

10.18 Considerthe

in an order consistent with the results of a

of arranging n players
problem
for this
competition (Exercise 7.104). Prove a lower bound of Q(/?logA7)
sorting to it. Show that a reduction to sorting can also be helpful in
reducing

round-robin
problem

by

a good

finding

10.19 Let S be a
given

in

of n points
order.

set

an

that

problem.
vertices

are

Prove

arbitrary

standard polygon

10.20 We

for this

algorithm

that

of an arbitrary

it takes

Q(a7

(i.e., in consecutive

representation

log a)

convex polygon. The points

are
time to arrange
the points into the

order).

Section 9.5.2that 7 multiplications

(instead of the nominal
8) are sufficient
2x2 matrices, and that this fact leads to a better
the product of two arbitrary
algorithm. It is possible to compute the square of an 2x2 real matrix
multiplication
does not
(Exercise 9.21). Discuss why this observation
only 5 multiplications
seen in

have

to compute
matrix
with

10.4.

Theorem

contradict

is a square matrix
10.21 A lower triangular
matrix
(fl;/) such that, if j > /, then a,, =0 (in other
on
or
all
are
below
the
main
An upper triangular
nonzero
entries
words,
diagonal).
that
the
nonzero
on
or above the main
is defined similarly,
entries
are
matrix
except
if
there
exists
an
to
an
nxn
Prove
lower triangular
that,
algorithm
multiply
diagonal.
matrix
whose running time is O (T(n)), then there exists
matrix,
by an n xn upper triangular
two arbitrary
nxn matrices whose
time is 0(T(n) + n2).
an algorithm to multiply
running
You
can assume that T(cn) = 0(T(n)) for any constant c.

10.22

if there

that

Prove

time

running

matrices
any

10.23 The transitive

A* =1

where /

running

time is

there

to multiply two nxn

exists an algorithm

0(T(n) + n2).

You

can

lower

to

triangular
multiply

assume

matrices

two

that T(cn)

arbitrary

whose
nxn

= 0(T(n)) for

c.
closure

+A +A2+

is the

algorithm
then

whose

constant

exists an

is 0(T(n)),

n x n

of an n

xn

matrix

A is

defined as

follows:

\342\226\240\342\226\240\342\226\240

+A\"~\\

identity matrix.

if A is a Boolean matrix
to an adjacency matrix
of a graph,
that,
corresponding
matrix of the transitive
closure of the graph.
then A* corresponds to the adjacency
is performed according to the Boolean rules.)
(Assume that multiplication

a. Prove

340

Reductions

b.

Prove

if the

that,

polynomial

in n,

assume that

T(cn)

10.24 Let S be a

transitive

then

matrix

= 0(T(n))

closure

can be computed in time T(n), where

can be computed in time 0(T(n)).

multiplication

for any

constant

T(n) is a
You can

c.

a weighted
undirected graph in the
points in the plane. The points define
is the complete graph (i.e., every
two vertices
are connected),
way. The graph
to the Euclidean distance between
the two corresponding
the weight of an edge is equal
and
time of an MCST algorithm for
points. Show a lower bound of Cl(n log/?) for the running
set of n

following

this

case.

in the plane. The diameter of S is the maximal

distance between
the problem of finding
the diameter
by DM. Let A and B be two
whether
A and B are disjoint
sets of n real numbers. Denote the problem of deciding
by DJ.
for DM that uses 0(T(n))
Prove that, if there exists an algorithm
arithmetic operations (you
then
there exists an algorithm
for DJ that uses
can assume
any reasonable operations),
+ n) operations.
0(T(n)

10.25 Let

S be a set of

two points

in

S.

points

Denote

^k-)-:-^.-:-l^.-.-.-.-.-.-.-.^
f1''\\\342\226\240

'-^.-:-l->:^l^J;-

-r-: -r-\342\200\242\342\226\240-'
-:-r-r:- - -: -V-CC->\\-V^

11

CHAPTER

NP-COMPLETENESS

Give

me where

to stand,

Archimedes

and

(287-212

I will

move the earth.

B.C.)

Introduction

11.1

This chapter

from other chapters. In the previous

chapters, we mainly
them
to specific
and
applied
solving algorithmic problems
techniques
can be
that
if
efficient
be
nice
all
had
would
algorithms
elegant
problems
problems. It
discoveredby a small set of techniques. But life is rarely that simple. There are still
that we have learned so
that do not seem to succumb to the techniques
many
problems
that we just have not tried hard enough, but we strongly
far.
It is possible
suspect that
we
there are problems that have no good general efficient solutions. In this chapter,
some
of these problems.
describe techniques for identifying
that we have seen so far were bounded
times
of most of the algorithms
The running
in
We
call
such algorithms efficient algorithms,
the
size
of
the
by some polynomial
input.
is quite

different

for

studied

the corresponding
is efficient if its
algorithm
size of the input n. Recall
and

call

required to represent
algorithms

is

denoted

that

problems tractable problems.In

that the

input.

is 0{P{n)\\

time

running

The

by P (for

size of

the

input

class of all

polynomial

words,

where P(n) is
is defined as

problems that

time).

other

This

can

we say

a polynomial
the

in

an

the

of bits

number

be solved

that

by efficient

may seem to be

strange

are not efficient

Surely,
algorithms
by any standard
in
run
time
107a?
that
are
are
not
even
that
efficient,
matter,
(for
algorithms
though
they
is
definition
valid
for
this
two
the
reasons.
it
allows
Nevertheless,
First,
linear).
are
we
about
to
which
and
most
of
the
explore; second,
important,
theory,
development
the vast majority
It turns out that
of the tractable
it simply works in practice.
problems
In other words, the
than others).
have practical solutions (of course, some are better
definition.

that run

in

time

O (nl0)

NP-Completeness

342

times of

we encounter

in practice
are mostly smallThe
is
also
true:
opposite
usually
quadratic).
degree polynomials
not
are
times
whose
are
than
usually
any polynomial
practical
running
larger
Algorithms
for large inputs.
is known.
no
are many problems for which
There
algorithm
polynomial-time
that are yet to be
of these problems may be solved by efficient
Some
algorithms
We strongly
that many problems cannot be solved
however,
discovered.
suspect,
like to be able to identify
so that we do not have to
We would
such problems,
efficiently.
In
this
we discuss how to deal
for
a
nonexistent
time
searching
algorithm.
chapter,
spend
with problems that are not known to be in P. In particular, we discuss one specialclass
in one class
these
called NP-complete
of problems,
problems
problems. We can group

running

above

(seldom

becausethey
any

all equivalent

are

one NP-complete
problems.

complete

for

The

chapter

problems

and

show

several techniques

These solutions

a strong

sense

\342\200\224
there

an efficient

exists

proof

problems,

they would

Even

known.

be

surely

very

for

algorithm

exist efficient algorithms

there
is no efficient algorithm

of that belief is

no

NP-complete

have been

problems

but

for all NPfor

any

were efficient
since they
complicated,
if there

of
(maybe even thousands)
which
is
is
so
this
why
NP-complete,
subject
important.
consists of two
we define the class
of NP-complete
First,
parts.
how to prove that a problem
Then, we present
belongs to the class.

for

researchers

eluded

have

in

problem if and only if there

There
is a general belief that

problem,

NP-complete
algorithms

that

algorithms

polynomial

years.

hundreds

to be

found

and

may

many

So far,

for

examples

be optimal,

not

and

solving
they

NP-complete problems approximately.

may not always work, but they are better

than nothing.

11.2
We

will

Reductions

Polynomial-Time
restrict

ourselves

those problemswhose

in

this

answer

section

is either

to decision problems;
yes or no. This restriction

that

is,

we consider

makes

only

the discussion

Most problems can be easily

to decision
converted
problems.
in
a
for
the
size
of
the
maximum
looking
matching
given graph,
>
we can ask whether there exists a
to solve the
If
how
of
size
k.
we
know
matching
\342\200\224
decision problem,we can usually solve the original
for
by binary
example,
problem
search.
A decision
Let U be
can be viewed as a language-recognition
problem
problem.
the
set of all possible inputs
\302\243
to the decision
problem. Let L U be the set of all inputs
for which
to the
the answer to the problem
is yes. We call L the language
corresponding
The decision
problem, and we use the terms problem and language
interchangeably.
the
introduce
We
now
to
L.
or not a given input
problem is to recognizewhether
belongs
in
notion of polynomial-time reduction
use
we
is
the
main
tool
which
between
languages,
and

the

theory

simpler.

For example,instead

this

of

chapter.

Definition:
Let L,
We
U2.
say that

polynomial-time

and

Lx

algorithm

L2

be two

languages from
reducible

is polynomially
that
converts

each

the

to L2

input

spaces

input
if

there

U\\e U{ to

\\

exists
another

and

11.2

well defined
is also

size of the

in the

polynomial

L{ if

such that u^e

U2

u2e

input

the

in

polynomial

input

spaces U x
size of u x.

input
the

in

and

x.

and

L2. The

if u2e

only
We

that the

assume

so,
\302\243/2,

343

Reductions

Polynomial-Time

is

algorithm

of

notion

size

is

the size of u2

in particular,

in the definition converts one problem

to another.
If we have
then
the
we
can
two
to
an
for
algorithm
compose
algorithms
produce
algorithm
Denote
the
conversion
and
denote
the
for
L\\.
algorithm
by AC,
algorithm
L2 by AL2.
an arbitrary input
Given
u x to an input u2 e U2\\ we
uxe
U\\ we can use AC to convert
then use AL2 to determine whether
to L2, which
will
tell us whether ux
u2 belongs
to L x. In particular, we have the following
theorem.
belongs

The

mentioned

algorithm

for L2,

an

11.1

\342\226\241
Theorem

reducible

If L\\ is polynomially
algorithm for L2, then

comes from

an

cases likely,

the
for

inputs

the definition

of L2 involved in the
Thus, if L \\ is polynomially

problems are equivalent

precise proof of this

If

L\\

and

can

We

essence

u3

in

reducible

relation

or

of

equivalent,

simply

nontrivial

all

particular,

polynomial-time

The

(we

algorithms

leave

reducibility\"

\"polynomial

if

tractable
the

is

of the

an efficient

(This

to L2 and L2
to L 3.

the two

still

conversion

is one

a polynomial

of

the

method we present in
algorithm

problems.

reducible

is polynomially

cannot

to

reducible

reasons
this

be found.

whether
we try to find
cannot solve efficiently,
to be hard. The class of NP-complete
known
equivalent

equivalent,
In

to

algorithms

form

An input u { in L { will be converted first to

Since we use polynomial reductions and
L3.
is

functions

conversionalgorithm.
when

theorem.

compose

to L3.

to an input

two polynomial

The

next

the

in

have

exercise).

L \\ is polynomially

from

then

all

polynomially
the other.

as an

is polynomially

Proof:
algorithm

to

11.2

\342\226\241
Theorem

L 3, then

are

L2

because
fact

is shown

as

and

reducible

polynomially

transitive,

It is

versa.

problem.

Two languages Lx

is

each

of L { can be
possible, and in many
reduction
are only a small
fraction
of all
reducible to L2, then we regard L2 to

inputs
L2.

that L{ is polynomially
to L \\. This asymmetry

that any input

requires

vice

not

but

x.

\342\226\

reducible

polynomially

of reducibility

of L2,

input

for

fact

the

symmetric;

is

a polynomial-time

discussion.

preceding

not

that L2

imply

equivalent

that

harder

not

that

fact

the

to

converted

be the

does

to L2

to L2 and there
is
polynomial time algorithm

the

is

of reducibility

notion

The
reducible

possible

proof follows from

The

Proof:

is a

there

function,

we chose

the

result

an

is a

a conversion
input

u2 in

a composition

L2
of

polynomial-time

to use polynomials.)

\342\226\2

is to look for equivalent

problems
When we are given
a problem
that we
it is equivalent to other problems that
are
hundreds of such
problems
encompasses

chapter

NP-Completeness

344

state

we

Before

started with
NP-completeness
the theorem, we must
explain

of

theory

The

Theorem

and Cooks

Nondeterminism

11.3

We

notions.

several

of Cook

theorem

remarkable

will skip several technical details. An excellent

and
Johnson [1979]. The theory of NP-completeness
for this area
most
of which is beyond
large theory, called computational complexity,
that help us to use the theory.
to some parts
this book. We limit the discussion
in precise
detail describing
mathematical
We have not gone into
great

discussion

and

intuitive

is

is Garey

an algorithm

is. This is not

the

of

part

of

scope

terms what

practical algorithms, as long as we

(e.g., additions, comparisons,
computers
is
algorithm
very important, however, for

for describing

important

steps that are supported

reasonable

use

[1971].

try to keep the

reference
book

will

all

by

definition of an
accesses).
memory
have
used decisiontrees to prove lower bounds in Chapter
bounds.
lower
(We
6,
proving
The most fundamental
model
of computation
restricted
is a
but this is a very
model.)
used model is that of a random
access machine.
commonly
Turing machine. Another
for our purposes,
these
and other reasonable models are equivalent
because
Fortunately,
an algorithm
from one model to another
without
the
we can transform
changing
running
A precise

time
the

more

by

We first

which

mystery.

One

and

leads

realistic goal.
the

deterministic

name

has,

algorithm

non-

rather

to all

addition

in

of the

development
the

of a
not

theory
the

for using

techniques

the regular operations of a

will call nd-choice. As the

a very powerful primitive, which

we
the nd-choice primitive
is used
to handle choices,

algorithm,

suggests,

one.

This notion is

important to the
class
than it is to

existence of this

nondeterministic

a nondeterministic
is more

was proved with

will not use any

that NP-completeness is something

as an abstract notion,
and
algorithm

to think

Nondeterminism

that require

details

any

nondeterminism.

of

notion

people

many

of the

explanation

theory.

the

think of

should

go into

not

will

to discuss

need

intuitive,

as a

it is

but

model

specific

factor. Cook's theorem,

for example,
valid for other models as well. We

a polynomial

than

Turing machines,
here, since we

use of

but

it does

so

in

an

This primitive is associated with a fixed number of choices, such that, for
each
the algorithm
follows
a different
We
can assume,
computation path.
without loss of generality,
that the number of choices is always
two.
Let L be a language
unusual

that

way.
choice,

deterministic

regular

end,

to recognize.

want

we

it decides

We say
following

during

outcome

an

in the

nondeterministic

of the

The key

difference

a language.
recognizes a language

recognize

they

way

algorithm performs
primitive, and, at the
and
between
deterministic

nd-choice

algorithm

if

the

is satisfied:

input

the

uses

with

nondeterministic

the

that

condition

Given

lies

algorithms

jc,

input

accept x.

not to

or

whether

an

interleaved

steps

nondeterministic

Given

x,

it is

execution

of the

algorithm

In other words, the algorithm

to L to arrive
at an accept
at
L
to
to arrive
belonging

of

possible to convert each

the

be to

will

must

into

algorithm

provide

outcome,
an accept

and

accept

jc,

at least
it

must

outcome.

nd-choiceencountered

a real
if and

choice such

only

if

x e

one possibleway
not provide

Notice

the

any

asymmetry

that

the

L.
for
way

inputs
for
in the

belonging

not
definition.

inputs

11.3 Nondeterminismand Cook'sTheorem

345

L may have many

to a reject outcome. We
require
only that the
paths
at
least
of
choices
for
one
xe
L.
On the other
algorithm
every
\"good\" sequence
a reject outcome, no matter which
x 4 L, we must
reach
choices
we
hand, for every input
substitute for the nd-choices.
is
sometimes
called
The nd-choice
for
primitive
guessing
reasons.
x e L is the
obvious
The running time for an input
of a minimum
length
An

input

xe

has

execution sequence
nondeterministic
to L

belonging

to

leads

that

accept outcome. The running

xe
running time for inputs

an

of

time

to worst-case

refers

algorithm

L (inputs

not

are ignored).

Let's see an

of a nondeterministic
of
example
algorithm. Consider the problem
whether
a
has
a
The
is
a
G=(V,
E)
deciding
given
perfect matching.
following
graph
We maintain
nondeterministic algorithm
for this problem.
a set M of edges, which
is
We examine all the edges
of G, one edge e at a time,
and use an
initially
empty.
nd-choice
to whether or not we include
e in M. When we are done
corresponding
all the edges, we check to see whether
M is a perfect matching. The checking
examining
in linear time, since we have to determine only whether
M contains
can
be done
exactly
whether
and
vertex
is
incident
to
one
from
M. The output
V
each
| 112 edges
exactly
edge
is yes if M is a perfect matching,
no otherwise.
of the algorithm
and
This is a correct
nondeterministic
then

there

for perfect

algorithm

is a

sequence of choicesthat

only if the existence of a

will see more examplesof
Nondeterministic
Not all

nondeterministic

the

of size exactly

efficiently

is to

problem

in the

algorithms

are very

algorithms

(1) if

a perfect

exists,

matching

outputs
put it in M; and (2) the algorithm
was proved (because of the checking).

will

matching

perfect

problems can be solved

supposethat

matching because

but

powerful,

section.

next

their

power

the

whether

is

not

unlimited.

algorithm. For example,

by a nondeterministic

determine

yes
We

maximum

matching

in

given

use the nondeterministic

matching
algorithm to find a
of size k if it exists, but we cannot easily determine (even nondeterministically)
matching
of a larger size.
that there is no matching
whose
The class of problems
for which there exists a nondeterministic
algorithm
reasonable
is
is
NP.
It
seems
to
time
a
in
the
size
of
the
called
running
polynomial
input
than
more
deterministic
believe that nondeterministic
are
much
powerful
algorithms
an NP problem
are is to exhibit
But are they?
One
algorithms.
they
way to prove that
to prove
that
that is not in P. Nobody has been able to do that yet. In contrast, if we want
that
the two classes
are equal (i.e., P = NP),
then
we have to show that
every
problem
deterministic algorithm. Nobody has
belongs to NP can be solved
by a polynomial-time
the
proved that either (and few believe it to be true). The problem of determining
between P and NP is known as the P = NP problem.
relation
We now define two classes, which not only contain numerous important problems
that are not known
to one another)
to be in P, but also contain the hardest
(all equivalent
in
NP.
problems
graph is

Definition:
NP

k.

problem

is polynomially

Definition:

belongs to NP,

We can

X is called an
reducible to X.

problem
and

called
NP-hard.

X is

(2) X is

NP-hard

an

problem

NP-complete

if every

problem

problem in
if

(1) X

NP-Completeness

346

of NP-hardness

definition

The

then

to P,

belong

that

proof

Cook [1971]proved
exhibited
NP-complete

implies
that

one such problem, which

proving that
problem,

easier.Given

a new

NP-complete

problem,

problem

is

it is

Y,

if

that,

problem is ever

NP-hard

any

imply that P = NP.

there
exist NP-complete

would

we

other

problems

sufficient

to prove that
to

reducible

polynomially

he
problems; in particular,
Once
we
have
found
an
shortly.
are also NP-complete becomes

describe

will

proved to

Y.

Cook's

or

problem,

other

any

follows from

This

next

the

lemma.

11.3

Lemma
\342\226\241

A problem

Proof:
is

to

reducible

is a

reducibility

the

definition

Y.

But

2 in

condition

By

polynomially

belongsto

problem if(J)X
NP-complete
reducible to X,for someproblem

is an

is polynomially

transitive

of

since

and (2')
NP,
is NP-complete.

Y that

every problem

NP-completeness,

Y is

relation, every problem in

reducible to

is polynomially

well.

as

\342\226\

much easier to

It is

prove that

news. As
more choicesfor proving
more good

[1972] found
hundreds of
same

24

problems

problems(maybe

even

have been

are polynomially reducible than

for the whole theory.
anchor

more problemsthat are

2'. Shortly after Cook's result
that he proved to be NP-complete.

thousands,

it is

the

depending

discovered to be

NP-complete.

on

became

In

known,

Karp
time,
variations of

Since

count

the

to prove
there is
we have

NP-complete

we

how

And

that

next section, we

problems
NP-completeness
proof. We
The
without
most
difficult
part of
NP-complete
problems
proof.
not
2
to
condition
(but
(or 2').
always)
verify
mention
the problem that Cook proved to be NP-complete,
and
The problem is known
as satisfiability
(SAT). Let 5 be a Boolean
of
normal form (CNF). That
is, S is the product
(and)
conjunctive

present
examples
also list several other
such proofsis usually
We now describe
the idea of the proof.
in

problems

condition

important

problem)
five

expression

two

Thus, Cook has found

we find more and

2 directly.

condition

the

NP

in

X and

to

reducible

polynomially

NP

of NP-complete

with

their

sums (or). For example,S = (x+y

and
+z)
(x +y +z), where addition
-(x+y+z)is
variable
to
each
the
and
and
and
or
Boolean
multiplication
correspond
operations,
either 0 (false) or 1 (true).
A
into
transformed
Boolean
can
be
CNF.)
(Any
expression
Boolean expression is said to be satisfiable
of Os and Is to
if there exists an assignment
its variables
such that the value of the expression
is 1. The SAT problem is to determine
whether
a given
a satisfying
is satisfiable (without
expression
necessarily
finding
*=1,
S is satisfiable, since the assignment
assignment). For example, the expression
Is to the variables of a
it.
y=\\, and z=0 satisfy
We call an assignment of Os and
several

Boolean

expression

a truth

The SAT problem

assignment.
NP because we can

guess a truth assignment and check that

the expression in polynomial
that SAT is NPtime.
the proof
The idea behind
hard
is that a Turing machine (even a nondeterministic
on a
all
of
its
and
one)
operations
*
the
that
mean
we
'described\"
given input can be describedby a Boolean expression.
By
if and only
if the Turing
at an
machine will terminate
expression will be satisfiable
it satisfies

is in

11.4 Examples

state

accepting
becomes

for

the

quite large

number of

described by

steps the
an

instance

is not easy to do, and such an expression

size is no more than a polynomial
its
in the
yet
makes.
NP
can
be
Therefore, any
algorithm

complicated,

machine

Turing

of a

347

This

input.

given

and

of NP-Completeness
Proofs

SAT problem.

theorem:

\342\226\241
Cook's

The SAT problem is NP-complete.

11.4

In

of

Examples

this

we prove

section,

following

set, 3SAT, 3-coloring, and

detail below. The techniques we
at the end of the
are
summarized

dominating
more
they

we must

problem,

always!) easy,

first prove

then

reduce

Proofs

NP-Completeness
the

that

\342\226\

that

a known

the

five problems are NP-complete:vertex

cover,
Each of these problems is described in

clique.

use for proving

To

section.

problem

The reduction

11.1.

To

rather

than

NP-completeness
to NP, which
is usually

prove

belongs

NP-complete problem to our problem in

order used for the five problems

we present
make them easier to understand,
This order is indicated in
the tree order.

time.

are typical,

NP-completeness

in

this

Fig.

11.1

in

order

Figure

11.1

The order of NP-completeness

proofs

in the

text.

(but not
in

Fig.

of difficulty

by the numbers

edges.

and
new

polynomial

is illustrated

section

the proofs

of a

of

the

NP-Completeness

348

11.4.1 Vertex Cover

E) be

G =(V,

Let

an

undirected

incident

G is

every edge in

The Problem
k, determine

at

Given

one of

an

undirected

a vertex

cover of G is a set of

vertices

such

that

these vertices.

least

G has

whether

A vertex

graph.

to

graph G

=(V, E)

cover containing

< k

and

an

integer

vertices.

11.4

Theorem
\342\226\241

The vertex-cover problemis NP-complete.

Proof:

size <k and

problem belongs to NP,

vertex-cover

The
check

it easily

in polynomial

time. To prove

since we
that

can
guess
the vertex-cover

a cover

of

problem is

to it. We choose the clique

NP-complete we have to reduce an NP-complete
problem
which is described next (the proof that the clique problem is NP-completewill
problem,
in Section 11.4.4). Given an undirected
be given
a clique
C in G is a
graph G =(V, \302\243),
subgraph of G such that all vertices in C are connected to all other vertices in C. In other
The clique problem is to determine, given
a
words, a clique is a complete
subgraph.
>
G
and
an
whether
G
contains
a
of
size
k.
We
have
to
transform
k,
graph
integer
clique
an arbitrary
instance of the clique
into an instance of the vertex-cover
problem
problem
such that the answer to

is positive if and only if the answer to the

is positive. Let G=(V,E) and k represent
an
Let G =(V, E) be the complement
of
arbitrary jnstance of the clique problem.
G;
graph
namely, G has the same set of vertices and two vertices are connected in G if and only if
are not connected
in G. We claim that the clique problem is reduced to the vertexthey
cover problem represented by the graph G and n-k (where n is the number of vertices in
that C = (U, F) is a clique in G. The set of vertices
G).
covers all the
V-U
Suppose
in U (they are all in G).
edges of G, becausein G there are no edges connectingvertices
Thus, V - U is a vertex
cover
in G. Therefore, if G has a clique
of size ky then
G has a
vertex
let D be a vertex cover in G.
Then, D covers all
cover^f size n-k. Conversely,
the
in G, so in G there could be no edges connectingvertices
in V -D.
edges
Thus, V-D
generatesa clique in G. Therefore, if there is a vertex cover of size k in G, then there is a
time,
be performed in polynomial
can obviously
clique of size n -k in G. This reduction
since it requires only the construction
\342\226\2
of G' from G (and the computation
of n - k).

11.4.2

Dominating

Let G =(V, E) be
that

every

the

clique

vertex-cover

corresponding

vertex

an

problem

problem

Set
undirected

of G is either

graph.
in

D or

A dominating
is adjacent to

at

set D is a
least

one

set of vertices in G such

vertex from D.

11.4 Examplesof

Given

The Problem
k, determine whether

\342\226\241
Theorem

an

G has

undirected

graph

a dominating

Proofs

NP-Completeness

G = (V,

E)

set containing

and

< k

an

349

integer

vertices.

11.5

The dominating-set problemis NP-complete.

Proof:

<k and

check

vertex-cover

The
that

dominating-set
problem
it is a dominating
set

problem to

the

belongs to NP since we

can

guess

in polynomial time.
easily
Given an arbitrary
problem.

dominating-set

We

a set
reduce

of size
the

instance (G, k)
G' that has a

our goal is to construct a new

graph
< k. We start
set of a certain size if and only if G has a vertex cover of size
dominating
with
vertices
and 2\\E | new edges to it in the following way (see
G, and add \\E | new
vw and two new
vertex
each
of
For
G, we add a new
(v,
w)
11.2).
edges
edge
Fig.
a triangle.
into
Denote the
In
we
transform
and
other
words,
(v, vw)
(w, vw).
every edge
time.
new
graph
by G'. It is easy to construct G' in polynomial
set
of
size
m if and only if G has a vertex
We now claim that G' has a dominating
of the new vertices
set of G'. If D contains any
cover
of size m. Let D be a dominating
set (both
v or w and the set will still be a dominating
it can be replaced by either
then
vw,
we can
of generality,
loss
that vw covers). So, without
v and w cover all the vertices
assume that D contains only vertices from G. But, since D dominates all the new
it is also a
it must contain at least
one vertex from each original
hence,
vertices,
edge;
for
then
each
a
cover
vertex
for G. Conversely, if C is
vertex
cover
G,
edge is covered
since all
The old verticesare also dominated
are dominated.
vertices
by C, so all the new
\342\226\
the edges are covered.
of

the

vertex-cover

problem,

vw

ft

VZ

uw

zw

Figure

11.2

The dominating-set reduction.

350 NP-Completeness

11.4.3 3SAT
3SAT

The

problem

is a Boolean

expression

is a simplification
in which each

Given a

Problem

The

contains

clause

of the regular SAT problem.

clause contains exactlythree

Boolean

exactly

expression

3SAT

that each

such

CNF

in

variables, determine

three

An instance of
variables.

is

it

whether

satisfiable.

11.6

Theorem
\342\226\241

3SAT is N?-complete.

Proof: This
the additional

can be

3SAT

guess a

SAT problem becausethere

is

requirement of three variablesper clause.We will show that a solution

used to solvethe regular SAT. First, 3SAT clearly belongsto NP. We

can

truth

arbitrary

instance

of

has

each

clauses,

verify that it satisfies the

of SAT. We will replace

and

assignment

Let E be an

which

easier than the regular

seems

problem

variables.

three

exactly

arbitrary clause of E such that k >4. We write

we do not use *,) only for convenience of notation.
several

each

clauses,

satisfiability.
into C such
=

C'

We use new

with

several

\342\200\242
\342\200\242
Let C =(x\\ +jc2+ \342\200\242
be an
+xk)
in its \"positive\" form (i.e.,
We now show how to replace C with

idea is to

variables

different)

of E

variable

a 3SAT

into

clause

the

(and

The

variables.

three

with

that transform

y\\iy2>~->yk-s

each

clause

each

time.

in polynomial

expression

to

introduce

new

without

formulation

for each

clause. C is

variables

affecting

its

transformed

that

\"'

(x[+x2+y[)-(x3+yl+y2)'(x4+y2+y3)

(**-i+**+)'*-3)-

is satisfiable if and only if C is satisfiable. If C is satisfiable,

then
one of
1. In that case, we can set the values of the y(s in C such that all
in C are satisfied as well. For
if jc3 = 1, then
we set y, = 1 (which
takes
example,
= 1), and the rest of the
care of the first clause),
since
(the second clause is okay
^2=0
jc3
if x,r = 1, then we set y\\,y2,
the rest to be 0,
to be 1, and
.v/s to 0. In general,
--^1-2
which
satisfies
C\". Conversely,
if C is satisfiable,then we claim that at least one of the
We

x,s

that C

claim

the *,s
clauses

be

must

be

must

set to

_L

all

if

Indeed,

xfs

are 0,

then

the

expression

becomes

CVi)*CVi+>;2),CV2+>;3)
Cy*-3).
expression is clearly unsatisfiable.
this
variables
three
that has more than
we
can replace any clause
reduction,
Using
with several clauses, each with
clauses
It remains to transform
three variables.
exactly
with one or two variables.
If C has only
two variables,
namely, C=(x{ +Jt2), then
\"\342\226\240\342\200\242

This

C' = (xx

where z is a new
C' =

where both

+jc2 + z\,

+x2+z)'(xx

(x{ +y
y and

variable.

Finally,

+ z)-(x{ +y +
z are

new

z)-(jr1

variables.

if C

=jc ,, then

+>>+z)-(jt1

+y + z),

11.4

a general instance of SAT

if the other one
and
only

is satisfiable if

instance

done in

polynomial

11.4.4

reduced

have

we

Thus,

one

that

of NP-Completeness

Examples

of 3SAT such
can clearly be

an instance

into

is.

The

351

Proofs

reduction

time.

\342\226\2

Clique

The clique problem was

definedin

Section

when we

11.4.1,

discussed

the

E)

integer

vertex-cover

problem.

Given

Problem

The

an

undirected

G contains

whether

ky determine

graph

a cliqueof

size

G =(V,
> k.

and

an

11.7

\342\226\241
Theorem

The clique problemis NP'-complete.

Proof:

The

and

check

vertices

problem.
Consider

Let
the

clique problem belongs to NP since we can guess a subset

that it is a clique in polynomial
time.
We reduce SAT to the

E be
clause

an

E,- = (x+y+z

We associate
also
appear in other
they
purposes).

arbitrary

\"column\"

Boolean

+ w)

(we
of

four

expression
use four
vertices

in

CNF,

variables
with

the

E =EXE2
only

variables

of >k
clique

''

Em.

illustration

for

in

even
\302\243,

if

is, the graph G will have a vertex for each

The
each
variable.
of
question is how to connect these vertices such that G
appearance
a clique of size >k if and only if E is satisfiable. Notice that we are free to
contains
which
choose the value of k because we want to reduce SAT to the clique problem,
of
a solution
of the clique problem. A solution
means that we want to solve SAT using
that
the clique problem should
work
for every value of k. This is an important flexibility
of
k to be equal to the number
is used often in NP-completeness
We will choose
proofs.
clauses m.
The edges of G are as follows. Vertices from the same column (i.e., vertices
Vertices
from different
of the same clause) are not connected.
associatedwith variables
variable
the
same
unless they correspond to
columns are almost
connected
always
two vertices
That is, the only time we do not connect
form.
appearing in complementary
from different
clauses
is when one corresponds to a variable
x and the other to x. An
+ z)-(y +z), is
E = (x+y+~z)-(x+y
which
to the expression
example,
corresponds
in
in
11.3.
G
can
be
constructed
time.
presented
Fig.
clearly
polynomial
claim that G has a cliqueof size > m if and only if E is satisfiable. In fact,
We
now
the construction
m independent
guarantees that the maximal clique size does not exceed
that E is satisfiable.
of E. Assume
such that each
Then, there existsa truth assignment
clause contains at least one variable whose value is 1. We will choose the vertex
for the clique. (If more than
in a clause is set
one variable
corresponding to this variable
The
result
is
since
a
the
indeed
time two
we
one
choose
to
1,
arbitrarily.)
clique,
only
are
are not connected is when
the
columns
vertices from different
they
complement of
clauses.

That

NP-Completeness

352

Figure11.3 An
each other, which

assumethat
from

of the

cannot

happen

clique reduction
u+)>+z)-(i+y+z)-o>+z).

a clique

G contains

column

each

course

of

example

(since

two

in

of size >m.

vertices

from

a consistent
The

the expression

truth assignment. Conversely,

contain exactly one vertex

must

clique

the same

for

column are never

connected).

We

of 1. If any variables are not

in this
assign the corresponding variablesa value
assigned
can
be assigned
in the clique are
manner, they
arbitrarily. Since all the vertices
connectedto one another, and we made sure that x and x are never connected,this truth
is consistent.
\342\226\
assignment

11.4.5

3-Coloring

Let G =(V, E)
to the vertices

be an

have the

color.

same

The
whether

such

coloring of G is an
each vertex is assigned one color and no

undirected

that

graph.

Given an
Problem
G can be coloredwith

\342\226\241
Theorem

A valid

undirected
three

graph

G=(V,

assignment

two

adjacent

of colors

vertices

determine
\302\243),

colors.

11.8

3-coloring is N?-complete.

Proof: The

3-coloring
problem
belongs to NP since we can guess a 3-coloring and
valid coloring easily in polynomial
We reduce 3SAT to the 3time.
This is a more complicatedproof
for
two reasons.
First, the two
coloring
problem.
deal with
different
versus
problems
(Boolean
objects
graphs). Second,we
expressions
cannot
one
with
another
vertex,
just replace
object (e.g.,
(e.g., clause); we have to
edge)
deal with the whole structure. The idea is to use building
and then to tie them
blocks
instance
of 3SAT. We have to construct
G such
a graph
together. Let E be an arbitrary
if and only if G can be 3-colored. First,
that
E is satisfiable
we build the main triangle M.

check

that

it

is a

11.4 Examplesof
a triangle,

Since M is

F (for

true),

(for

used only
colors

the

with

another

build

vertex

M.

in

assignment

We

We

do

with the

that

six new

introduce

11.5.

(since

vertices

labeled 7, and one

to 7 and jc, yy or z
new vertices in the

vertex
the

that

condition

the \"colors\" T
These colors are

connect

and

for each

outer

xy

vertices

yy or

J, and

x is

if

them

is

the

same

one variable in each clause has

that the clause is (x+y+z).

Assume

to the

z.) Let's

one vertex

call the

(they are labeled

by

as shown

vertices,

existing

is only

there

A, where

one
triangles, all sharing
colored with the color 7, then x
to A), and vice versa. This
is

k +1

at least

construct.

following

x,
have

connected

both

consistent,so that

(The labels are

is that,

idea

they are

with the meaning of J.

now
have to impose the

consistent

value 1.

Fig. 11.4). The

F

with

M with

in

whose vertices are labeled

are k variables, we will

Mx
triangle
So, if there

be colored

must

label

We

the bottom

(see
triangle
Fig. 11.4).
these
they are not part of the graph. We will later associate
of truth values to the variables
of E. For each variable x, we

and

proof;

vertex A (see

common

We

false),

the

for

colors.

at least three

it requires

353

Proofs

NP-Completeness

three

in

new

O in the

whole

the

vertices

figure),

and

in

Fig

graph

connected
the

three

vertices (labeledby / in the figure). We claim that

this construct
guarantees that, if no more than 3 colors are used, then at least one of x> y>
of them can be colored A, since
or z must be colored7. None
they are all connected to A
connected to them must
If
all
are
colored
then
the
three
new
vertices
F,
(see
Fig. 11.4).
three
colors!
The
then
the inner triangle cannot be colored with
be colored A, but
is given
in Fig.
to the expression (x+y+~z)-(x+y+z)
complete graph
corresponding
triangle

the inner

11.6.
is
We have to prove two sides:
the proof.
(1) if \302\243
complete
can be coloredwith three colors; and (2) if G can be colored with three
then E is satisfiable. If E is satisfiablethen there is a satisfiable truth
colors,
assignment.
the variables
We color the vertices associatedwith
(7
according to this truth assignment
M is colored with 7, F, and A as indicated. Eachclause must
if x = 1, and F otherwise).
outer
at least one variable whosevalue
have
is 1. Hence, we can color the corresponding
vertex with F, the rest of the outer vertices with A, and the inner triangle
accordingly.
three
if G can
be colored with
three
colors.
Thus, G can be colored with
Conversely,

We can

satisfiable,

Figure

now

then G

11.4

The

first

part

of the

construction

in

the

reduction

of 3SAT to

3-coloring.

354 NP-Completeness

Figure11.5 The

to the

corresponding

subgraphs

clauses

in

the

of 3SAT to

reduction

3-

coloring.
y

Figure
colors,
three

we name
colors).

to a consistent
variable in
polynomial

11.6 The

the

colors

according

corresponding

to the

to (;t

+>>

+F)

coloring of M (which

+z).
\342\200\242(*+>>

must

be colored

with

11.4, the colors of the variables correspond

triangles
truth
The
that at least one
11.6 guarantees
in Fig.
construct
assignment.
in
each
is colored with
clause
T.
be
constructed
G
can
clearly
Finally,
D
time, which completes the proof.
Because

of

graph

the

in Fig.

Examples of

11.4

NP-CompletenessProofs

355

11.4.6 General Observations

complete.
not

to

look very
which

It is

briefly

first

The

always).

or similar

that
a problem
some general methods for proving
Q is NP\342\200\224
\342\200\224
that
to
NP
is
Q belongs
showing
usually
easy (but
that seems related
Then, we have to select a known
NP-complete
problem
to define
this \"similarity\"
since sometimes
the problems
Q. It is hard
goal,

here

discuss

We

different

to reduce

condition

(e.g., the clique problem

is sometimes a difficult
task,

a good idea to

try

several

reductions

and

SAT).

several

with

the right

Finding

can be

which

learned

until a

problems

problem
by

only

from

experience.

successful one is

found.

We

stress

that the reduction is

The

done

from

an

instance

arbitrary

of

the

known

NP-

the
complete
proofs is to perform
reduction
to
remember
the
order
to
that
the
is
ensure
NPway
right
a black-box
complete
problem can be solved
by
algorithm for Q. This is a little
The natural thing to do when
counterintuitive.
a problem Q is to try solve
it. Here,
given
however, we try to show that we can solve another
(the NP-complete
problem
problem)
the solution
of Q. We are not trying to solve Q\\
using
are several degrees of freedom that
There
can
be used in the reduction.
For
in
if
a
can
be
convenient
includes
then
its
value
set
Q
parameter,
any
way. (In
example,
in the problem that is reduced
to Q, which
cannot
be fixed!)
contrast with the parameter
therefore, we can use it in any
Again, Q is just a tool to solve the NP-complete
problem;
to special
its
cases in other ways, besides fixing
way we wish. Q can be restricted
a
of
For
we
want
to
use
certain
may
only
types
input
(e.g.,
regular
parameter.
example,
we have is the fact that
flexibility
graphs, biconnected graphs) for Q. Another
important
the efficiency of the reduction
is unimportant,
as long as the reduction can be done in
time.
We can ignore not only constants
and, for example, double the size of
polynomial
the problem, but we can also square the size of the problem! We can introduce
in a graph by a new large
vertex
polynomially
many new variables, we can replaceeach
the
on.
There
to
be
efficient
bounds of a polynomial),
and
so
is
no
need
(within
graph,

problem to Q.
backward.
One

most

error

common

since
the reduction
is not meant to be converted into
found to be equal to NP, if ever).
There
are
some
common techniques used in

(again, Garey

and

Johnson

[1979]

provides

problem is a

many

in such

an

the

(at least

algorithm

examples).

until

P is

of the reductions

construction

special caseof

not

The simplest one is

is, then the proof is

immediate, since solving
Q implies
solving the NP-complete problem. For example,
the set-cover problem. The input
consider
to the problem
is a collection of subsets
of
a
k.
set
an
The
to
there
and
is
determine whether
Uy
Si,S2,...,5\342\200\236
integer
problem
exists a subset W \302\243
contains
at least one element from
Uy with at most k elements, which
each
the vertex-cover
set 5,. We can see that
problem is a specialcaseof the set-cover
to
the
of
set
in
which
U
vertices
V, and each set 5, correspondsto
problem
corresponds
an edge
incident to that edge.
if we can solve the
and contains the two vertices
Thus,
we
can
then
solve
the
vertex-cover
for
set-cover
sets,
arbitrary
problem.
problem
this approach.
It is not true, in
be very careful, however, when
We
must
using
a
to
makes
that
more difficult.
more
that
problem
problem
requirements
adding
general,
showing

that

an NP-complete

Q.

If it

NP-Completeness

356

Considerthe
cover

vertex-cover

independent

and

problem,

be solved

can

it

extra requirements
found

vertex-cover

the

an

independent

In

constraint
that
the vertex
we are looking for a small
set at the same time. (An

7.115).

(Exercise

sets to such

this problem is

however,

fact,

time

polynomial

candidate

the

limit

in

words,

not adjacent to one another.) This

problem
or the independent-set problem, because

are

that

other

In

about more requirements.

to worry

have

of vertices

than either

more difficult

seems
we

a set

set is

vertices.

a vertex cover and

that forms

of vertices

set

that we add

Suppose

problem.

two adjacent

include

not

must

that

extent

an

It

that the

out

turns

easier

an

can be

the minimum

easily.

Another

object

in

done

is mapped

problem

in a local

involves local

easy technique

relatively

one

manner,

an

into

of

independently

reductions. In

this

an

case,

problem. The mapping

object
the other objects. The NP-completeness
of the other

is
proof

We replaced each edge in one graph

dominating set problem followedthat pattern.
were
sufficient
to reduce the
in the other graph. These local replacements
by a triangle
in
the
the
in this
to
define
The
is
best
difficulty
technique
objects
way.
problem.
blocks
as we did, for example,
The
most complicated
technique is to use building
the
The
blocks
of
on
in the
usually
proof
3-coloring
problem.
NP-completeness
depend
each one separately is impossible. We
have
to consider
all
one another, and designing
in order to coordinate the design
of the different blocks.
the objectives of the problems
of the

More NP-Complete Problems

11.4.7
The

some more NP-completeproblems

that
are useful as a basis
in
A
in Garey
the
ones
the
list
is given
exercises).
very large
(e.g.,
than
more
the
for
the
reduction
is
sometimes
Finding
right
problem

list contains

following

reductions

other

for
and

Johnson

half

the work.

[ 1979J.

Hamiltonian

cycle: A Hamiltonian
once. The problem
Hamiltonian circuit. The problem
is
(Reduction from vertex cover.)
graphs.
each

vertex exactly

Traveling salesman:Let
salesman

tour

is

Hamiltonian

path.

The

G contains

integer
reduction from clique.)

3-dimensionalmatching:
of

triples

(x, y,

z) such

that

problem

problem

X,ye

is

graph

a simple

open

is to determine whether
is NP-complete for both

directed

that

path

a given

graph
and

undirected

cover.)
set in an

undirected

graph

The problem is to

set

an independent
Let

xe

in a

path

graphs. (Reduction from vertex

set: An independent
Independent
no two of which
are connected.
whether

NP-complete

A travelinga weighted
graph.
complete
G and a
to
The
is
determine, given
cycle.
problem
of its
exists a traveling-salesman tour such that the total length
reduction
from Hamiltonian cycle.)

directed

ky

contains

contains

E) be

G =(V,

Hamiltonian path: A Hamiltonian

contains
each vertex exactly once. The

vertices

is a simple cycle that

whether a given
graph
for both undirected and

graph

determine

a Hamiltonian

number W, whether
there
<
is
W.
edges
(Straightforward

contains a

in a

cycle
is to

A\\ K, and
Yy

and

>

k vertices.

E) is

given

a set of
and

an

(Straightforward

sets of size k. Let Mbea

disjoint
Z. The problem is to determine whether

Z be
z e

with

G =(V,

determine,

set

11.5

a subset of

there exists
two-dimensional

with

problem

357

Problems

NP-Complete

exactly once. The

each element
is the
regular

contains

that

matching

(Reduction from

For Dealing

Techniques

corresponding

problem.

matching

bipartite

3SAT.)

The input is a set X such that

whether
is to determine

Partition:

s(x). The problem

element

each

xe

an associated

has

size

it is possible
to partition the set into two
total size. (Reduction from 3-dimensional
Matching.)
(Notice that this problem, as well as the next problem, can be solved
efficiently
by
However, since the
algorithm Knapsack (Section 5.10) if the sizes are all small integers.
of bits required to represent that
of the input is the number
size
such algorithms,
input,
are
in
the
which are calledpseudopolynomial
size
of the input.)
algorithms,
exponential
is
a
setX
such
that
each
element
xe
X
has
The
an
size
associated
Knapsack:
input
there
is a subset B^X whose
s(x) and value v(x). The problem is to determine whether
total value is > v. (Reduction
from partition.)
total
size is < s and
whose
is a sequence
of numbers a\\,a2, ...,a\342\200\236,
Bin packing: The input
and
two other
the set can be partitioned
numbers b and k. The problem is to determine whether
into k
subsets such that the sum of numbers in each subset is < b. (Reduction from
partition.)

subsets

with

11.5

the same

exactly

for

Techniques

with NP-Complete

Dealing

Problems
notion

The

is a basis for
no polynomial algorithm

of NP-completeness
for which

problems

an

theory that allows us to identify

to exist. But proving
that a given

elegant

is likely

problem is NP-complete does not make the problem go away! We still need to solve it.
different
from the
The
for solving NP-complete problems are sometimes
techniques
NPseen.
cannot
solve
an
we
that
have
.We
(most probably)
techniques
previously
precisely

problem

complete

have to compromise.
robustness,

in

different

situations,

that

guarantee a

on

bound

in

degree

compromises

completenessof

the

something. The same algorithm

different

compromises.

lead to

Of particular

the

algorithm. So, we
concern the
optimality,
solution.
There are other

a polynomial-time

sacrifice
not

may

algorithm.

approximation

which

resulting

An algorithm

common

most

The

of

all

with

completely

efficiency, or

guaranteed

alternatives as well,

and

the

(or

optimal
are

interest

of imprecision.

We

may

be used

result is called

precise)

approximation
algorithms
see three examples
will

an

that can

of such

later.

algorithms

In Section

6.II,

The most famous

known

to be

primality-testing

such
in

P,

but

we discussed

that

ones for primality

testing,

believed to be NP-complete

is not

algorithms,

probabilisticalgorithms

are the

algorithms

approximation

can be used as
algorithms.

We

either.

will not

describe

It is
they requires knowledge of number
theory.
cannot
be
a
solved
problems
by
polynomial-time
mistakes
with low probability
for all inputs.
Therefore,

because

commonly believed that NP-complete

probabilistic
algorithm that make
are more
such algorithms
likely to be effectivefor problems
Such problemsare
to be NP-complete.
but are not believed
algorithms

may make mistakes.

a problem that is not

part

of

other

strategies

that

are not

not

\342\200\224
for

known

common.
example,

to be in P
Probabilistic
as part of

NP-Completeness

358

Another compromise
can

We

times.

The problem
exclude

try

with
for

inputs

from

vertices)

well on

the

example, the

the

Another

definition.

We describeseveral
We start

section.

with

These

branch-and-bound.

an

of
two

approximation

We

describe
involves

is an

example

the

for
in

abstract a

which

techniques
an

only

the average
be useful if the

the right

distribution,

work

that

algorithms

completeness

involving

some

only

polynomial
from

problem

the

whose running

For
inputs.
for bipartite graphs
a real-life situation we
special

time

is included

input

times

in the abstract

are exponential,

but

they

may be sufficient.

techniques and illustrate

and useful techniques
general

algorithm or

Backtracking

which

on

isolated

very complicated.
of the algorithms; namely,

is usually

which

can

inputs

in designing

these

several examples of approximation

11.5.1

random

to

difficult

is

lower

may

inputs

difficulty

efficiently
can be solved

small inputs,

(e.g.,

with

major

is algorithms

example

it

a graph

Finding

them,

problem

example,

assumption.

also compromise
to work
algorithm

well for

reasonably

For

their

when we
Therefore,
that any extra condition

sure

make

follows

very difficult.

vertex-cover

7.110).

(Exercise

work

designed

time

polynomial

Such trivial
average.
for certain types of

can

we

we can allow

the

is analyzing

average

Finally,

should

in

of inputs

is usually

however,

in

is defining
average.
approach
which the particular problem is trivial

participating

distribution

problems

NP-complete

worst-case running
on the average.

for polynomial

requirement

this

Algorithms

significantly.

actual

the

involves

solve

to

them

with

examples

are similar. They can be usedas a basisfor

for small inputs.
We
then
algorithm

optimal

in

called backtracking

this
and

either

give

algorithms.

and

Branch-and-Bound

Consider
the 3-coloring problem,
techniques
through an example.
under
certain
to
n
vertices
of a graph. This
colors,
constraints,
assigning
of a problem that requires
finding optimal values (colors in this case) for n

these

for each parameter

parameters. In the 3-coloring
example, there are three possiblevalues
is 3\",
the number of potential
solutions
Therefore,
corresponding to the three colors.
which is the number
Of
of all possible ways
of coloring
three
colors.
n vertices with
of possible valid solutions will
course, unless there are no edges in the graph, the number
be quite
a bit smaller than
on the possible
the edges impose constraints
3\", because
To
all
we
can start by
the
of
vertices,
colorings.
explore
possible
ways
coloring
an
vertices
and continue coloring the other
assigning
arbitrary color to one of the vertices
\342\200\224
while maintaining the constraints
that adjacent vertices
namely,
imposed
by the edges
must
be colored
colors
with different colors. When
we color a vertex, we try all possible
that are consistent with
the previously
colored vertices. This process can be performed
and branch-andwhich
is the essence of the backtracking
by a tree-traversal algorithm,
bound techniques. To avoid confusion
the vertices of the graph and the tree, we
between

the vertices of the tree nodes.

branch
The root of the tree corresponds
state
of the problem, and each
to the initial
to
a
decision
colors
three
one
Denote
the
R(ed),
corresponds
by
concerning
parameter.
we can pick
and G(reen). Initially,
color
w
v
and
and
vertices
two
B(lue),
adjacent
any
them, say with B and G. Since they will be colored differently in any valid coloring, it is
will

call

11.5

not

which

important

is why

we can

we choose

colors
with

start

Techniques

For Dealing

with

(we can always

coloring

the final

permute

two vertices instead

of one.

The

359

Problems

NP-Complete

coloring),

which

of these

coloring

two

vertices correspondsto the initial state of the problem,

which is associated with
the root.
The tree is constructed as it is being traversed.
At each node t of the tree, we select the
next
vertex u of the graph
or three
to color, and add one, two,
children to t according
to
the number of colorsthat can be used to color u. For example,if our first choice (after v
has already been colored G), then
and w) is w, and if u is adjacent to w (which
there
are
of coloring
two possible ways
w, B or R,
root.
We then pick one of these
children,

colored, there is less flexibility

of children is likely

in coloring

add

we

and

and

two corresponding

continue

the rest

of the

this process.
vertices;

children to
a vertex

After

the

is

the number

therefore,

as we go deeperin the tree.

If we manage to color all the vertices
we are done.
of the graph, then
More likely,
however, we will reach a vertex that cannot be colored (sinceit has three adjacent
\342\200\224
we go up
vertices already colored with the three colors). At that point, we backtrack
other children.
An example of a graph
and
the corresponding
3the tree and explore
in
tree
in
11.7.
Notice
this
once
the
backtrack
is
colors
of
that,
case,
Fig.
coloring
given
to color
the rest of the graph
there
is only one way
is
vertices 1 and 2 are fixed,
(which
in
the
the rightmost
found through
tree).
path
of this tree-traversal
based on induction.
We can think
algorithm as an algorithm
of whose
We have to strengthen
the hypothesis
some
slightly to include coloring graphs
will have to
vertices have already been colored.In other words, the induction hypothesis
a partial 3-coloring:
deal not with coloring graphs from scratch, but with completing
to be

smaller

Induction hypothesis:We
that

has

know

how

to complete

vertices that are not already

cannot be completed.

< k

3-coloring

the

of a

3-coloring

colored, or to determine

graph

that

the

1B,2G

Figure 11.7
backtrack

tree.

An

example

of backtracking

for 3-coloring.

(a) The

graph

(b) The

NP-Completeness

360

Given a

possible colors

k-\\

uncolored

(one

(Gf var

in Fig.

is given

all
its

with

now have

(which

problems

find

and

used for

the vertex

color

we

remaining

The algorithm

by induction.

3-coloring

the

one of them

have already been

all colors

If

and solve

a time)

at

vertices)

Algorithm

we pick

colored,

yet
it.

3-coloring cannot be completed. Otherwise,

colors

possible

not

to

be assigned

can

that

then the

neighbors,
the

that are

k vertices

with

graph

11.8.

U) ;

and U (a set of vertices

that have
Input: G =(V, E) (an undirected
graph),
with
their colors). { U is initially empty )
already been coloredtogether
of one of three colors to each vertex
of G.
Output: An assignment
begin

ifU =

then

\"coloring is

print

completed\";halt

else

a vertex

pick

3 do

if no neighbor

add

in U

not

forC := 1 to

to U

ofv is colored with

C;

C then

color

with color

3-coloring(G,U)

end
Figure11.8Algorithm
It is

problem

that

not

common

is quite

to

hard

come

in a

results
in

\"good\"

order, we

does

not

specify

have

a degree

point

shortly.

up

tree

an

how

find

the solution

to pick

is a

Since

minimum
color
similar

the graph rather

to the one for

than

just

a yes

3-coloring,

but

3-coloring

for problems
Consider

the minimum

involving

be

to this
the

finding

coloring

colors required

We

3-coloring.

we

next,

return

the general

number of

of branches may

number

will

can build

quite

large.

so far

described

we

be chosen

can

vertex

function.

or no answer for
the

The algorithm

any

of backtracking

variation

the

for

traversal

design heuristics. We

use to

(or maximum) of some objective

\342\200\224
we are now interested in finding
problem

of

of nodes (Exercise11.34).This
the tree in
Our hope is that, by traversing
number

early enough.

can

we

and an order

vertex.

next

the

of freedom that

Branch-and-bound

exponential

algorithms.

backtracking
will

a graph

with

with

3-coloring.

to

a tree
Each

its
can be colored either
by one of the colors already used (unless one
The 3-coloring
or by a new
color.
neighbors already uses that color),
algorithm is thus
modified
in two ways: (1) the constant
number of colors
3 is replaced by the maximal
be
used so far, and (2) the algorithm does not terminate
when
V = U, since there may
better ways to color the graph.
The problem
is that this algorithm
a leaf is reached (i.e.,
backtracks
only when
is
V = U), since a new color can always
be assigned
to the vertex. Thus, the algorithm
almost guaranteed to have poor performance
We can
the graph is very
(unless
dense).

new vertex

improve

the

of

performance

of this

algorithm

by

the

following

observation,

which is

the

11.5

For Dealing

Techniques

basis of the branch-and-bound

leaf and find a valid coloring

method.

Suppose
k colors.

with

with

that we

361

Problems

NP-Complete

traverse the

all the

tree

Suppose further that,

and reach a vertex
path

after

to a

way

backtracking

that
color
steps up the tree, we traverse another
requires
know a better solution.
number k +1. At this point, we can backtrack, since we already
At each node, we compute a lower
bound
on
Thus, k serves as a bound for backtracking.
the tree. If that lower bound
that
can be found farther
down
the best solution
is greater
we backtrack.
One key to making a branch-and-bound algorithm
than a known
solution,
if we want to maximize the
is computing
efficient
bounds,
good lower bounds (or upper
Another
is
a
traversal
order
so that good solutions
function).
good
objective
key
finding
case we can backtrack earlier.
are found fast, in which
of integer
We
illustrate
this idea through the
linear programming
problem
in Section
is also mentioned
The problem is similar to linear programming,
(which
10.3).
with
that
of the variables
but
the extra constraints
the values
are integers.
Let
be vectors
be the vector of variables', ax,a2, \342\200\242\342\200\242\342\200\242,ak
of real numbers,
! = (*!,jt2, ..,*\342\200\236)
and c i, c2,..., ck be real numbers.
The problem is to
each of size n\\ and b\\,b2,..;bk
maximize
the value of the linear objective function

several

the integrality

under

~d\\

* * *

+C2*2+

Z=CXXX

(11.1)

+C\342\200\236Xn

of J

constraints

and the

following

constraints

-X <bx

a2 'X

<b2

(11.2)

ak-x<bk.
(All

tf/S

b(s are

and

constants.)

is NP-hard.

It

can be

easily posed as

is in

integer

to NP

is

complicated.

quite

The following
(The problem here

is

an

is to

such

that

function
Z

= 1 if v, belongs
Xj
is

=X\\

JC2

implies

* * *

that we

to

for

all

n
the

+ *\342\200\236,
want

to

per vertex

0<^; < 1

programming

maximal

the
are

formulation of the clique problem.

than
to decide
rather
whether a certain
clique,
variables X\\,x2, ...,*\342\200\236,
to the vertices,
corresponding
maximum
and
=0
The objective
otherwise.
jc,
clique,

linear

integer
find

sized clique exists.) There

which

problems

(we show one example below). Therefore,

fact NP-complete, but the proof that it belongs

problems

programming

integer

programming

NP-complete

Many

!</<\302\253,

select

as many vertices

as we can.

There

is one

constraint

NP-Completeness

362

and one
+

Xj

Xj

for

pair of

each

constraints guarantee

therefore,the

linear

Integer

(v,, v;) 4

v, such that

0 or

to either

that are

not

1.

of

set

second

The

cannot

adjacent

a clique.
be solved with

E.
both be

selected;

are selected form

that

vertices

and

variables

the

vertices

two

that

vertices v,

restrict

of constraints

set

first

The

<

vertices

pair of nonadjacent

each

for

constraint

can

programming

branch-and-bound

the

using

by

is the same problem without

the
restriction
to
(which
corresponding linear program
of
the
linear
the
bounds.
The
solution
consist
of
to
program
compute
may
only
integers)
the solution will include
in which case we are done. More
however,
likely,
integers,
that
the solution of the linear
values. For example, assume
some noninteger
program
is
with
the
associated
(0.1, 1,...,0.5)and z = 7.8. Since the linear
clique problem
with
less restrictions
the objective function
than the integer linear
maximizes
program
it finds is an upper
bound
on the maximum possible for the
program, the maximum
linear program. Therefore, we cannot hope for a clique of size greater than
7.
integer

of

kind

This

be

can

information

helpful

farther

As

tree.

the

down

in

regular

some choicesas we go down the tree, and a node lower in the

backtracking,
to a subproblem of the original
For example, the subproblem
tree corresponds
problem.
to selecting v and w to the clique, and eliminating
u and x, in which
case
may correspond
v and w and excludes u and x. If at
we are trying
to find the maximal clique that
includes
that point the solution
of the linear program gives us a bound
that is less than a size of an
then
we
can
backtrack.
This
is
the
essence
of the branch-andalready-known
clique,
bound
method.
We are trying
to find upper bounds (or lower bounds,
if the objective
function is supposed to be minimized)
that will allow us to backtrack as early
as possible.
We can also use the result of the linear program to help us choose the branching.
For example,
since a2 = 1 in the noninteger
solution, we may guess that jc2 = 1 is the
as well.
This may not be a good guess,but it is an example of the kind of
integer solution
heuristics
that we are looking
for.
We try to increase the
of finding the
probability
we make

optimal

know that being \"right\"

all
the time is probably
(We
quickly.
the problem is NP-complete.) We can set x2 = 1, update
the constraints
of all vertices not adjacent
the
to v2 to 0), and
solve
resulting linear

solution

since

impossible,

(e.g., set

the

values

program. If at some
where a is smaller than
Thus,
us to

allows

the

when

we

are

the

other

the

next

the

linear

done

the modified linear

and

with

program has a

clique known so far,

serves two purposes:

maximal

program

backtrack,

prune
of the

whole

point

the

we
It

gives

bounds

\"most

likely

subproblem

to explore.

This

heuristic

depends

done by

that consists
breadth-first

of

the

search,

best

solution

depth-first

of z=tf,

on

will

when

thus

the efficiency
and

the particular

the

and

hope that,
be able to

We

\342\200\224
and
of pruning
The amount
subproblems
substantially.
\342\200\224
the
divide
to
on
the
heuristic
algorithm
problems
depends

approximation

can be

upper

choices to make next.

to succeed\" subproblem, we

hints at which

it also

Extensive researchhas been done in this area.

lead
Branch-and-bound algorithms
to the optimal solution
are explored or pruned.
If this takes too long, we can terminate
an

value

maximal

can backtrack.

to choose

application.

all subproblems

and obtain
of the tree

algorithm

found so far. The traversal

search, or a combination.

An

extreme

11.5

example of

example,

the

in

(the

idea

This

is a

solution

first

we can

algorithm,

lose less flexibility

with

color the

by

outcome

the

by a certain heuristic)
of the algorithm.

order of

in reverse

vertices

the color

fixing

363

Problems

NP-Complete

(chosen

path

leaf) as

at a

(usually

coloring

that we

being

the

is talcing

early

terminating

a feasible

to

leads

For Dealing

Techniques

of a small-degree

that

For

degree

vertex).

simple greedyalgorithm.

11.5.2

with Guaranteed

Algorithms

Approximation

Performance
In

this

we discuss approximation algorithms

bin
and the Euclidean traveling
packing,

section,

vertex cover,

for

three

NP-complete

salesman

approximation algorithms have guaranteed performance. That

solution
they produce is not too far from the optimal solution.

is,

problem.
we can

problems:
All these

prove

that

the

Cover

Vertex

for finding
the minimum vertex cover of
algorithm
find
The
is
to
a
cover
that contains no more than
graph.
algorithm
guaranteed
twice
cover. Let G =(V, E) be a graph
the number of vertices containedin a minimum
in G. Since M is a matching,
and
let M be a maximal
its edges
have no vertex
matching
in common,
and since M is maximal, all other edges have
one vertex
in common with at
in M.
least one of the edges
We

with

start

a simple approximation

a given

11.9

D Theorem
set

The

of all

vertex cover
size

vertex

Therefore,

set of

is

at least

to

incident

more

the edges

than twice

the

of a

number

maximal

matching

of vertices

of a

M is

minimum-

a vertex
cover, because M is
belong to M forms
in particular,
the edges of M.
cover all the edges \342\200\224
cover
more
than one edge of M.
a matching, a vertex of M cannot
\342\226\
half of the vertices
of M must belong to every vertex cover.

vertex

Every
since

But,

no

cover.

Proof: The
maximal.

vertices
with

vertices

that

cover must

maximal matching

edges are covered.

would
like to find a
we
Since the vertex cover includes all the vertices the matching,
small maximal matching. Unfortunately,
the problem
of finding the minimum maximal
a maximal matching with
number
of edges) is also NP-complete
smallest
(i.e.,
matching
another
and
Johnson
[1979], problem [GT10]). Exercise 11.35 discusses
(Garey
with
for
the
vertex-cover
approximation
algorithm
guaranteed
performance
problem.
We

can

find a

by

simply

collecting

edges

until

all

in

One-Dimensional

Bin Packing

is concerned with packing

different-sized
problem
objects into fixedas possible.
For example, we may want
few of the bins
the
to move
as few cars (or the
same
car as few times) as possibleby
contents of a house using
is a 3-dimensional
the cars as densely as possible.Moving
problem, but we will
packing
version.
We will also assume for simplicity that all
on the one-dimensional
concentrate
the bins have size 1.
The

bin

packing

sizedbins using

as

364 NP-Completeness

The Problem
0 and

between

The one-dimensional
in
problems
blocks need

bin

put

the
the

for this

bin

has

that

used bins. This algorithm

worst case, as is shown

The

than

twice

fact

that

sizes (in

be placedin

the sum
the

of

of bins

case

all items

which

a new

Bin

bin,

and

the

packing

for each

then,

bin if there is no
fit is

First

algorithm.

bound

20 PT

room in

not \"too

less than

half

any

bad\"

/, to
of
in

otherwise, the items in the

number
of bins used is no more

Therefore, the
items (rounded up). The theorem
best solution cannot be less than

the

the

full;

of all
in

is

OPT

where

bins,

first bin.

the

sizes

the

number

out that the

It turns

leave two bins

fit cannot

First

bin could

in the first

at most

requires
of bins.

number

Proof:
second

problem is to put x,
room for it, or to start
is called
the first fit
in the next theorem.

algorithm

fit

first

memory, and

11.10

\342\226\241
Theorem

minimum

memory-management

of available memory.

chunks

large

in

blocks of

different-sized

many

problem (Exercise11.8).

heuristic

Xj in the first

arises, for example,

problem

packing

is at

subset

each

which there are requestsfor

to be allocatedfrom several

is an NP-complete
One

in

of real numbers
each
subsets as possible
most 1.

into as few

numbers

the

a set

be

Xy,x2,...,xn

of numbers

the sum

that

such

Let

1; partition

from

follows
the

are perfectly packed).

11.10 is

by Theorem

given

the

of all the

sum

\342\226\

The

conservative.

quite

constant

of

reduced to 1.7, by a much more complicated analysis. The 1.7

constant
is tight, since there exist cases in which
first fit requires 1.7 times the optimal.
First fit can be improved with
the following
case
simple modification. The worst
occurs
when many small numbers appear at the beginning.
Instead of placing the
numbers
in the bins in the order they appear, we sort them
first in decreasing
order, and
then
use first fit. This modified
algorithm is called decreasing first fit, and, in the worst
its solution comes within
case,
a constant
of about 1.22 from
the optimal
(we omit the
2

in

the

can be

theorem

proof).

11.11

\342\226\241
Theorem

The decreasing first

OPT is the

This constant

is

There are other

complicated.

The
natural

strategies

fit

also

tight.

methods

we

First
leading

described

at most

\342\200\224

OPT

+ 4

bins, where

of bins.

number

minimum

requires

algorithm

and

fit

decreasing

to better

are typical

approaches corresponding to

what

first

constants.

one

are

fit
In

most

of heuristics
would

both
cases,

They

algorithms.

probably

do

simple heuristics.
the analysis is

by

hand.

present
We have

Techniques For Dealingwith

11.5

seen
large

it

Therefore,

inputs.

approaches can perform

that straightforward
is very important

however,

times,

many

to

the

analyze

365

Problems

NP-Complete

for

poorly

quite

of these

performance

algorithms.

Euclidean

Salesman

Traveling

salesman problem (TSP), is

of TSP
here a variation
weights correspond to Euclideandistances:

The traveling

The Problem

Let

corresponding to the
iltonian cycle (traveling

The problem is

still

designing

somewhat

by assuming only
the direct
distance

the

tree

but

we

cities;

algorithm for
that

the

between

of

in

points

the

more

the Euclidean
(We

problem.

two

satisfy

the

the

length

of the

best TSP tour.

This

this

the

than

which

route

any

tree
claim

(here,

that

through
cost =

the cost

because a TSP tour

from a TSP tour makes
is so

in

assumption

inequality,

triangle

is shorter

points

assumption helps

can relax

the minimum-cost spanning

by computing
much easier problem (see Section7.6). We

than

many

that

plane
Ham-

a minimum-distance

that

see
the

distances
any

a set

find

with

constraint

additional

among them.

tour)

will

problem

important

with the

starts
is a

which

is no

of n

salesman

states that
other points.)

The algorithm

be
...,C\342\200\236

C{,C2,

location

NP-hard,

an approximation

distance),

an

We discuss

applications.

of
is

it a
containing all vertices; therefore,removing
any edge
minimum-cost
tree.
whosecost
is
thus
at
least
that
of
the
spanning
spanning tree,
A spanning
to a TSP tour. We need to
does not correspond directly
tree,
however,
traversal
of the
search
it. First, consider the circuit
that consists of a depth-first
modify
in the opposite direction whenever the
from
tree (starting
any city), and includes an edge
a tree-shaped
search
backtracks.
(This circuit corresponds, for example, to traversing
to
the
with
on
exhibits
both
sides
of
hall,
going
right.)
Every
every
by always
gallery,
edge will be traversed
exactly twice, so the cost of this circuit is twice the cost of the
TSP
the cost of the minimum
twice
minimum-cost spanning
which is no more than
tree,
direct routes instead of
tour. We can now convert
this circuit into a TSP tour
by taking
the same
(see Fig. 11.9). That is, instead of backtracking
using
backtracking
always
to the first new vertex. The assumption
that
the distances
are
edge, we go directly
Euclidean is important,
because
it guarantees
that the direct route
between
any two cities
is always at least as good as the nondirect
route.
The length of the resulting
TSP tour is
of the minimum TSP tour,
thus
still no more than twice the length
it
is
often less
although
a cycle

than

that.

The running

Complexity

the minimum-cost

O (n

log

(see,
\302\253)

spanning

for example,

time of this
tree

algorithm,

Preparata

and

algorithm

which,
Shamos

is dominated
in

the

[1985]).

case

time
of
by the running
of Euclidean graphs, is

NP-Completeness

366

(b)

(a)

Figure 11.9 (a) A

middle

and

point,

(b) A

tree,

spanning

TSP

the tree

from

obtained

tour

by

starting

at the

right first.

going

Improvement

The

Another

tour.
the

tree,

from

have

we

algorithm

4'sloppiest\" part
way

to

by repeating

the

Eulerian

just

the

of

described can be
is the conversion

algorithm
look at this

each edge

effectively.

An Eulerian

odd-degree

nodes

in the

graph

then

can convert

We

circuit.

from

conversion is that

twice. We

must

include

tree. There must

be

in the following
the tree traversal

improved

it builds

obtain
the

tree

The

way.
into

a TSP

an Eulerian circuit on top of

tour
shortcuts
by taking

the TSP
into

an Eulerian

only even-degree
an even number

graph

nodes. Considerall
of them (otherwise,

more
the

the

is impossible, since this sum is exactly

twice the number
of
of edges).
If we add enough
to the tree to make the degrees
edges
all nodes even, then
we get an Eulerian graph. Since the
TSP
tour will consist of the
Eulerian
of the
the length
circuit
like
to minimize
(with some shortcuts) we would

total sum

of all degrees would

be

odd,

which

edges. Let's abstract the problem.

their
are given a tree in the plane and we want
to add edges to it, minimizing
total length, such that the resulting
add at least one edge to
graph is Eulerian. We must
each vertex
of odd degree.
Let's try to add exactly one. Suppose that there are 2k
vertices of odd degree.If we add k edges, each connectingtwo odd-degree
then
vertices,
all
vertices
will have even degree. The problem
a matching problem. We
becomes
thus
a
want
to find a minimum-length
that covers all odd-degree vertices. Finding
matching
additional

We

Gabow
for general graphs
in 0(n3)
(see
perfect matching can be done
works
or
that
Lawler
to
due
is
a recent algorithm,
[1976]
[1988],
[1976]). There
Vaidya
for the special
is a
this
case of Euclidean distances in time
(Whether
0(n25
(log\302\253)4).
better algorithm
in practice
is not clear.) The final TSP tour is then obtained from the
Eulerian
tree plus the minimum(which includes the minimum-length
graph
spanning
shortcuts.
The
TSP tour obtained
length
matching) by taking
by this algorithm for the
11.11.
tree in Fig. 11.10 is given in Fig.
minimum-weight

11.5

Techniques

For Dealingwith

(b)

(a)

Figure 11.11The
tree

spanning

plus

minimum

Eulerian

circuit

(b) The tour

the matching,

and

its

obtained

TSP

corresponding

(a) The

tour,

circuit.

the Eulerian

from

11.12

\342\226\241
Theorem

The improved algorithm

times the

367

Problems

NP-Complete

length

a TSP

produces

of the minimum

tour whose

length

is at

most 1.5

TSP tour.

the shortcuts (since there may not be any in the worst case),
The circuit consists of the tree
circuit.
length of the Eulerian
of the tree is at most the length
and the matching. We have already seen that
the length
TSP
of a minimum
to prove that the length of the matching
is
tour;
hence, it is sufficient
of a minimum
TSP tour.
Let Q be a minimum TSP tour.
is
a
at most half the length
Q
all vertices.
Let D be the set of odd-degree vertices in T. We can obtain
cycle containing
of D such that the sum of their lengths
two
is no more than that of Q in
disjoint
matchings
the following
way (see Fig. 11.12). We start with an arbitrary vertex v of D and match it
is its closest neighbor clockwise in Q. We then continue matching
in
to a vertex of D that
direction. If the matched
a clockwise
vertices
are not neighbors in Qy then the distance
between them is no more than the length
of the path connecting them in Q (by the
is obtained
This
us
one
processgives
triangle property).
matching. The second matching
of both
The sum of the lengths
counterclockwise.
by repeating the same
process
in Figure
11.12. But, since M was a
matchings is at most the length of (?, as is shown
of D, its length
is at most half the length
of Q.
\342\226\
minimum-weight
matching

Proof:

and

We

Finding

minimum-cost
whether

it

achieving
type

of

problem

will

a minimum-weight
but
tree,
spanning

is possible
this

ignore

on the

concentrate

will

\342\200\224
and

to improve

This

algorithm
We abstract an easier

constant.

algorithm:

either

perfect
it results

then

design the

heuristic

takes

matching
in

a better

much

bound.

It is

than

longer

finding

an

open problem
the
constant
of 1.5, or to find
a faster
algorithm
illustrates one of the main characteristics
of this
problem
accordingly.

\342\200\224
or relax

still

some parts

of

the

original

NP-Completeness

368

Figure 11.12

11.6
The

Two matchings

is at most

sum

whose

of the

that

TSP tour.

Summary
have generated

should

chapters

previous

somedeservedoptimism

our

about

ability

should bring us closer to reality.

There
are many
This
to design good algorithms.
chapter
be solved with elegant,
efficient
cannot
algorithms.
important problems that unfortunately
We have to be able to recognize
these problems and to solve them with
a less than
When
we
lines
of attack.
solution.
a
is
to
have
two
us,
optimal
possible
problem given
We can try to use the techniques introduced
in the previous
to
solve
the
chapters
or
the
we
can
to
in
to
show
that
use
the
introduced
this
problem,
try
techniques
chapter
is
the
turns before we take
problem
right
NP-complete. To avoid making
many
wrong
we need to develop an intuition
of problems.
for the difficulty
approach,

The

discovered

was

result

similar

[1972] presented
included
this

work,

was

of NP-completeness

notion

in

Section

11.4).

their Turing

and

the

in

24

of

list

Further

and

Notes

Bibliographic

Both Cook

the

and

Union independently
Karp

were

lectures

Several

of languages

acceptance
different

requirements

is PSPACE,
turns out that

which

polynomial

space

nondeterministic

in

are

Cook

by a nondeterministic

other
algorithm

words,
can

award

[1983],

not known to

a Boolean
from

NP

are

which

due to

in part

and

be

in

in

Karp

P have

of all problems
expression is always
of
is that the definition

is asymmetric; there are

Another
important class

for rejecting an input.

that can be solved using
does not add more power when
an

[Savitch 1970].In

Karp

[1973]).

the complements

algorithm

for accepting
and
all problems

contains

nondeterminism

polynomial space

by

the Turing

published

other notable classes

Cook [1971](a

Levin

by

problems (some of

received

of problems
that
been studied. One such class is co-NP, which
contains
in
NP.
For example,
the problem of deciding
whether
false
to
co-NP. The reasonco-NPis different
belongs
[1986].

paper of

seminal

NP-complete

important

award

in

introduced

Soviet

Reading

algorithm

any problem that

also be solved in

can

It

space.

polynomial

is limited to
be

polynomial

solved

in

space

Further Reading 369

Notes and

Bibliographic

is an example of a problem
where
(Even
[1976]),
Tarjan
completeness is defined
is
to
the
it
for
defined
the
class
that
the reductions
can use
NP, except
similarly
way
There
of
is
classes
P
a
between
and
PSPACE
called
the
polynomial space.
hierarchy
to note that at present there is no proof that
polynomial-time hierarchy. It is important
PI We do not know
of any problem
from
any of the classes mentioned aboveis different
to PSPACE and does not belong
that
to P. There are, however, problemsthat are
belongs
by

a deterministic

that

The generalized

algorithm.

for PSPACE

is complete

to require exponential time

and Rabin [1974]).

known
more

HEX

game

and

and

[1972]), or even

and Stockmeyer

(Meyer

space

(Fischer

There are
The

several

are the

isomorphism can be solved

Luks [1982]), but

that are

problems

ones

notable

most

to be

known

not

graph isomorphism
times for

in polynomial

either

and

in NP-complete.

P or

Graph

testing.

primality

(see, for

cases

special

many

in

example,
and Solovay and

Rabin
[1976]
problem is still open.
for
Monte Carlo probabilistic algorithms
primality
testing.
determine
that a number is prime with
little
error
(These algorithms
very
probability, and
make
no
when
number
is
not
error
determine
that
a
a
Goldwasser
and
they
they
prime.)
a
that
tests
Killian
Las
without
[1986]
present
Vegas probabilisticalgorithm
primality,
is polynomial.
Another seemingly simple problem
time
errors, whoseexpectedrunning
which is to determine whether
that
is still open is the
a given
even-cycle
problem,
directed graph contains a (simple) even-length
Manber
(see Klee, Ladner, and
cycle
A wealth of information
and
related
can be
about
[1984]).
subjects
NP-completeness
column by Johnson
in Garey
that
found
and Johnson [1979] and in an NP-completeness
A
since
natural
to
ask
is
has been appearing in the Journal
1981.
of Algorithms
question
in NP are either NP-completeor are in P. This question was
whether
all problems
answered
that, unless P = NP, there are infinitely
by Ladner [1975] who
proved
partially
in between.
classes
many
An
that runs in polynomial
time on the average for the
Hamiltonian
algorithm
one
for
is
is
in
and
Valiant
and
described
[1979],
satisfiability
cycle problem
Angluin
for
the
and Brown [1985b]. An approximation
described in Purdom
weighted
algorithm
Strassen

vertex-cover

Pitt

with

These

[1988] for a

precise

(It is less than

performanceof

for k

these

(see,

and

estimating

given

by

Horowitz

the
Pearl

is

and

are given

by

Brelaz

\"almost

for

for k = 5.)

for

it is

However,

more to the definition

of
Franco
The
[1986]).
example,

Sahni

Heuristics for
and

graphs

average

Turner

Turner

size of

independent

likely that the

the average
best

by

(see

than

the

of n.

good
to the

known

guaranteed
given by Wigderson [1983]. Backtrack
and Baumert
and
[1965] (see also Bitner
Reingold
Knuth [1975] describes a technique
for
[1978].)

coloring

Golomb

cover. Gusfield

[1979],
all\"

finds

algorithm

the minimum-weight
of this algorithm.

Wilf [1984] proved that the

(for a fixed k) is a constant

1 million

running time of backtrack

[1984].

of

[1981]. The

are due

graph

techniques are describedin

fact,

k-co\\ohng

graph

algorithms

of the algorithms
for
bound
approximate
[1975]

In

= 3, and

strength

that

explanation

successful

proven

definition).

tree for
200

intuitive

(for fixed k)

k colors

heuristics are

backtrack

most twice

is at

Even

and

Bar-Yehuda

in

a more

present

coloringgraphs

simple

is given

whose weight

[1986]

[1988].

general

problem

vertex cover
and

the

present

[1977]

is

programs.

A general

discussion on heuristics is

NP-Completeness

370

The
an

The

traveling

in

problem

salesman

terms

problem is given
1.5 for the Euclidean

problem

An

and

Kernighan [1973]. The algorithm

which is the best bound currently

problem,

to Christofides [1976]. A book edited

Shmoys [1985] contains 12 articles covering
heuristics
and their analysis, branch-and-bound

and

including

very close to
[1982].

Karp

algorithm

approximation

in Section

given

studied

most

the

is due

algorithm,
Kan,

is probably

fit

that comes

heuristic

problem, see Karmarkar

solutions.

of proposed
by Lin and

first

decreasing

Johnson

of the bin-packing

solution

optimal

the performance
of first fit and
et al. [1974]. For another

on

bounds

11.5.2 are provedin

NP-complete
for the general
bound

the

achieving

of

known for a polynomial

Lenstra,

by Lawler,
most

of this

aspects

algorithms, special

Rinnooy

problem,

cases, and

applications.

Exercises

Drill

You can complete these exercisesusing

exercises from this chapter.

the NP-complete

only

problems discussed

the

in

text or

in

given

in

other

11.1

Prove

that

all problems

in

P are

polynomially

according

equivalent

to the

definition

Section 11.2.

11.2

Prove

that

not allow
11.3

11.4

of

the definition

one algorithm

Write

the

3SAT

expression

different

language

does

languages.

that

Draw

the

Section

Draw

the

graph
Section

11.4.5)

the knapsack

the reduction

reduction of SAT

to

3SAT

(given

in

from

obtained
for

of SAT to

the reduction

the expression

\342\200\242

(x +y

+ U+V)'(X+y).
the

clique

problem (given

in

+ z).

+ v+z)-(jc+y

that is

{x +y + z) \342\200\242
(x + v +F)

11.7 Prove that

from the

+ W+U+V)'(X+y+'z+W

from
graph that is obtained
11.4.4) for the expression

U+y + z)-U+> + z)-(jc

(given in

is obtained

the expression

(x+y+^+W+U+V)'(x+y+Z

11.6

two

Consider the following algorithm to determine

whether
a graph has a clique of size k. First,
we generate all subsets
of the vertices containing
k vertices.
There are 0(nk)
exactly
subsets
Then, we check whether
altogether.
any of the subgraphs induced
by these subsets
is complete. Why is this not a polynomial-time algorithm
for the clique problem, which
that P = NP?
implies

Section 11.4.3)for

11.5

algorithm recognizes a

a nondeterministic

how

to recognize

+z).

problem is NP-complete.

of 3SAT to

the

3-coloring

problem

Creative Exercises 371

11.8
11.9

Pose

You

integer

linear program.

Exercises

exercises using
chapter.

complete these
exercises from this

can

11.10 Prove

only

the NP-complete

problem is NP-complete:Given
contains a spanning

the following

that

k, determine whether G

an integer

problems discussed

an undirected
tree

T such

that

in

the

text or

in

graph G = (V, E) and

each vertex in T has

< k.

degree

11.11

problem as an

the 3-coloring

Creative

other

problem is NP-complete.

the bin packing

that

Prove

Prove

vertex-cover

the

that

graph are

restricted to

have

problem remains NP-complete even

if

all

the vertices

in

the

even degree.

in Chapter 5.
discussed
11.12 Consideragain the problem of finding large induced
subgraphs
has
Suppose that, instead of the requirement that each vertex in the induced subgraph
is the formulation of the problem
in
that its degree be < d. Here
degree > d, we require
an undirected
and two integer
terms of a decision problem. Given
graph G = (V, \302\243),
parameters d and k, determine whether G contains an induced subgraph H with at least k
vertices, such that the degree of each vertex in H is < d. Prove that this problem is NPcomplete.

11.13

the following

that

Prove

G=(V, E)

integer

problem is NP-complete:Given
contains a subset of

the following

that

an undirected
problem is NP-complete:Given
k, determine whether G contains a clique

k, determine whether G

an integer

connected graph
size k and an

of

k.

set of size

independent

11.14 Prove

an

and

graph G = (V, E)
whose induced subgraph

an undirected
k vertices

and

is

acyclic.

11.15 Let E

be a CNF

expression

once as x. Either

are satisfiable

11.16Prove

as the one
assignment such that
variables

satisfying

assignment

Use

Exercise

11.18The input

looking

exactly once
whether

as x and

exactly

such expressions

of 3SAT,

The problem

with
is NP-complete: The input
is a Boolean expression in CNF
whether there exists a
clause, and the problem is to determine
such that in every clause exactly 2 of the 4 variables are true. (Hint:

per

is again
whose
for

x appears

11.16.)

there exists a
variables

in

2-in-4SAT

that

Prove

exactly 4

variable

called l-in-3SAT, is NP-complete.The input is

a satisfying
is to determine whether there exists
every clause exactly one of the 3 variables is true.

the following variation

for 3SAT.

that

each

algorithm to determine
that this problem is NP-complete.

or prove

the same

11.17

such that

a polynomial-time

find

CNF.
The problem is to determine
whether
that every clause contains
an odd number of
are 1. For example, if the input is a 3SAT input,
then
we are
such that, in every clause, either 1 or 3 variables have value
1.

a Boolean

satisfying

values

assignments

expression

assignment

such

in

372

NP-Completeness

at this problem is that the or operations are replaced with exclusive

or
of the problems in Exercises
11.16 and
like another variation
seem
fact this problem can be solved in polynomial time! Find a polynomial-time
to look

(Another way

This

operations.)

11.17, but

in

may

algorithm for it.

11.19

is

The input

undirected

an

degree). Prove that

exact coverby

11.20 The

3-sets

(X3C)

and a collection of

elements

elements. The problem

Sj

in which all vertices have

graph (i.e., a graph
problem remains NP-complete for regular graphs.

, S,,

is to

that each

such

Sin

the

regular

the clique

problem is
of

subsets

determine

the

The input

following.

whether there exists a

element of S is contained

is a set S

eacn
\342\226\240$*\302\273
containing

S, S,,S2

with

same

3n

exactly three
of

subcollection

subsets

one subset S, . Prove that

in exactly

is NP-complete.

X3C

11.21 Prove

following
problem is NP-complete:Given
determine
whether the vertices of G can

the

that

3a7

with

vertices,

that each group

question is to

contains

three elements,
whether

determine

graph G=(V, E)
into n groups, such
each other. In other words, the
undirected

an

be partitioned

each connected

to

the graph can be partitioned

n triangles.

into

11.22 Let G = (V,

U, E) be a bipartite graph such that V is the set of vertices on one side, U is the
V corresponds
vertices on the other side, and E is the set of edges connecting them.
to
is
and U to a set of parts. A machine
connected
to
a part u} if the
a set of machines
v,
machine is used to work on that part. Suppose that a room can accommodate
at most K
machines and unlimited
number of parts (for simplicity,
we assume that all machines
have
the same size). We assume
that we have as many
rooms
as needed, but we want
to
minimize
the movements of parts
from
one room to another.
Each
edge (vhpj) is
associatedwith a cost c (v,, py), which is the cost of moving part u} to machine v, if the part
and the machine are not in the same room. We define
the cost of a partition of machines
and parts into rooms as the sum of the costs of the edges
that
parts to machines
connecting
are not in the same room. Prove that the following
the
problem is NP-complete:Given
whether
it is possible to
graph G, the parameter K, and another parameter C, determine
the machines and parts into rooms
with cost <C. (In other words, the set of
partition
vertices should
be partitioned
into subsets, each with at most K vertices from V, such that
the sum of the costs associated with the edges that connect
subsets
vertices in two different
does not exceed C.)
set of

11.23 Let

four
of S each with
set, and C = IC,, C2,...,Ck} be a collection of subsets
Two subsets of S are said to be connected if they contain a common element. A
C is said to be a cycle if C, is connected to C/+1 for all /', 1 <i<k-\\,
and C] is
to Ck. An subcollection C'^C is called acyclic if it does not contain a cycle.

S be a

elements.
collection
connected

Prove that
element

in

11.24 Assume
that

the

Hamiltonian

the

are defined

that

NP-complete:Determine
such

that

(1) every

element of S is included

whether

two

subsets

in at

there

have

least one subset

collection C
at

most

one

of C.

Prove
is NP-complete.
graphs
path problem for undirected
cycle problem for undirected
graphs is also NP-complete. (Both

in

Section

11.4.)

undirected graph G =(V, E) and two distinguished

is no polynomial-time algorithm
to determine
path whose end vertices are v and w unless P = NP.

is an

a given
in C

Hamiltonian

Hamiltonian

input

Prove

and (2) every

common,

that

the

problems

11-25 The

problem is
subcollection C

following

an acyclic

contains

vertices v and w in G.
whether G contains a

373

Creative Exercises

11.26 Consider

whether
a graph G=(V, E) contains
the problem of determining
a
path with given end vertices v and vv (see Exercise 11.25). Let G' be the graph
obtained by adding two new vertices v and vv and two new edges (v, v) and (vv, vv). If G'
contains a Hamiltonian path, then its two end vertices must
be v and vv. Therefore,
such a
in
G
with
the
v
a
end
vertices
vv.
to
Hamiltonian
and
What
have
we
path corresponds
path
again

Hamiltonian

just proved?

11.27

that

Prove

graph

undirected

graph

^-coloring
G=(V, E)

is NP-complete. The
an

and

integer

is

problem

k, whether G can

to determine,

be colored with

given

an

at most

colors.

11.28 Prove that,

is a polynomial-time approximation
that can color any graph
algorithm
the minimal
number
of colors required to color that graph, then

if there

with less

than

times

4/3

P=NP.

11.29

Prove

of at
11.30

most

Given a
problem, called feedback edge set, is NP-complete:
E) and an integer parameter k, determine whether G contains a set F
edges such that every directed cycle in G contains at least one edge from F.

the

that

directed graph
k

following

G =(V,

Let Q be some NP-complete

problem
involving undirected graphs. Suppose that you find a
polynomial-time algorithm that solves Q for some particular restricted classof graphs
(e.g.,
Eulerian graphs). Does this algorithm
planar graphs, graphs containing
perfect
matchings,
can
be solved in
imply that all NP-complete
problems
involving undirected graphs
time when restricted to that class?
polynomial

11.31 Let G =(V, E)

distinct

vertices

whether

there

be an

undirected

Prove
exist k paths in G
of G.

and let (v,,

graph,

w,),

(v2,

(vk, wk) be k

w2)

pairs of

problem is NP-complete:Determine
following
such that path / connects v, to vv,, and all paths are vertex

that

the

disjoint.

11.32

Let G = (V, E) be an undirected

graph, such that each vertex is associated with some task.
if the corresponding tasks cannot
at the same time
Two vertices are connected
be performed
is the only limit
on concurrency.
Any set of tasks
(e.g., they need the same resource). This
in one step. Prove that
the
that no two of them
are connected
can be performed
such
is
a
and
an
G
iven
G
k,
=(V,
\302\243),
parameter
NP-complete:
integer
following problem
graph
whether all corresponding tasks can be performed in at most k steps.
determine

11.33 Let G = (V,

11.34

are
graph such that the edges incident to each vertex
nodes.
order. There is no relationship
between ordering at different
is to find an Eulerian tour covering
G that
Suppose further that G is Eulerian. The problem
v at an edge e,
If the tour enters a vertex
satisfies the following \"noncrossing\" property:
then
the next edge in the tour must be adjacent
to e in the cyclic order imposed on the edges
incident to v (from either side of e). Oneway to view this property is to look at a road map.
The goal is to travel through all the edges (road segments) such that an intersection (vertex)
is never crossed except from one edge to its neighbor. Prove that determining
whether such
an
tour exists for a given graph and cyclic orderings is NP-complete. (The
Eulerian
for planar graphs, but the proof is more difficult.)
remains
NP-complete
problem
E)

be an undirected

ordered

in a cyclic

Show an

example

3-coloring

11.35 The following

in

is a

the simple

which

a graph results

in

exponential

simple heuristic for

backtracking algorithm
number
finding

described

in Section

11.5 for

of nodes.

a vertex

cover.

In

each

step of the

algorithm,

374

NP-Completeness
is added to the cover,
then it is
of highest degree (ties are broken
vertex
arbitrarily)
terminates
when
removed from the graph together with all its incident edges. The algorithm
an edge is removed only
after
a vertex incident to it is
Since
no more edges remain.
indeed finds a vertex cover. This is a greedy algorithm
included in the cover, the algorithm
of
since it always selects the vertex with the highest \"payoff.\" The worst-case behavior
of a graph and an order of execution of
this algorithm is not very good. Show an example
cover
with more than twice the number of vertices of
that leads to a vertex
this
algorithm
the

the

11.36

cover.

minimum

Show

of the

instance

an

solution using 5/3 times

11.37 You

are

a traveling

you

are

planning

bin

more

for which the

packing
problem
bins than the optimal solution.

salesman of
your

flight

1980s.

the

algorithm

has asked you to

care about the cost
you want to maximize

Your boss

You do not

itinerary.

first-fit

gives

visit n cities, and

of travel \342\200\224
the

is going to pay your expenses anyway.

What
is the benefits
as
a
In
other
want
to
the
maximize
of the
words,
frequent flyer!
you
length
you
entire trip. You can assume that the mileage computed by the airlines is actual
so
mileage,
this is an Euclidean
airlines.
You
Also, do not worry about choosing different
problem.
just want to maximize the total mileage. Prove that this problem is NP-complete, and
company

receive

will

suggest
11.38

approximation

algorithms for

it.

Let Ci, C2, .., Cn be a set of coursesoffered

the time intervals (not necessarily
disjoint)

certain university,
and let f,, t2,..., tn be
which the courses are offered. For
example, f, may be Tuesday from 10:00 to 11:00,t2 may be Tuesday from 10:30 to 12:00,
and so on. Your job is to assign classroomsto the courses.
The only requirement
is that no
two
courses
is to use the minimal
overlap at the same time at the same room. The goal
number
of classrooms that satisfies
the requirement.

a. Reduce this
b.

Discuss

problem

to a coloring

why your solution

complete problem).

at a

during

problem, and

does not

imply

design

that P = NP

an efficient algorithm

(even though

coloring

to solveit.
is an NP-

12

CHAPTER

ALGORITHMS

PARALLEL

with one

person

a person

with

two

what time
is never sure.

knows

watch

watches

it

is;

Anon

small
first place, it is to be remarked that, however
the republic may be, the representatives must be raised to a
certain number, in order to guard against the cabals
of the
few; and that, however large it may be, they must be limited
to guard against the
to a certain
number, in order
confusion
of the multitude.
In the

James

1787

Introduction

12.1
The

Madison,

subject

science within
computer

of parallel computing has moved from

It is expanding very
a decade.

science).

from 2 to

There are numerous types

65,536 processors.

machines, evenas far

one \"generic\"model

The

differences

to mainstream computer
the exotic
fast (even relative to other areas of

of parallelcomputers

in operation,
ranging
the various existing parallel
We can no longer adopt
are major.

between

user is concerned,
and hope that it adapts to all parallel computers.
computation
them
correct
is much more
them, and proving
Designingparallel
algorithms,
analyzing
than
the corresponding
difficult
steps for sequential algorithms.
to cover all (or even most) areas in parallel
We
cannot
chapter
hope in this short
of examples
and
We present a variety
using different models of computation
computing.
We try to give the flavor of parallelalgorithms
and
to explore the
different
techniques.
with
them. We start
characteristics. We then
some common
in designing
difficulties
of
models
follow
used in this chapter, and
the main
describe
parallel
computing
briefly
as

the naive

of

ins

Parallel

376

Algorithms

and techniques.
are running
time and
for sequential algorithms
in
as
These
utilization.
well, but we
parallel
algorithms
important
space
resource
is
the
number
other
resources.
One
about
of
also worry
must
important
and they
cannot
There are problems that are inherently
be
sequential,
processors.
number
of processors is available. Most other
even if an infinite
\"parallelized\"
we use \342\200\224
can be parallelized to a degree.The more processors
however,
up
problems,
\342\200\224
the
faster
the algorithm becomes. It is important
to study the
to a certain limit
to be able to characterize the
that have
of parallel algorithms, and
limitations
problems
is
Since
the
number
of
solutions.
fast
limited, however, it is also
processors
parallel
very
Another
use
the
to
important issue is communication
processors
effectively.
important
to exchange data between two
the processors.
longer
Generally, it takes
among
than it does to perform simple operationson the data.
some
Furthermore,
processors
be
\"close\"
to
one
whereas
other
be
farther
another,
processors
may
may
processors
it is important
to minimize
and to arrange it in an
communication,
Therefore,
apart.
issue
is
which
is a major problem
effective way. Yet another
important
synchronization,
that
run
on independent
machines connected by some
for parallel algorithms
network.
Such algorithms
are
called distributed
communication
usually
algorithms.
in this book.
For lack of space,we will not consider distributed algorithms
We discuss
of algorithms

examples

with

The main

that assume

models

only

instruction

SIMD

full

each

in

step.

example

execute a

different

program

machines.

Unless specified

prominent

12.2

We

to

only

We
p

is

case we say
be

taken

utilization

of the

model.

the MIMD

section,

the

parallel

algorithm

from

the best

of the

processors is

\302\243(\342\200\236,p)=am

achieves

known

The

ratio

efficiency

and

one type

S(p)

book.

of model,and

this

in

that

definitions

of algorithms for
/?), where

is most

a perfect

sequential
the

used

apply
a

includes

it.

by T(a?,

algorithm

algorithm

of this

is the

input size

\\)/T(n,p) is called the

in which
effective when S(p)=p,
= T(n,

speedup.

algorithm.
of a parallel

The

value

An important
algorithm,

which

of T (a?,

1) should

measure of
is defined

the

as

= j>o>_.

pT(nyp)

The efficiencyis the

and examples

model

scope

that are

the

those

discussion

covers

sections

time of an

on

emphasis

some general

of processors.

algorithm.

that

can

processor
Multiple-Data)

(Multiple-Instruction

machine modelsis beyond

following

the running

the number
of the

speedup

machine is a

each

which

in

exact

the

are called

restriction

connection

The

assume

major models, with

description of the

denote

MIMD

called

are

this

follow

machines.

otherwise, we will

to execute

all processors
that

Parallel computers

computers.

this

in

Each

models.

more detailed
and

a few

We include,

many

computers

of parallel

survey

comprehensive

chapter.

restrict

of Parallel Computation

Models

mention

computation

Parallel

Multiple-Data)

(Single-Instruction

of such

synchronization.

of parallel

models

Some
same

of complexity
measures
are

measures

ratio

of the

time used by

one

processor

(with a sequential algorithm)

12.2 Modelsof

and

time

total

the

processors. (The total

of processors.) The efficiency

used

by p

by the number
time that is not

multiplied
processors'

wasted,

Parallel

actual elapsed

is the

time

time

of

the percentage

indicates

to the sequential algorithm.

compared

377

Computation

the

p)= 1,

If E(n,

the execution of the algorithm

is equal to the amount
the
In this case, we get
of work required by
sequential
algorithm.
the
such
an optimal efficiency is rare, because
use of the processors. Obtaining
optimal
most
of the time the parallel algorithm
introduces
some overhead
that was not required
is to maximize the
by the corresponding
sequential algorithm. One of our goals
the

then

done by

of work

amount

all

throughout

processors

efficiency.
we could
fix p, according to the number
we design a parallel algorithm
of
and
to
minimize
available
to
so
would
us,
T(a?, p). But, doing
try
processors
potentially
whenever
the number of processors changes. It is more
require a new
algorithm
that works for as many
values
of p as possible. We discuss
desirable to find an algorithm
next how to translate
that works for a certain value
an algorithm
of p to algorithms for

When

smaller values of p,
modify

an

factor of k

of

not

T(n,p)=X
In other words, we

more time.

to

steps in

significantly.

efficiency
an

with

algorithm

can use a

factor

the modified

construct

We

with k

algorithm

original

the

changing

with

algorithm

k>\\.

constant

without

k less

of

algorithm by
emulates

one processor

which

In

T{n, plk)~kX,
processors running
replacing
the

This principle cannot be applied to all situations.

the
interconnection
p,
algorithm may depend on a certain
in
Section
or
the
decision
discussed
12.4),
(as
concerningwhich

processors

may require computation time. However,this

and useful. It shows
folding principle, is quite general

emulate

principle,

for

any

for

step of

the

of one

step
For example, k may
of the
pattern

execution

processors.
divide

each

can

we

general,

called

processors

to

the parallelism

we can reduce the number of

significantly. If, for example, the original
(which was designed for a large p) exhibits a good speedup, then we can obtain
algorithm
the same
value of p. Therefore, we
speedup for any smaller
algorithms achieving about
should try to get the best speedup with the maximal number of processors, provided that
can
still
is good (i.e., close to 1). Then, if we have fewer processors,
we
the efficiency
other
with
the
small
use the same algorithm.
On
hand, parallel algorithms
efficiency are
that we have an
useful
for a large number of processors. For
example,
suppose
only
which
that the speedup
with
T(n, \\) = n and T(n, n) = \\og2n,
implies
algorithm

processors without

S(n) = n/\\og2n
Suppose

The
that

now

running

folding

the

changing

\342\200\224
a

very

time

of the

is possible),

On the other hand,

speedup (less than
The various

if p
2 with

efficiency

speedup

impressive

that the number

is 7(1024,

algorithm

is a

which

-16,

\342\200\224
and

of processorsavailable

parallel

that

then

speedup

the

running

of

about

E(n, n)= l/log2\302\253.

that
n = 1024.
and
=256,
p
= 41og21024= 40 (assuming

the
to

efficiency

us is

256)
25

over

time would

the sequential algorithm.

is not a good

be 640, which

16 processors).

in the way
the processors
computation differ
mainly
will
and synchronize. We
consider
full
assume
only models that
and concentrate on different
communication
The sharedparadigms.
synchronization,
there is a random-access shared memory,
that
that any
such
memory models assume
with unit cost. This assumption
variable
of unit-cost
access
processor can access any
or
the
is
size
of
of
but
it is a
the
of
number
the
unrealistic,
memory
processors
regardless

communicate

models

of parallel

Parallel Algorithms

378

most difficult
are connected

model

Other

hardware.

in

network.

interconnection

an

through

way to

easiest

the

usually

to implement

model

the

but

communication,

assume that
An interconnection
models

handle

they

way
12.3.

several

discuss

memory is

Shared

models differ in
shared-memory
different alternatives in Section

The

good first approximation.

access conflicts. We

the

the

it is

processors
be

can

network

and two
correspond to the processors
them.
vertices are connectedif the corresponding
processors have a direct link between
has a local memory that
it can access quickly. Communication is
Each processor usually
to arrive
at their
which
done through
may have to traverse several links
messages,
the
of
the
communication
on
the
distance
Therefore,
destinations.
speed
depends
different
have been studied as
the communicating processors. Many
between
graphs
will
several
networks.
We
mention
12.4.
interconnection
popular ones in Section
with
are
Parallel
message
computers based on interconnection networks
passing
sometimes calledmulticomputers.
that we discuss is that
of
A systolic
model
Another
systolic
computation.
line.
The data move through
the
in a
architecture resembles an assembly
processors
and
are
on
them.
Instead
of
fashion,
very simple operations
performed
rhythmic
having
a shared (or nonshared) memory,
the processors
receive their input
from
to access
their
are discussed
in Section
operate on it, and pass it on. Systolic algorithms
neighbors,
12.5.
A basic
theoretical
model that we will use only for illustration purposes is that of a
A circuit is a directedacyclic
circuit.
in which the vertices correspondto simple
graph
and
the
show
the
movement
of the operands. For example,a Boolean
operations
edges
circuit
is one in which
all indegrees
are at most 2 and all operations are Boolean
vertices
for input (with indegree of
(or,
and, and not). There are designated
operations
0), and for output (with outdegree of 0). The depth of a circuit is the longest path from

by a

represented

an input to

12.3
A

an

depth corresponds to

The

output.

fully

consists

of steps.

algorithms
each

synchronized
In

each

of several

consists

computer

shared-memory

step,

in

Exclusive-ReadExclusive-Write
Write

(CREW)

the

The
result of
alternatives

two

EREW

processors and

a shared

We assume

that

but

we assume
they

way

on

computation

the data
(In

that all memory is

handle

memory

We

memory.
the

it

practice,

global.)
The

conflicts.

does not allow more than one processor

Exclusivesame
time. The Concurrent-Read
memory
model allows severalprocessors
from the same memory location
to read

same
time, but only
Concurrent-Write
(CRCW)
at

Machines

processor performs an operation

or writes
into the shared memory.

and

processors

to handle

model

(EREW)

location

same

the

time.

running

parallel

section.

this

reads from the shared memory,

possesses,
each processor may also have local memory,
The shared-memory
models
in the
differ

to access

the

for Shared-Memory

Algorithms

use only

the vertices

that

such

graph,

one

at the

processor

model poses no

CREW

models
at the

write.

can

Finally,

the

Concurrent-Read

on memory conflicts.
defined, but it is not clear

restrictions

are well

what

is

the

same time to the same location. There are several

\342\200\224
concurrent writes. The weakest CRCW
the only
and
model
writing

for Shared-MemoryMachines

12.3 Algorithms

location
to

we

model

CRCW

values to

different

write

Surprisingly, as we

assume that
same location

the

to

write

another

We

a simple

with

start

problem

looks

that

thing.

at the

same

12.3.2,

If two

are labeled, and

processors
the same
time, the highest-labeled

the
at

is

an

that

processor

arbitrary

the

to

write

processors

the

time,

is very
when
that,

feature

this

to

processors

same

same

attempt

halts.

algorithm

powerful.

Another

several

processors

processorsucceeds.Yet

succeeds.

Addition

Parallel

12.3.1

in Section

assumption

possible

several
the

location

same

the

see

will

is to

alternative

\342\200\224
allows

time only if they all write

same

the

at

book

this

in

utilize

379

example of a

The Problem

Find

of two

sum

the

at first glance.

sequential

inherently

by induction, for

developed

algorithm,

parallel

AZ-bit

numbers.

binary

The regular sequential algorithm

at the least significant
starts
bits,
It
seems
that
we cannot
be sure of the
time with a possible
carry.
bits are added, since there may
until the two / - 1 least
significant

adds two

and

or

not be

may

at a

bits

of the

outcome

/th

step

a carry.

it is possible to design another

algorithm.
use induction on n. It will not help much to

Nevertheless,

to az, since
this
go from n-\\
The
has
a
much
divide-and-conquerapproach
implies
for parallel
better potential
algorithms, since it may be possible to solve all smaller parts
in parallel.
into two subproblems of size nil (we
Suppose that we divide the problem
We can find the sums of the two pairs in
that n is a power of 2 for simplicity).
assume
If the sum of the least
But we still have the problem of the carry.
significant
parallel.
pair.
pair has a carry, we have to changethe sum of the most significant
have a
are only two possibilities \342\200\224
we either
The key observation here is that there
to
include
we
can
the
induction
or
we
do
not.
Therefore,
strengthen
hypothesis
carry
We

an iterative sequential algorithm.

without
We

The

cases.

both

initial

an

pair

significant

we

or

exceptthat

no

with
sums

find

also

either

carry.

numbers: L,

get four

corresponding

problem is to find
Suppose that we now

modified

whether

initial

for the
it

Lc, /?, and

most

generates

which

/?c,

the

carry,

a carry.

on whether
/?c, depending
L is replaced
by Lc.
of size
solve a problem

same
pair,

significant

The

L had

sum

the

of the

correspond

problem for

sum

with

respectively.

final

a carry.

sum

to

final

sum

initial

an

For each

5, without

The

the

sum

initial

and

with

numbers

two

this modified

solve

of

carry,

both

pairs.

the least

and

the

of these sums,
carry,

Sc is the

is L

and

same as 5,

n by two subproblems of size nil and a constant

\342\200\22
of (conquer) steps. Since both
can be solved in parallel
subproblems
\342\200\224
can
access
different
bits
the
the
we
obtain
that
processors
assuming
independently
that T(n, n) = (9 (loga?).
recurrence relation
T(n,
n) = T(n/2, a?/2)+ 6>(1),which
implies
this algorithm
assumes
since both subproblems are completelyindependent,
Furthermore,
not
for
one
be
This
best
addition
the
model.
EREW
the
(see,
parallel
may
algorithm
only

We

number

Parallel

Algorithms

for example,

Ladner and

parallelization

of an algorithm.

380

in parallel,

quickly

very

12.3.2

Fischer

in an

of

example

an

easy

problem can be solved

further improved.

maximum

the

Find

a good
that the

Algorithms

Maximum-Finding

The Problem

is

it

clear

becomes

it

can be

solution

the

but

[1980]),

Once

among

n distinct

numbers,

given

array.

We solve this
The algorithms

different shared-memory models\342\200\224

EREW
and CRCW.
models use techniques that are used for many other problems.

for two

problem

both

for

EREW Model
The

algorithm

sequential

straightforward

comparisons.

of a

think

can

We

of the

for

n -1
requires
the two numbers,

maximum

the

finding

comparison as a game

between

played

to
problem is thus equivalent
whole
set.
An
efficient
being
in parallel
is to use a tree. The players
are divided into pairs for
way to run a tournament
the
first round
of players),
(with possibly one player sitting
out, in case of an odd number
all the winners
are again divided into
and so on, until
the finals.
The number of
pairs,
rounds
is riog2Az].
We can obtain
a parallel
algorithm from the tournament
by assigning
a processor to every game (think
of the processor
as the referee
of the game). We have
to ensure, however,
that each processor
knows the two competing
This can be
numbers.
the winner of the game
in the larger indexed position of the two
arranged
by putting
That is, if the game is played between xf and Xj such that j > /, then the maximal
players.
of Xj and Xj is put in position
jc 2, -1 to x2j
round,
j. In the first
processor
Pf compares
the

with

larger

running a

(1 <i <nll\\

^i^n/4),

EREW

model

Let's try

algorithm

of the

of the

maximum

the

second

involved in

The running
time
of this simple
the number of processors.

we

presented

just
the

Since

T(n\\nl2\\)=\\\\og2n\\.

efficiency

the

them if necessary;
in
Since each number is

so on.

is sufficient.

maximum-finding

the winner

exchanges
and

to minimize

now

The

and

The

winning.

with

tournament

*4/ 0

two

parallel algorithm

requires

\\nll\\

is clearly

algorithm

and

processors,

to

jt4/_2

time, an
O (log n).
we

have

achieves T(n, 1)= \302\253-1,

sequential
algorithm
(n, nil) = 1/log2fl. If

is E

round, P{ compares
one game at a
only

nil

the

are available

processors

that
is a part of another
algorithm
algorithm
modifications,
With
some
is
and
efficient.
algorithm
simple
however, we can achieve
a parallel
time of O (logn) with O (1) efficiency.
as
The total number
is n -1, the same
of comparisons
required for this algorithm
the sequential algorithm. The reason for the low
are
is
that
processors
many
efficiency
anyway

(e.g.,

requires

them),

idle
and
nl\\og2n

in

if the
then

rounds.

later

performing

load

processors.

per group,

and

assign

maximum-finding

this

We can

improve the

balancing
We divide the
in

a group

the
input

to each

by reducing

efficiency

following
into

way.

Suppose

the number of
that

we

about
A?/log2A? groups, with
the first phase, each

processor. In

processors

only about
log2A? elements

use

processor

finds

12.3

for Shared-Memory

Algorithms

381

Machines

the sequential
algorithm that takes about
group, using
log2fl
steps. It
the maximum among about
but
there
are
now
maximums,
nl\\og2n
to
the
tournament
The
time
of
use
this
enough
processors
algorithm.
running
algorithm
that n is a power of 2) is r(AZ,[Az/log2Az]) = 21og2AZ. The corresponding
(assuming
efficiencyis E (n)= lfr. Next, we formalize the idea we just used for saving processors.
We
call
a parallel algorithm static if the assignment
of processors
to actions is
and for each processor Pp
predefined. We know apriori, for each step / of the algorithm
the

the

in its

maximum

remains

find

to

now

is a static

example,

Pj uses at

the operands

and

operation

algorithm,

static
If there exists an EREW
total number of steps (over
all

static algorithm

with

Let a{ (i =
of the

processors

in step /

processors

to perform

with

stePs

|~0,7(5/01

p processorsin
of steps is now

the

1,2,...,/)

step /,

do not

we

and

in which

total

the

be

available

the

^+1

problem,

at=s.
\302\243

sit processors
the

of steps

number

If #,<$//,

have to change

(following

algorithm

of the

complexity

then

of 0(1).)

We have

algorithm.

original

folding

it.

performed by

all

then there are enough

we replace

Otherwise,

emulate

the

principle).

steps

taken

step

by the

The total number

= 2l

*+--i>-

/=1

(slt)\\

Hence,

O (t).

is equal to the sequential

algorithm has an efficiency

modified

Proof:

processors)

sit) =

T(n,

with T(n, p) = O (t), such

that
the
is s, then there exists an EREW

algorithm

if s

that,

(Notice

for

algorithm,

\"games\" are prearranged.

12.1

\342\226\241
Lemma

the

The maximum-finding

/.

step

all the

because

time of the

the running

modified

is still

algorithm

O (t).

\342\226\2

after
Brent [1973] (which contains a proof of
as Brent's lemma
the same
lemma
shows that, in some
cases,
spirit of a more complicatedcase). Brent's
the total
the efficiency of a parallel
depends
mainly on the ratio between
algorithm
and the running
time of the sequential
number of operationsperformed
by all processors

is known

12.1

Lemma

algorithm.
know
which
We need the restriction
to static algorithms because we must
to
emulate.
Lemma
12.1is
valid
for
nonstatic
as
well, provided
processors
algorithms
that
the emulation
can be done quickly.
An example
of a case where Lemma 12.1is not
valid is as follows. Supposethat
there
are n processors and
n elements.
After the first
decide
on
some
of
the
the
results
of
the
first
to withdraw.
(based
step,
processors
step)
after the second step, the third step, and so on. This algorithm
The same thing happens
is

similar

to the

processors
step, we
computation

tournament

withdraw.

need to
time

know

algorithm,

If we

try

which

to find that

out.

to

of

which
that, in this case, we do not know
except
emulate
the remaining processors, say, after
the first

them

are

still

active.

But,

it may

require some

Parallel Algorithms

382

Model

CRCW

find the maximum in less than log2fl

so.
The following
whose
algorithm,
comparisons
only
steps
the
of
concurrent
writes.
We
use
the
is
illustrates
time
0(1),
power
parallel running
in
more
can
write
which
two
or
to
the
same
writes
concurrent
of
version
processors
location at the same time only when they write the same thing.
a processor
use n(n-\\)/2
We
processors, and assign
P^ to each pair {/,j) of
shared
variable
We also allocate another
elements.
v, for each element xh and initialize
and writes a 0 in the
v, to 1. In the first step, each processor comparesits two elements
the smaller
element.
Since only one element
is larger
variable associated with
shared
1
Exercise
In
one
remains
also
the
second
all others,
than
(see
12.12).
v,
step, the
only
with
the
winner
can
determine
that
it is the winner and
can
associated
processors
two steps, independent
of
this fact. This algorithm
announce
n\\ its
requires
only
We call this
however, is very poor, since it requires
0(n2)
processors.
efficiency,

It may seem

first

at

a parallel algorithm
are used. But

that

if

the two-step

algorithm

We can improve

cannot

this

is not

algorithm.

of the two-step algorithm

by using a method similar
model. We divide the inputs into groups such that we can
of each group
allocate enough processors to find the maximum
by the two-step
algorithm.
of candidates
the
of
As the number
number
available
declines,
processors per candidate
increases,and the group size can be increased. The two-step algorithm shows that, if the
size of a group is ky then
to find the group's maximum
k(kl)/2 processors are sufficient
in constant
time.
Assume that we have n processors
and that n is a power of 2.
overall,
In the first round, the size of each group is 2, and
the maximum
of each group
can
be
found in one step. In the second round,
n
nil
are
have
elements
and
we
still
left,
only
If
we
set
the
size
of
each
to
then
we
have
be
nl%
4,
processors.
group
groups, allowing
us to allocate 8 processors per group. Thisis sufficient,
since
4 \342\200\242
(4- l)/2 = 6. In the third
elements.
Let's calculate the maximal
round, we have nl% remaining
group size that we
can afford. If the group size is #, then the number of groups is fl/8g, and there are 8g
To
use the two-step algorithm,
we
need g(g-\\)ll
processors available per group.
of size g\\ therefore,
we must have g(g - l)/2<8#,which
processors for a group
implies
that g < 17 (it is simpler
to use g = 16). We leave it to the reader to verify
that the size of

to

the

the

one

for

be

can

group

the

efficiency

the EREW

squared

in

each

round,

leading

to

an

algorithm

that

requires

O (log logn) rounds.

is

slower
that the two-step algorithm
(O (log logn)
versus 0(\\/n) of the
better. It is 0(l/loglogAz),
This technique has been called divide
since we divide
and
two-step
crush,
algorithm.
the
into groups of size small
with lots of
them
\"crush\"
we
can
that
input
enough
is not limited to the CRCW
model.
processors. This technique

this algorithm

Although

versus 0(1)),

12.3.3
The

the

its

efficiency

The Parallel-Prefix
problem

parallel-prefix

design

operation

of numerous
\342\200\224

namely,

it

slightly

is much

Problem

is important because it serves

as a major building block in
associative binary
be an arbitrary
parallel algorithms. Let \342\200\242
satisfies x%(yZ) = (x9y)9z \342\200\224
which
we will simply call

12.3

for

Algorithms

can
product. For example, \342\200\242
represent

addition,

383

Machines

Shared-Memory

or

multiplication,

two

of

maximum

numbers.

for

denote

by PR(i, j) the product

The sequential
\\<k<n.
k,

all

the prefixes in
we use is divide

compute
method

Induction

The caseof

is trivial.

element

one

half

values

by

associativity

Complexity
we have n

are

is divided

input

in

as

not

\342\200\224
we

a power

n is

simply

The

to solve.

easy

of 2.

problem

parallel-prefix

be

parallel,

for

is given
two

sets

in each

disjoint

in parallel

solved

of size az,

12.1.

then

read
x[Middle]
Although several processorsmust
distinct
so a CRCW model is not required.
locations,
(since the sequential
E(n
yn) = 0(\\/\\ogn)
algorithm
we
cannot improve the efficiency
Unfortunately,

recursive call of

the

of them

one-half

and

steps,

same

can be allocated
to
they can also be

clearly

runs

use x[Middle].

all

time,

they

T(n, n)

Overall,

in

If

model.

EREW

becausethey

at the

nil)
the

the

under

used

we use

that

(notice

in Fig.

and

PR(1,

computing

by

induction

by

first

the

in half,

input

of PR(\\,k)
half can be

values

can be obtained

known

into

values

the

by dividing
the

obtain

n/2
The combining step requires
but an CREW model is required

subproblem.

performed

The

operation). The algorithm

The

we

Thus,

\\<k<nl2.

Both subproblems can thus

for the problem
processors

algorithm.

the

proceeds

algorithm

for n/2<m<n

terms

Both
of the

k%

PR(\\,m),

\342\200\242
PR(n/2+\\ym).

The

induction.

for all

+ k\\

PR(n/2+\\,n/2

directly. The

how to solve

We know

compute PR(\\,k)

is trivial

problem

prefix

parallel-prefix
problem is
As usual, we assume that

conquer.

is to

goal

elements.

solving each

each

and

\\<k<n.

that

\342\200\242
\342\200\242
\342\200\242
\342\200\242
Our
\342\200\242xj.

version of the

..., *\342\200\236,
compute

x \\, jc2,

numbers

k, such

jc/*jc/+i

The

order.

hypothesis:

for nil

and

* * *
\342\200\242
for all
\342\200\242**,

x \\ mx2

the products

We

Given a sequenceof

Problem

The

O(n)

all

write

= 0(\\ogn),

to
and

steps).

of this

algorithm by using Brent's

is O (n logn). The waste
comes
of steps used in the algorithm
lemma. The total number
can compute all the prefixes
from the second recursive call. A sequential
algorithm
if we want to improve the
we
the second recursive call. Therefore,
without
efficiency,
number
of steps is reduced. We do that
must improve the algorithm
such that the total
next. An EREW algorithm
for
this problem
(with the same resource
bounds) is the
of
Exercise
12.18.
subject

the Efficiency of

Improving
The

trick

is to

use

the

set

of all

Jt,s

with

that

a?

is a

i even.

the rest of the prefixes

finding
all i such that 1 < i < n/2, then,

but to divide

in a different
the
input
are n processors. Let E denote
If we find
the parallel
in \302\243,
then
prefixes of all elements
for
(those with odd indices) is easy: If PR (1, 2/) is known
we
need
to compute one
for each odd prefix PR (1, 2/ +

induction

same

the

way. Assume again

Parallel Prefix
hypothesis,

power of 2

and

that

there

1),

Parallel Algorithms

384

Input: x (an
{

(x, n) ;
range 1 to n).

Parallel_Preftx_l

Algorithm

a powerof

that n is

assume

we

x (the

Output:

in the

array

contains

element

/th

the

/th

prefix).

begin

PPJ(Ln);

end

procedure PP_1 (Left,

Right)

begin

if

Right

= 1

Left

then

:= x[Left]

x[Right]

%x[Right]

else
:=

Middle

(Left +

Right

binary operation }

is an associative
{ \342\200\242

- l)/2 ;

in parallel

do

{ assignedto
+
1,Right); {assigned
PPJ(Middle
+ 1 to Right do in parallel
for i := Middle
Middle);

PP_l(Leftf

x[i]

P,

to Pnl2

to Pn/2+\\

to Pn }

:=x[Middle]mx[i]

end

Figure 12.1 Algorithm

more

Parallel

prefix

J.

of the elements in E in two

for
all
the
and
we store
1</<az/2,
jc2/-i*jc2/
In
other
we
left
the
of
all
of
with
their
elements
E
words,
*2/compute
products
The result
Then, we solve the Az/2-sized
neighbors.
prefix problem for E (by induction).
for each x2i is the correct
with
since
each
the
prefix,
jc2/-i x1{already includes
product
And if we know
the prefixes
of all the even indices,then
we have
already seen how to
the odd prefixes in one more parallel step. We leave it to the reader to verify
compute
product

phases.
result in

that

this

First,

2/)

(f7?(l,
we

algorithm

\342\200\242

compute

requires

Jt2/+1).

We

can

find

the

prefixes

(in parallel)

only

the

EREW

model.

The

algorithm

is given

in Fig 12.2

(see also Exercise12.17)

Complexity

in parallel
Parallel_Prefix_2 can be performed
the
half
of
to
a
is
call
The
recursive
processors.
problem
applied
size, so the running
of steps S(n)
time
of the algorithm is O(logAz). The total
number
satisfies the recurrence
that
relation
implies
S(n) = S(n/2) + n-1, 5(2)=1, which
this
that
to
S(n) = 0(n). But,
we
now
use
Brent's
lemma
can
implies
improve the
we
can
Brent's
with
in
time
the
run
to
lemma,
efficiency.
By
0(logA?)
modify
algorithm

in O (1)

time

with

Both

loops

in algorithm

nil

to an 0(\\)
only 0(n/\\ogn) processors,leading
efficiency. The key to this improvement
is using
call
one
recursive
of
(instead
while
still being able to perform the
only
two)
merge
step in parallel.

12.3

Input: x (an

that n is

assume

we

x (the

Output:

in the

array

a powerof

2 )

contains

the

element

/th

385

Machines

Shared-Memory

(x, n) ;
range 1 to n).

Parallel_Prefix_2

Algorithm

for

Algorithms

/th

prefix).

begin

PPJ(I);

end

procedure PP_2 (Inc);

begin

= nil

if Inc

then

:= x[n/2]mx[n]

x[n]

else
for i := I to
x[2

is an
{\342\200\242

\342\200\242

n/(2

do in

Inc)

\342\200\242
\342\200\242
/ Inc ]

associative

parallel

\342\200\242
\342\200\242

:= x[2

binary operation}

Inc-Inc

\342\226\240
\342\226\240
\342\200\242
/ Inc ] ;
jc[2

PP_2(2Inc)

do in parallel
for i := J to n/(2 Inc)-\\
\342\226\240
\342\226\240
\342\200\242
\342\200\242
\342\200\242
/ Inc -\\-Inc ] := x [2 i Inc ] \342\200\242*[2
/ Inc -\\-Inc
x[2 \342\200\242

end

Figure 12.2 Algorithm

12.3.4

environment

than

only access to
one

element

more

is much

it

Generally,

with

arrays.

link

list is

the

at a

elements of the

time

with

J'refixj.

in Linked Lists

Ranks

Finding

Parallel

to

difficult

through
no

with

deal

lists, for

Linked
the

possibility

head

linked representations

example, are inherently

of the

list,

we

then

of parallelism.

In

sequential.

have

many

in

to traverse
cases,

parallel
If the
the

however,

list

the

in a contiguous array;
the
to them) are actually
stored
In such cases,
of the array.
order
imposed on the elements
by the list is independent
where parallelaccessto the list is possible, there is hope for fast parallelalgorithms.
of the
The
rank
of an element in a linked
list is defined here as the distance
element has rank
element from the end of the list (thus,
the head has rank az, the second
n- 1, and soon).
list

(or pointers

The Problem
storedin an array

We can solve

the

Given

A [\\..n ],

sequential

linked

list of

problem

by simply

all of

elements,

compute for each element

traversing

its rank

the

list.

which are
in the list.

The method we

will

Parallel

386

Algorithms

for
use for designing a parallel algorithm
element.
to
each
each
Initially,
processor
element in the list. In the first step, each
the first step, each processor knows
After

problem is
knows
processor
this

called doubling.

We

assign

of its

only
right
neighbor
finds the neighbor
of its neighbor.
at distance 2 from
its element.
the element
If,
k
from
its
the element
at distance
then in one
element,
at step /, each processorknows
until
the
step each processorcan find the element at distance 2k. This processcontinues
to the right
of / in the list that
element
Let N [i ] be the farthest
end of the list is reached.
is /'s right neighbor (except for the last
is known to Pt at a given moment. Initially,
N[i]
whose right neighbor is nil). Basically,in each step, Pt updates N[i] to N[N[i]]
element
is reached.
Let R[i] be the rank of /. Initially, R [i] is set to 0,
the end of the list
until
the
last
element
in the
for
list, for which it is set to 1 (this element is detectedby
except
with a nonzero rank
When a processorencounters
a neighbor
its nil pointer).
/?, this
of rank 1 knows its
Initially, only the element
processor can determine its own rank.
of rank 2 finds
After the first step, the element
that its right neighbor has rank
rank.
1, so
the second
rank
and
rank
4
are
3
it knows that its own rank is 2. After
determined,
step,
If f\\ finds that N[i] points
to a \"ranked\"
so on.
element of rank R after d doubling
and
is given
in Fig 12.3. This
steps, then /'s rank is 2d~l +/?. The precisealgorithm
can be easily adapted to the EREW
model (Exercise
12.4).
algorithm
processor

(N);

List_Rank

Algorithm

the

N (an array
N (the rank
Output:
Input:

range 1 to
element

in the

of each

of
in

indices).
the

array).

begin

D:=l;
{

each

do in

can have

processor

we use only

local

}
P( is activeuntil

{ processor

parallel

own

its

D variable;

D variable

one

R [i

] becomes

nonzero }

R[i]:=0;

ifN[i] =
while

R[i]

then

nil

= l

R[i]

=0do

ifR[N[i]]*0then

R[i]:=D+R[N[i]]

else

N[i]:=N[N[i]];
D := 2D
end

12.3 AlgorithmListRank.

Figure

Complexity

The doubling

of

most

the

E(n,

list

in at

n) = 0(\\llogn).

[~log2\"l
Improving

process

guarantees

steps.

Therefore,

the

efficiency

that each

processor

will

reach

the end

efficiency
T(n, n) = 0 (logn). The
requires making a major modification

is
to

12.3

the

the total

since

algorithm,

of

amount

Machines 387

for Shared-Memory

Algorithms

logn). (See the

is 0(n

work

discussion on recentresults.)
rank
allows us to convert a
computation

Bibliography

for a

section

The

time

an array in O(logAz)
ranks are computed,

into

list

linked

than

the

(even though
perfect efficiency).
locations in the array, and the rest
can be copied into the appropriate
on the array, which is much easier.
can then be performed directly
computation
less

with

elements

the

We

can

parallelize

many types

the

tree

in parallel

(e.g.,

time of

an

such

as

need

az-1,
in

instrumental

another

the

the

the

find

is not

tree

number

balanced,
Euler's

The

approach.

the tree.

is the

a whole level of

maximum).

of

height

(where

algorithms on trees,

parallel

designing

to

algorithm

However,when

is quite good.
and we

approach
high

by operating on

trees

on

algorithms

is proportional to
is O(logAz)
height

its

and

of

tournament

the

algorithm

balanced,

reasonably
this

of the

The Euler's Tour Technique

12.3.5

as

all

After

and

The running
If the tree is
of nodes), then
may

be

tour technique

is

the

height

unbalanced

on

especially

trees.

Let T be a tree.
representation,
of the list of
/ is

since

edges incident

to

that include

records

contains

additional

known),

and

the

a pointer

T is

that

assume

We

one

with

represented

the

by

adjacency-list

regular

pointer E(i) to the start

the vertex / (\302\243(/) nil if this list is empty). Each list
to store only y,
corresponding
edge (/, j) (it is sufficient
Each (undirected)
to the next edge in the list Next(iy
j).
feature.

As

is a

there

usual,

edge (/, j) is represented

copies, one for (/, j) and one for (/, /). The
by two (directed)
extra pointer for each edge that points to the other copy of that
when
need that extra pointer to find the edge (/, /) quickly
edge (/, j) is given.

feature

additional

We

edge.

is an

This representation

is illustrated

in Fig.

12.4 (the

pointers

connecting

the copies

are

not

shown).

The

tree

that

key idea
forms

behind

an Euler's

appears twice). Once this

the

Euler's

tour of
list

the

is built,

tour technique is to construct a

version
of the tree (in
directed
many operations on

the

of

which

can

edges

of the

edge
be performed
every

E
(1,2)\342\200\224>
I\342\200\224\342\226\272

(1,5)

\342\226\272
(2,3)

\342\226\272
(3,2)

\342\226\272
(4,2)

\342\226\272
(5,1)

Figure

tree

list

12.4 The representation of the

tree.

\342\226\272
(2,1)

\342\226\272
(2,4)

Parallel

388

always traverse
parallel.

construct

will see

two

perform
us to

examples

list

was linear.

together

operations

perform such operationson

of

such operations.

But

algorithm

sequential
with

first

the
the

we

traversal.

can
This

tree

efficiently in
discuss how to

tour.

Euler's

the

allows

tree

the

the

though

and

tree

the

\"linearization\" of
We

as

list almost

the

on

directly

Algorithms

tour of T (with
an
Euler's
It is easy to find
edge
appearing
twice)
every
the opposite
the tree using depth-first search, taking
order
sequentially. We can traverse
in parallel.
the search backtracks. We will do a similar thing
Let
of an edge whenever
in
the edge following
the
tour.
We
claim
that
NextTour
NextTour (/, j) denote
(/,
j)
edge
which can be easily computed in parallel:
is defined by the following
rule,
i)

if Next

(j\\

i) is

not

nil

otherwise.

\302\243(/)

{Next(j\\

incident to a vertex is consideredin a cyclic order, such

last on /s list,
to by \302\243(/), is
tne
then
the first edge on that
if
list,
that,
pointed
12.4 (we assume that
with
the
For example, if we start
taken.
edge a in Fig.
edges
then the tour consists of the
incident to each vertex are ordered
clockwise),
edges
Next (J, i) to follow (/, j) in the tour, we
a, d, #, c, /, e, b, h, and back to a. By choosing
to j are chosen.
guarantee that (/, /) will be chosen only after all other edges incident
the
subtree
rooted
at
will
be
traversed
we
backtrack to /.
before
Therefore,
j
completely
We
leave
the proof that this procedure is correctas an exercise.
Once the list is constructed, we can choosea starting
(r, t) (any edge will do),
edge
and mark the edge before it in the tour as the end of the list. The vertex r is chosenas the
root
of the tree. We can now
the edges according to their
in the list by
number
position
R
in
Let
denote
the
rank
of
the list (e.g.,
(/, j)
algorithm
(Fig. 12.3).
edge (/, j)
List_Rank
=
where
n
is
the
of
We
now
show
two
number
/?(r, t) 2(n -1),
vertices).
examples of
\342\200\224
on
the
tree
the vertices in preorder
and computing, for
traversal,
operations
ordering
all vertices,
the number of their
descendants.
Let (/, j) be an edge
in the tour. We call (/, j) a forward
if it points away
edge
from
the root, and a backwardedgeotherwise.
of the edges allow us to
The
numbering
if
between
forward
and backward edges: An edge
distinguish
(/, j) is a forward
edge
R (/, j) > R (/, /). Since the two copies
we
a
of the edge (/, j) are connected
pointer,
by
can easily determine
we can make
which
one of them is the forward
Furthermore,
edge.
this determination
for all edges quickly
in parallel.
We are interested in forward
edges
because they impose preorder on the vertices.
Let (/, j) be the forward
edge leading to j
In other

words,
(/, 0 is

/ is

/s

list

the

of edges

If/(/, j) is the
number for j is n

the tree).

of forward

edges following
of r,
(The preorder number
which is the only
to it, is 1.) We can now use a
forward
edge
pointing
variation
of the doubling
(/', y), the value of
edge
algorithm to compute, for each forward
We leave the exact implementation of this
as an exercise.
/ 0\\ 7 )\342\200\242
algorithm
doubling
The secondexample
involves
for each vertex y\", the number of vertices
computing,
below j (descendants)
in the tree.
Let (/, j) be the (unique)
forward
edge leading to j.
Considerthe edges following the edge (/', j) in the list. The number of verticesbelow j in
of forward edges that are below j in the tree. We already
the tree is equal to the number
(namely,

(/, j)

in the

list,

parent

in

the preorder
vertex
with no

then

number

-f

(/', j).

12.4

know

how

edges

that

to compute
are after (/,

edges

/(/,

/) following

of j is equal
performed

in

We can

list.
the

list.

y),

compute

similarly

easy to see that

It is

is

which

the

of forward
of forward
of descendants

the

number

the

number

number

be modeled

can
to

the

nodes and

two

by

graphs

nodes

389

The
be

Networks

for Interconnection

networks

processorscorrespond

(almost
are connected

always undirected). The

if there is a direct link

and can also

processors. Each processorhas local memory,
local
the
memories
of
other
network,
Thus, all memory
processors.
but
the cost of accessing a variable
on the locations of the
depends
the variable. A shared-memory
access
as a local access
may be as quick
to be in the same processor),
or as slow as a traversal
of the
happens

the corresponding

between

access, through
can be

the

shared,

and

processor
the

(/, /)

Algorithms

Interconnection

(if

j)

the

value of/(/,

the

parallel
in

Networks

Interconnection

0. which can be found

again
to/(/,
by the doubling technique.
j)-f(j,
the
and
for
of descendants can both
for preorder
number
finding
numbering
=
in time T(n, n) 0 (logn)under
the EREW
model.

algorithms

12.4

in

for

Algorithms

variable

whole network (in case

communicate

Processors

variable

that

is located

is a

simple chain). It is usually somewhere in between.

to access
a shared
by messages. When a processor wants
at another processor, it sends a message asking for this variable.
the

graph

is routed across the network.

networks.
The simplest
graphs have been suggestedas interconnection
ones include linear arrays, rings,
trees, stars, and two-dimensional meshes(grids).
binary
the better the communication becomes. But edges
The
more
edges we add to the graph,
increase
the
for the layout
of the wires, which
are expensive
area
required
(e.g., edges
We have to find the right tradeoff. There is no
increases the time for communication).
on a certain graph
one type of graphs that is good for all purposes. The performance
of the particular
on the communication patterns
algorithm. There are,
depends
heavily
that are very useful. We list some of them below, together
several
however,
properties
with
of interconnection
networks.
examples
is the maximum
The diameter of the graph is of great importance. (The diameter
between any two nodes.)
It determines the maximum
number
of all the shortest
distances
nwhereas
an
of hops that a message
have
to
take.
An/ix/j
has
diameter
In,
may
grid
+
node
balanced
binary tree has diameter 2log2(fl l)-2. A tree can thus deliver a
message(in the worst case) much faster than a grid. On the other hand, the tree has a
major bottleneck. All traffic from one half of the tree to the other half must pass through
the root. The two-dimensional
mesh
has no such bottlenecks, and
it is very symmetric,
which
is important
for algorithms that communicate
in a symmetric fashion.
A hypercube
is a popular topology that
combines
the benefits of high
symmetry,
routes
alternate
between
A
dsmall
two
and no bottlenecks.
diameter,
nodes,
many
n = 2d processors.
of
consists
dimensional
The
addresses
of
the
are
hypercube
processors
each
address
contains d bits. ProcessorPx is
integers in the range 0 to 2d 1. Therefore,
one bit. The distance
to processor Pj if and only if / differs from j by exactly
connected
is never more than
we can go from
betweentwo processors
d, since
Px to Pj by changing
A four-dimensional hypercube is shown
in Fig. 12.5.
d bits (one bit at a time).
at most
The

message

Numerous

Parallel

390

Algorithms

12.5

Figure

The hypercube
any

two

a rich

provides

(e.g., we

processors

combine the

with

hypercube

A four-dimensional

hypercube.

are many

connection, since there

can changethe

bits

appropriate

the mesh architecture

by,

for

of the

faces

12.4.1
We

Sorting
with

start

on an

We

order).

embedding

example,

include
hypercube. Other suggestednetworks
connectedcycles,quad and octal trees, mesh of trees, butterfly,
the

any

between

routes

different

in

can also
meshes in

shuffle, cube-

the perfect
more.

and

Array

the relatively simple problem of sorting

P X,P2, i>,Pn> and n inputs X\\,x2,

on

of processors.

an array

There

one
smallest
the
such
that
the
goal
input among
processors
we may want to assign
input is in P x, the second smallest is in P2, and so on. In general,
more than one input to each processor. We will see that the same algorithm
be
can
to
Each
this
case
as well. The processors are connected
in a linear fashion.
adapted
processorP{ is connected to Pi+l, 1 < / < n.
are

input.

processors

is to

The

Since each
comparisons
the

array.

and

distribute

processor
possible

Each
...,*\342\200\236.

processor

holds

the

can
exchanges

communicate
can be

only

done

In particular, in the worst case, the

it takes
for an input to move from

with

must

algorithm

its

with

neighbors,
that

elements

allow

the

only

are consecutive

in

- 1 steps, which

is

The
array
of
is
number
to
the
as
its
number
follows.
Each
algorithm
basically
processorcompares
one of its neighbors, exchanges the numbers if they are in the wrong order, and then does
the
same
with the second neighbor. (We must
because otherwise the
alternate
neighbors,
same
numbers
all numbers
will be compared again.) The same process
continues until
are in the correct order. The steps are divided
into odd steps and even steps. In the odd
with those of their
numbers
their
right
steps, the odd-numbered processors compare
in the even steps, the even-numbered
neighbors;
compare their numbers with
processors
those
of their right neighbors (see Fig. 12.6). This
are synchronized
all processors
way,
and
a comparison
If a processor does not have
always involved the correct
processors.
the

time

one

end

of the

to

the

other.

12.4

X2

Xt,

X$

X$

X(y

X-j

;*8

corresponding

this step.

through
in Fig.

12.7.

sort

be

very

algorithm

Odd-even

12.6

neighbor (e.g.,

the

This algorithm

is called

numeric

first

of the

example

sort.

transposition

second step), it remains

idle
transposition sort. It is given
in Fig. 12.8. Notice that
presented

in the

processor

the odd-even
is

algorithm

example is completeafter only six

hard to detect in a network.
Therefore,
run to its worst-case completion.

in the

the

Networks 391

X i

Figure
the

for Interconnection

Algorithms

However, early termination

it is better to let
cases
many

steps.
in

can
the

seems
of
natural
and clear, but
its
Algorithm
proof
Sorting_on_an_Array
correctness is far from obvious.
For one thing,
an element
from
its final
may move away
destination. For example,in Fig. 12.8, 5 moves to the left for two steps before it starts
to the right,
and 3 moves to the
leftmost
and stays there for three steps
moving
position
before it moves back to the right.
is
the
correctness
of parallel algorithms
Proving
of
because
the
the
of
different
actions
the
difficult,
interdependencies
among
processors.

Algorithm
Input:

Output:

(x, n)

Sorting_on_an_Array

x (an array

x (the

the

in

range

az,

that xt

such

order, such

in sorted

array

1 to

the

that

resides

at

Pt).

/th smallest

element is

begin

do in parallel

\\

n/2\\

times

/>2/-i and

elements
and exchange them
such that \\<2i<n}
their elements and exchange them

P2i comparetheir
if necessary ; {for all i,

P2j and

P2t+\\

if necessary

compare

; (for

[ifn is odd, then

all i, such that

this

step

is done

< 2/

only

<n}
|_a?/2J

end

Figure 12.7 Algorithm

Sorting_on_an_Array.

time }

in

Pt).

Parallel

392

Algorithms

L
5

J
7

J
6

L
2

j
8

J
5

L
2

J
7

L
4

/
i

l_

j
5

J
5

L
6

/
i

12
Figure

The behavior
on

An

example

of odd-even

transposition sort.

processor affects all other processor, and it is usually

are correct; we have
and prove that its actions
processor
of

one

only

processors

12.8

L_

one

hard to

focus

to consider all

together.

12.2

\342\226\241
Theorem

When

Proof: The

terminates,

Sorting_on_an_Array

algorithm

the numbers

are sorted.

of processors
induction on the number
(or elements). If
then
one
sorts
the
two
We assume
numbers.
processors,
comparison
that
the theorem
Let's
is true for n processors
and consider the case of n + 1 processors.
focus our attention
on the maximum
element and assume that it is xm (e.g., x5 in Fig.
In the first step, xm will be compared to either xm_\\ or to ;cm+1, depending on
12.8).
whether
m is even or odd. If m is even,
then no exchange will take place because xm is
than
this
is
the
same
at Pm-\\
But
as the case of xm residing initially
greater
xm_x.
exactly
an
can
loss
of
we
without
(and
Therefore,
assume,
exchange taking
generality,
place).
that m is odd. In this case, xm is compared to ;cm+1, exchanged,
and moved step by step
to the right
at xn+\\
it arrives
it is greater than
all other
(since
numbers) until
(diagonally)
and stays there.
This is its correct
so the sort works correctly for the maximum
position,

there are only

proof

is by

two

element.
too. There are
the elements are sorted
correctly
induction. To do that, we have to map the
execution
of the n processors in the array of size n +1 to a possible execution of n -1
is done as follows. Consider the diagonal
processors in an array of size n. This mapping
formed
the
movement
of the maximum
element
by
(see Fig. 12.9). The comparisons
the
maximum
element (i.e., those that are on the diagonal) are ignored. We
involving
divide
the other comparisons into
and the one
two
the one below the diagonal
groups,
above it. We then \"move\" the triangle
one step down. In other
the diagonal
above
in
the
for
the
For
called
words,
comparisons
upper
step /+1.
triangle, step / is now
1
consider
the
12.8.
versus
and
4
of
8
first
versus 2 in the
comparisons
example,
Fig.
step

We now must show

other

elements,

and we

that

would

the

rest of

like

to use

12.4

Figure 12.9
first comparison is on
so we consider it

The

the

step involving
side of

5, 6

7 versus

versus 1, and

in the

so it

diagonal,

of step
(from

We can

n elements.

only

step

be part

to

diagonal,

consistsof

induction

The

now

proof of theorem

is ignored;

step

of

1) 4

versus 2.

ignore

simply

12.2.

step

is above

one

second

the

(instead

393

Networks

Interconnection

for

Algorithms

the

1). Therefore, step 2

this is a valid even
But,

step 1 (i.e.,

1) on

row

the left

(the last
involving
from
of
that
can
result
a
column), and the rest
running
sequence comparisons
is
the sort on n elements
for n elements. By the induction
the
hypothesis,
algorithm
\342\226\
correct; therefore, this sort is correct too, and it requires
only one more step.
the

and

diagonal

all comparisons

the

element

maximum

is exactly

So

each

we have

far,

assume

the

that

at P

reside

the

of one input

case

and

consider

redistribute

the

first the
elements

reside at
better
so
we
cannot
do
moved,

the largest

be

the sort is to

to achieve

inputs

is to

goal

x, and

must

elements

discussed

holds k

processor

k elements

repeat the

following

step

P2>
than

per processor. Supposenow that

We
case of only two processors.
such that the smallest k elements
It is

clear

until the sort is

worst case

in the

that

2k element

movements.

One

completed: P x

aH

way

its

sends

terminates
largest element to P2 and P2 sends its smallest element to Px. The process
when the largest element in P x is not greater than the smallest elementin P2. This step
is called merge-split. If we use this step as the basic step in the odd-even transposition
we can

sort,

possibly

an

extend

and
of a comparison
elements per processor. Instead
is
done.
a
elements, merge-splitoperation
for an array, its
in this section is optimal
presented
algorithm
total
for n steps; therefore, the
each running
n processors
to many

of neighboring

exchange

Although

sort

the

the

efficiency is low.

sorting
We

have

n2. The low

is not surprising, since an efficient

efficiency
The array cannot
elements
that are far away.
must be able to exchange
we present
interconnection networks
an exchange.
In the next
such
section,
designed specifically for efficient sorting.

of steps is

number

algorithm

12.4.2
When

number

as

Sorting
we

design

of steps.

independent

as

sorting
support

that

are

Networks

we are concerned
an efficient sequential algorithm,
of a parallel algorithm,
we must also
In a design
Consider mergesort (Section 6.4.3). The
possible.

only
try

to
two

with

the

total

the steps
calls
recursive

make

Parallel

394

Algorithms

and they can be performed in parallel.

the merge
However,
We
in
manner.
the
/th
in the
is
a
serial
element
place
algorithm
performed
part
If
we
can
/
1
elements
are
the
after
the
first
final
placed.
parallelize
merge,
only
array
we will be able to parallelize mergesort.
then
a different merge algorithm,
now
We
describe
developed
by Batcher [1968], using
We assume for simplicity
that
n is a power of 2. Let a i, a2,..., an
and conquer.
divide
and bub2,..., bn be two sorted sequences that we want to merge, and let jc j, jc2, ..., x2n
We want to merge disjoint
of
be the final merged order (e.g., x\\ =rmn(aXy
bx)).
parts
so that the final
becomes
This
is
done
these sequences in parallel
merge
easy.
by
\342\200\224
two
the odd-indexed
the two sequences into
elements and the evenparts
dividing
elements.
Each part
is merged
with the corresponding part
of
the other
indexed
sequence, then a final merge is performed. Let o {,o2,...,on be the merged order of the
are

completely

odd

independent,

the

of

a^a3y

subsequences

order of

the

x2n=en.

The

even

rest of

the

(see also Fig.

theorem

and
...,fl\342\200\236_i

bub3

merge

let ex,e2,

and

...ybn_Xy

a2,a4, ...,an and

is also easy to obtain,

subsequences

b2,bA...,bn.
as

can

be the
...,\302\243\342\200\236

be seen

the

by

and

following

12.10).
12.3

\342\226\241
Theorem

Following the notation

x2i=min (oi+i, et) andx2i+\\

above,

for

all i such

=max(oi+i,

that

we

\\<i<n-\\,

Consider ex. Since et is the /th element

in the merged
is
than
or
to
at
/
even
least
elementsfrom
et
greater
equal

order of
both

-+ *2

a2
nil

aA

merge

-\342\200\242
-*4

network

<*8

-* *10

b2

b3

b4
b5

nil

-\342\200\242
*ll

merge

-\342\200\242
x\\2

network

-\342\200\242
*13

be

-\342\200\242
*14

bi

-\342\200\242
*15

bs

-\342\200\242
*16

Figure

12.10

have

et).

Proof:
sequences,

merged

Clearly, x{=o{

The circuit for odd-even

merge.

the

even

sequences.

12.4
even element,

Algorithms

for

Interconnection

Networks

395

more odd element

that
et is greater than
is
than
or equal
to at
Therefore,
et greater
sequences).
least 2/ elementsfrom both sequences.
In other words, ex >x2i.
the
same
By
argument,
is greater than or equal to / + 1 odd elements,
which
that it is greater than or
tf,+i
implies
have to subtract 2 from
the index,
because
(We
equal to at least 2/ elementsaltogether.
the first elements
in both odd sequences do not
add
Hence,
any more elements.)
of the pigeonhole
But now, by a variation
\302\260i+\\
^x2iprinciple, both et and oi+l must be
e { and o2 can be equal to
equal (in some order) to x2i and x2i+{. Thisis so because
only
But, for each

(since

we started

x2 and jt3, and that

with

two

makes

we

can

add one

sorted

only

e2 ando3

fitjt4

andjt5,and

soon.

\342\226

property of Theorem 12.3is that the final merge can be obtained in

The
The parallel algorithm follows directly
rest is done by induction.
parallel
step.
from
the theorem.
the
12.10
illustrates
recursive
Figure
merge construction, and Fig.
12.11 shows the complete sort, which
is called
odd-even
The seemingly
mergesort.
small
boxes
on the left side of Fig. 12.11 (marked
\"nil
are recursive
sort\")
constructions of the whole sort. The numbers on the right side are the input in the sorted
order.
The

important

one

The recurrence

for the total number of steps TM(n)

relation
for
the
=
is
+
1.
the
total
of
number
n-\\,
Therefore,
TM(2n) 2TM(n)
TM(\\)=
merge procedure
that requires only O (n)
comparisons is O (n log az), in contrast to the sequential
algorithm
of the recursion,
which corresponds to the parallel
time, is O(logAz).
steps. The depth
The recurrencerelation
for the total number of steps Ts(n) for
odd-even
is
mergesort

Complexity

x\\6

Figure

12.11

The circuit for odd-even

mergesort.

396 Parallel Algorithms

Ts(2n)

= 2Ts(n)

circuit

contains

+ 0(n\\ogn)y
n processors

there are

of

computation

12.4.3
We

There are

a sequence

x \\,

inputs

(e.g.,

(see,for

2h

pixels

example,

- 1
processors,
such
that
*\342\200\236,

Jt2,...,

suggestedmainly

and its

depth

The
Ts(n) = 0(n\\og2n).
is 0(log2Az),
so overall

for

network

applications,

in a

picture) and

the

[Uhr

1987]).

In this

Mi-smallest element. This

example

a parallel

the

and

algorithm,

is a

each associated
xt resides initially

image-processing

on a Tree

Element

Ath-Smallest

the

that the interconnection

assume

now

that

implies

in the

processors

Finding

leaves.

which

\"column\"

each

the same processors

circuit. (Notice that although
be
almost
The only type
will
have
to
columns,
connected.)
fully
they
is
and
the
needed are
a
in the circuit
comparison,
only processors
and two outputs.
two inputs

0(n log2a?)
can be used for all
comparatorswith

75(2)=1,
in

algorithms

complete

with

at

a node

leaf

where

binary tree with n=2h~l

in the tree. The input

/. Tree
the

leaves

for manipulating them

pipelining

(which

is described

to the

correspond

section, we considerthe problem

illustrates
the translation of a sequential

use of

is

machines have been

in

are hierarchical
of finding

the

algorithm

to

more

detail

in

Section 12.5).
for
the &th-smallest element
finding
sequential algorithm
describedin Section 6.5. We assume, for simplicity, that the elements are distinct. The
one. In each step, a random element x is chosenas the pivot.
is a probabilistic
algorithm
The rank of x is computed by comparing
x to all other elements and, according to
whether
the rank is smaller or greater
than
that are less than x or greater
ky the elements
than x are eliminated. The algorithm
when the rank of the pivot
is k. The
terminates
number
of
iterations
is
and
the
number
of
is
expected
O(logAz),
expected
comparisons
in each iteration
of the algorithm:
0(n). There are three different
(1) choosing a
phases
random
its rank, and (3) eliminating.
We first describe
efficient
element,
(2) computing
even further.
the parallelization
parallel implementations of each phase,then improve
a random element can be achieved
Choosing
by a tournament
arranged on the tree.
*
Each leaf sends its number
to its parent where the number
with the number
'competes\"
of its sibling leaf by flipping a coin. The winning
number
is then promoted up again,
and
the same process continues up the tree until the root chooses the overall
winner.
(This
works only
in the first iteration; we discuss later how
it work after
some
to make
elements
are eliminated.) The winning
so
number
the tree,
is then \"broadcast\" down
that all leaves can compareit with their number.
at
of the pivot is known
If the identity
all the leaves, they
can
all compare
their numbers to the pivot in one step. They then
send
a 1 (if their number is smaller than
to their
or equal
to the pivot) or 0 (otherwise)
of the pivot is the
number
of Is that are sent up. Summing n
parent node. The rank
numbers
can then broadcast the rank down the tree,
root
up the tree is easy to do. The
and
each
leaf can determine whether
be eliminated.
or not its number should
Overall,
there are four \"waves\"
tree to choose a pivot;
of communication
the
(1) up
per iteration:
the tree to broadcast it; (3) up the tree to compute its rank; and (4) down the
(2) down
tree to broadcast
the rank.
First,

we

recall

the

12.4

The problem is

for

Algorithms

397

Networks

Interconnection

are eliminated, the tournament

is no
of the tree, except for one, are
eliminated.
The remaining
element in that half will be promoted to the root without
will
then
be
chosen with probability
It
!/2, while other elements are chosen
competition.
with
much
We want to preserve
smaller
the uniform randomness of the
probabilities.
choice.
We can preserve it in the following
that
way. Processors associated with values
have been eliminated in previous
rounds
send up a nil value.
Any element always wins
a nil value. Every competing elementhas an associated
counter, which is initially
against
1. The counter
of (real) \"opponents\" that
indicates the number
in the part
participated
number
in
of the tournament
this
element
the
of
elements
the
subtree
that
(i.e.,
involving
When an element
have not yet been eliminated).
wins a game at some
node in the tree, it
is promoted
element's
counter is added to its counter.
upward, and the losing
Every
with a biased coin according to the counters
of the competing
game is now
played
if x wins its first game (say, against y) and z advances
elements. For example,
by default,
then
counter
is 2 and z's counter
is 1. If jc now plays against z, then
the game
;t's
is
2:1
z
a
of
lA
of
with
bias
toward
x.
has
this
Overall,
winning
played
probability
game,
both x and y have
of lh \342\200\242
% = y/j of winning
both their games. This process
and
probability
that the final choice is uniformly
selected
the participating elements.
guarantees
among
longer fair.

In

Complexity
times
way

the

as

expected
A Sketch

number

The

the height

that,

extreme

the

of

the

sequential

tree.

some

of (parallel)
Since

algorithm,

running time is thus

of an

after

elements

case, all elements

one-half

in

steps involved

the

expected

in

each

phase

eliminates elementsin

this algorithm

number

of phases is

still

is equal to
exactly

four

the same

O(logAz).

The

O (log2Az).

Improved Algorithm

Most of the information

tree is a major bottleneckin the computation.
and all leaves are at
two
connections
must pass through the root,
but the root has only
we should at least make the
h - 1 from it. If we cannot improve the connections,
distance
we just described, the root and the leaves are
root
as busy as possible. In the algorithm
this
active for one step and then remain idle for about 2h steps. We can improve
new
We
do
all
all
the
time.
that
by initiating
algorithm
by making
processors busy
are completed. All those
iterations
iterations in every
step even before the
previous
The reason this
will proceed in a pipeline
fashion up and down
the tree.
iterations
It takes 2h-2
the running time is the following.
steps to select one
pipeline
improves
If
1 steps to reach the root, and h 1 steps for the root to broadcast the result).
(h
pivot
one
in the second step and
run
it in parallel to (but
we start another
tournament
step
h
can
in 2/z -1 steps. We
select
behind) the first one, then we can select two pivots
=
All
be
eliminate
elements.
in
3h-2
can
used
to
those
pivots
0(\\ogn) steps.
pivots
the search
Thus, instead of cutting
space by about half with one pivot, we cut it to about
h pivots,
and we do it without
of its original size with
\\l(h
+1)
significantly
spending
The
leaves start a new
more time. We can also interleave the different
phases.
the k smallest
is found) and the tournament
the rest
at each step (until
tournament
pushes
The

of the

root

of the

computation.

Parallel

398

Algorithms

12.1

D Example
An

of this

example

number.

smallest

so we

(In

shown.

The

numbers

outside

numbers

first

the

numbers.)

inside

the

nodes

one

second

is 5

(which

proceeds from

8,

we

and

is equal

number

For each

are

(Fig. 12.12e),and

from

obtained

4, 3,

then

1, and

fourth

element

is broadcast

case,

one

pivot.

of

nodes

the

this

we do

(which

need to run

by
not

process

&th-smallest

The expected

overall.

O(loglogAz)

second phase,
12.12(j),

Fig.

any

of this fact, as well as

slight improvements, can be found
The proof

in

should

be

\342\226\

to 0{

is reduced

time

running

graph

ProcessorP [/, j ] is
P[U

J;

+ 1],

P[i +

to P[0,

addition

the

of indices

and

with

elegant

without

the

The

and

Given

Problem

0] =

right

A
\302\243 L0,

two

j] resideat

matrix

regular

n-\\

C[0,

has

).

and some

results,

[1987].

the

column

same

nxn mesh (grid).

and it is connected
to P [i - 1,j ],
the boundaries
of the mesh are
- 1,0],
to P [0, n - 1] and
P[n

two-dimensional
y,

1]. We

it

In

other
we

all additions

words,

is more

present

running time (to

within

and

symmetric
a constant)

around.

wrap

the

that

and

is a

and

j] and B[i,
resides atP[i,j].

such

at row /
j-

this assumption;

A[i,

We use the

have

now

assume that
for example, P[0, 0] is connected
P[i,

simulation

log

the Mesh

P[\\,
0] (see Fig. 12.13).
are done modulo n. The algorithm

1]

subtractions

we consider

algorithm,

and Manber

in Greenberg

processor

l,y],

so that,

around,

wrapped
in

of the

details

the

Matrix Multiplication on

interconnection

The

In

element);

elements

log

12.4.4

is 3

12.12(g),

terminates.

the algorithm

show),

iterations (or eliminate

should
be run several times and
element is found.
set of

another

pivot

12.12(1). Once the

at Fig.

root

the

the

The regular algorithm requires O(logAz)

to eliminate
elements
steps
Since we can generateO (logn)pivots
at about
the same time, we savea

Complexity

factor

to

there was no

general, however,
eliminated until the

is discovered

element

smallest

value,
and

In Fig.

4.

fourth-

nodes is

below,
chosen

first

top

right

of all

contents

the

to

left

looking for the

to that number's

down. The

are broadcast

that

step,

ones

the

are

on they
and from then
start the
(3) arrives at the leaves,
the rank. The fact that 3 has rank 3 is discovered in

pivot

and the

with

of each

and

are those

1 to

is computing

which

this

from

the rank

case,

12.12

Fig.

ranks

the nodes

(Fig. 12.12d),the

in

numbers

the

this

show both

not

do

process is given

elementsare

down). The

numbers

nxn
P[i,

matrices

y], compute

A and

B, such

C=A-B,

such

that
that

multiplication
algorithm. The problem is
are at the right
at the right
place

k ] \342\200\242
B [*, 0], which is

the

inner

product

of the

first

row

initially
C[/,

j]

to move the data

time. Consider
of A

and

the

first

\342\226\240>o

Ui

*\302\251\342\200\2240\342\200\224G-

(EX-XE)
o

-U

-U

GX0O

G>0XE)

-U

4t

-uo-u

-uu^-u

U,

L*

L*

(00HE)

GHM)

G-XE

*k.

(EXEXE)

(EXEXE)
l*

<-*

-U

<-*

-uo-u

-UO-U

GXBXD

GXEXE)

-0\342\200\2240\342\200\224\342\202\254)-

(EX-XE)

GW3
*k.

L*

0M^)
&\342\202\254XE)

L*

(EXEXE)

-0\342\200\224GM\302\251\"-

EX\302\260XE)
U,
\342\226\240U

(EX*XE)
V*j
<-\302\273\302\273

l*

GXEXE)

GXXEX^E

(EX\302\260X3

\342\226\240uu^-u

GKEKE)

Uj
Uj
<-\302\273\302\273

GXXEXXE)
GKEKE)

Ui

Uj

GHHMH)

GK5KH)

<-*

GH00)

GX00)

(EX\302\260XE)

Ui

-0\342\200\2240\342\200\2240*

\302\253-.
\342\226\240U

00H3

Ui

GXEH3

00HE)

00X3

<-*

&\342\202\254X3
-U
O

-0\342\200\2240\342\

(EX3-Q
-U

Parallel

400

Algorithms

Ck

0-

Q^

Ck

-0-

6-

-O-

6^-

-<y-

Figure 12.13 A
column of
the

final

B.

We

and, at the

time,

would

This can

result).

like

-6'

-o^

two-dimensional

to be computed

C[0,0]

be done by

same time, shifting

has /4[0,0] and

by

first column

of B

of A

up

left one

the

to

one

step

have
step at a
a time. At the

also must

(which

P[0,0]

first row

the

shifting

the

mesh.

wrap-around

at

their product; in the second

the
and
to the
1] (from
B[\\,
0] (from below), adds their
right)
product
so on. The value of C[0,0] will be computed
after n steps.
The problem is that we need all processors to do the same thing, and they all need
to share the data. We have to arrange for the data to move such that not only P[0, 0] gets
what
it needs,
but every other processor does too. The
trick
is to rearrange the data
movements
in such a way that, throughout
the execution of the algorithm,
all processors
have
of
two
whose
The
is
the
initial
distribution
numbers
need.
always
key
product they
the
data.
We will rearrange the data such that each processor P[i, j] has A[i, i+j]
and
are modulo
then
afterward
n). If we can do that,
B[i+j> j] (again, all additions
every
of simultaneous
which bring A[i, i+j + k] and
row and column
shifts,
step will consist
+ k, y], 1 <k <az, to P[i, y], which is exactly what
We can achieve
P[U j] needs.
B[i+j
the
initial
arrangement
j steps
by shifting the ith row i steps to the left and the yth column
The precise algorithm is given in Fig
up, and by doing that for all rows and columns.
12.14.
the initial
data movement. The left side shows the data at
Figure 12.15 illustrates
the beginning,
and the right side shows the data after the initial shifts are performed.
first

step, P[0,0]
step, it gets A[0y
and
partial result,

The initial

Complexity
the

when

number

computations are involved

and

shifts,

for

row

of shifts is
each

so

\302\243[0,0],

shifts

more
far.

processor.

take nil

than

nll)\\

Then,
All

it computes

and

these

the

parallel steps (we can

same is

there are
steps

steps,

can be

true

for

involving

done

in

shift

the column

to the

right
shifts. No

both computations
The overall

parallel.

to
running time is thus O (n). The efficiency is O (1) if we compare the parallel algorithm
we
if
the regular 0(n3)
less
sequential algorithm. The efficiency is (asymptotically)
to asymptotically faster algorithms (e.g., Strassen's
compare the parallel
algorithm
algorithm).

12.4

Matrix

Algorithm

begin
for all rows

matrices).
of A and B).

(nxn
product

in parallel

do

shift row i of A

{ namely,

401

(A, B)

.Multiplication

Input: A and B
Output: C (the

Networks

for Interconnection

Algorithms

A [/,

perform

for all columns

left i steps

to the

j] :=A [/, (J + \\)mod

n ]

i times

in parallel

do

j steps
shift column j ofB upwards
{ namely, perform B [/, j]:=B [(/+
{thedata is now in the proper starting
for all i and j do in parallel

j times }

ny j]

\\)mod

positions

C[iJ]:=A[i,j]B[iJ];

for k := 1 to n-\\
for all i and j

do

do in

parallel

A[i,j]:=A[i,

(j+\\)modn]

B[iJ]:=B[(i

+ \\)modnJ] ;

C[iJ]:=C[iJ]+A[iJ]B[iJ]
end

12.14

Figure

012

013

0 14

b\\i

^13

b\\4

a2\\

a 22

0 23

b2\\

b22

^23

031

0 32

&31

0 13

0 14

011

b32

^43

bXA

24

021

022

b24

&31

bA2

^13

^24

0 33

0 34

0 34

031

0 32

033

b32

&33

bM

b4[

b\\2

^23

bM

<*41

042

043

044

041

042

043

044

bAX

b42

^43

^44

bn

^22

&33

bu

Figure 12.15The

The

we

examples

heavily

approach
powerful

an

have

algorithm

24

data

initial

in a

Routing

algorithm to

so

MatrixMultiplication.

012
^21
0 23

011

12.4.5

Algorithm

placement

and

the

result

of the

Hypercube

seen so
that

far

illustrate

runs on

an

the

interconnection

difficulty

of

the

architecture,

even

adapting

network.

then programming becomes very

to designing algorithms for interconnection networks
these primitives on the network,
to implement
primitives,
on

shifts.

initial

If algorithms
difficult.
is

to

then

a simple

depend
Another

define some
to design the

Parallel

402

algorithms

these

using

translate

to

network

Algorithms

We have to

primitives.

all algorithms

implement only
these

use

that

the

on another

primitives

The problem

primitives.

this

with

After all, there are major

differences
primitives.
approach
to
find
that
are
and
we
cannot
the
between
primitives
good for all
hope
topologies,
that
emulate
one
network
is
to
Yet
another
design algorithms
using
approach
topologies.
between
the two
will
an easy translation of algorithms
allow
This technique
another.
In this section, we
networks, but the translation
algorithm.
may not yield an efficient
is the

a general
routing
networks as though

discuss

briefly

interconnection

We assumea hypercube
In an EREW

topologies).

variable

the

scheme
shared

that

memory

connection

(similar

xh

algorithm

to design

for

algorithms

is available.

have been designedfor other

processors can in one
is
P(
responsible for one
processor

schemes

shared-memory algorithm,

different

the

read

any

or write

conflicts, a step in
way to look at

Another

variables.

that processor P{ accesses variable jtCT(/).

all
a, such
(Not
in
all
the
but
the
worst
case
we
can
be
variables
assume
time,
accessing
processors may
of the EREW algorithm,
and try to
that
do.) We concentrate on one (arbitrary)
they
step
emulate it using several steps of a hypercube. We now have a routing problem. We
that each processor P, on the hypercube
wants to send a messageto processor
assume
All messages
are sent at the same time, and our goal is
Pod) (which holds variable jcct(/)).
to route all of them through the hypercube
We assume that each edge of the
quickly.
is not only to
hypercube can deliveronly one message at a time. Therefore,the problem
find
short
between sources and destinations,
routes
but also to minimize the conflicts
arising from trying to use the same
edge at the same time. If two messages
try to use the
same
at
the
same
one
of
them
be
We
must
buffered.
also
want
to minimize
time,
edge
a

the

is as

step

buffer

us

allows

arbitrary variables. Suppose that each

not have
Since an EREW algorithm
does
consists of processors accessingdistinct

access

step

the

of

definition

permutation

requirements.

space

The best
generally

not

routing

possible

permutation is distributed
that works well on the
[1982].

The key

first phase,
(independently
describe

the

to analyze
among
average.

on the

depends

obviously

following

idea is to use randomization.

the best

Therefore, we are

the processors.
The

find

The

routing

consists

for

a scheme

suggestedby
of

two

the

since

routing,

looking

routing scheme was

it is

However,

permutation.

particular

to

permutation

phases.

Valiant

In the

each processorP, sendsits message

to a random processor, chosenuniformly
of the destination) among all other processors. (In a moment,
we

how

to select

this

random

processor

and how to

route

the

message

to it.)

In

will

the

the message is sent along a shortest

the random processor
route
between
that
received
the message and the final destination.
\342\200\22
All messages are sent in the same way, so we can concentrateon one message
\342\200\242
\342\200\242
\342\200\242
that from / to j. Let the binary
be
of
/
say,
possible
bd (with
bx b2
representation
'
we need
and that of j be c \\ c2 \342\200\224
padding of 0s at the beginning),
cd. In the first phase,
to find a random processor r. We find it by considering the bits representing
/ one by
to
the
one, and deciding
to
route
Vi
whether
with
corresponding
randomly
probability
we
or not. For example,considerFig. 12.16,and let /=000
and 7 = 110. First,
neighbor
to
send
not
choose
or
not
we
whether
that
decide
the
to
100.
randomly
Suppose
message
to send it to 100. Then, we consider the second
bit and decide whether
to change
it; in
second

phase,

12.4

403

Networks

for Interconnection

Algorithms

000

Figure 12.16An
decide

other words, we
the third bit. In
lines)

through

passes

next choice
much faster

without
than

it.

or

whether

not to

If we

decide

for

waiting

the

in

a 3-cube.

to 010.

message

processor is 011 and the

not to send a message,we

the next

we

round;

When the

passing.

message

send

routing

the random

12.16,

Fig.

of random

example

choice

consider

we

Finally,

route

(marked

in

is
the

is done.

routing

more
are
edge

available.

becomes

is not

It

of

probability

chosen

the

that local computation

concerningthe last bit is made,
assume

Each choiceis made locally.

have
Since all processors send messagesat the same time, a processor may
same
than one message waiting
to be sent along the
In this case, the messages
edge.
the
put in a queue (in a random order, if there is more than one) and are sent when
random

bold

make

immediately

difficult
being

processor

to see that
as

chosen

number

every
the

random

k (in

the

destination.

r to j is done deterministically.
'''
The

Suppose

range)

appropriate

The routing from

that /* and j differ

has the
the
in

same

random
t bits

with

and
the indices k{ <k2<
<kt.
messageis sent by changing bit A: i, then bit /:2\302\273
will be sent first from Oil to 111 (since
so on. In the example of Fig. 12.16,
the
message
the
destination
110 has a 1 in the first bit), and then
to
110 (the second bit is not

changed).

This routing
shortest routes (in

scheme
the

example

is simple
above,

to implement.
the route took

The length of each route,

with
of these routes is that,

The routes
four

edges

are

versus

not

the
necessarily
two edges of the

too large; it is at most 2d. The

main
have
few conflicts;
very high probability, they
property
to
be
is
in
Similar
schemes
the
O
thus,
(d) steps.
expected
completed
permutation
routing
based
on
have been suggested for other topologies. There are also efficient
algorithms,
to
for
interconnection
such schemes, to map
shared-memory
algorithms
algorithms
shortest

networks

route).

automatically.

however,

is not

Parallel Algorithms

404

12.5

Computation

Systolic

called
an assembly line. The processors,usually
resembles
but
not
as
in
one
a
are
(usually,
always,
very regular way
arranged
processing
in
a
them
fashion.
Each
move
and
the
data
dimensional
rhythmic
or two
through
arrays),
beat,
simple
operations on the data it received in the previous
very
processorperforms
contains very limited (if
Each
to the next \"station.\"
and moves the result(s)
processor
one variable at a time, rather than being
is \"pushed\"
Most of the input
memory.
any)
of systolic
architecture
is
loaded all at once to some memory locations. The advantage
and
is
and
in
terms
of
the
hardware
both
(which
simple),
speed
specialized
efficiency,
As in assembly lines, the key is to
(the number of memory access cyclesis minimized).
that
is
avoid the need to fetch tools, material, and so on, during the work. Everything
the
The
drawback
is
the
arrives
on
line.
big
inflexibility
required to perform an operation
are efficient
of such schemes. Systolicarchitectures
only for certain algorithms. We will
A

architecture

systolic

elements,

examples of systolicalgorithms.

see three

Matrix-Vector

12.5.1

We

with

start

Multiplication

a straightforward

example,

use

then

develop more

it to

complicated

algorithms.

Find the
Problem
column vector b of sizen.
The

xn

of an m

x-Ab

product

A with

matrix

to the partial
(stations), such that P,- is responsible for adding
br The data movements and the actions of each processor for
the
case
of n-m-A
b resides
in the
are illustrated
in Fig.
12.17. We assume that
The results are accumulated
into
them regularly).
appropriate processors (or is pumped
as they move from left to right through the processors. The x{s are initially
0. In the first
P \\
with
move
closer.
P
move
into
and
all
the
other
step, x! (= 0) together
a,,
\\,
inputs
\342\200\242
receives
and
moves
the
to
its
second
result
In
the
computes
jc, +0,,
fc,,
step,P2
right.
There

n processors

are

product

the

term

x\\=a\\\\'b\\from
moves

the

result

involving

the

left,

to its

together

right,

and

with

so

an

on.

from above;

In each

/-1

result,

which is equal to

above,

and the appropriate

or
it

from

clearly

below.

has the

\302\243
ajkbk,

element of b from

P{ computes

right value.

The main problem

the left,

from

x+ajjb,

The whole

and

passes
product

the

it

X\\+a\\2'bi,

computes

step, processorP, receives a

either

it to

local

its

memory
right.

is computed

of the

entry

appropriate

in

(as

it

matrix

is in

When

jt, leaves

m +n

steps.

and
partial
from

the figure)
the array,

Each
is the data movement.
designing
systolic
algorithms
data element must be at the right place at the right time. The only trick in this example
was to introduce
so that column / of the matrix arrives at P, in step /. This
delays
element
of A was used only
is
since
each
When the same value is
once.
example simple,
in

12.5 SystolicComputation

405

'44

-X\"i

0 34

<*42

0 33

0 24

<*41

0 32

0 23

0 14

031

a 22

013

021

012

.
*2

043

\302\253

J b3

J b2

*i

\342\226\240
W
.

J b4

>

OUTPUT

-+ *out =

as is

movement,

12.5.2
Let

times

several

jc

usually

in the

The Convolution

=jc i,

jc2,

vector

and
\342\200\224,xn

Problem

The

The

(as is

illustrated

is

called

w =

+ ab

12.17 Matrix-vector multiplication.

Figure
used

xin

the

it is much

case),

more complicatedto

design

its

next example.

Problem

w,, w>2....,wk

Compute y\\,yi,

the convolution

two

be

sequences

sucn

...,yn+\\-k*

of x

and

w.

We

of real numbers

tnat

J/ =

can reduce

with

w\\xt

the

k <n.

convolution

406

Parallel

Algorithms

problem to a matrix-vector product as follows.

I

-V,

X2
*v3

X2

A'3

*3

' '

X4

' '

x5

x4

*k
Xk

y\\

+ \\

xk+2

w,

yi

w2

^3

w3

(12.1)

Wk_

xn + \\-k

xn+2-k

' '

xn+l-k

yn

+ \\-k

xn

the matrix
the systolic algorithm
for this problem
in
by simply substituting
in Fig. 12.18.
Notice that each jt, is
(12.1) with the matrix of Fig. 12.17. This is shown
across
the array (except for the first k-\\
needed at the same time
jt,s which are not
we show
another solution
needed
Thus, a broadcast line is needed. Next,
everywhere).
with
no
broadcast.
to the convolution
problem
The processorsin Fig. 12.18 receive several inputs, but send only one output.
We
that
from
two
send
to
use
receive
directions
and
two
now
processors
output
inputs
directions. The idea is to move the vector x from left to right and the vector w from right
to left. The result
in the
We have to design this movement
y is accumulated
processors.
such that the appropriate values of w and x meet.
The problem with moving
the two
vectors in opposite
directions
is that they move twice as fast toward
each other.
As a
each
element
of
will
x
miss
half
of
the
elements
of
and
vice
versa.
The
solution
result,
w,
is to move
the vectors at half the speed. The left input will be **jc 1, nothing,
jt2,
in Fig. 12.19 (the
is illustrated
nothing,\" and so on, and the same for w. This solution
We

bullets

can obtain

correspond

to empty

slots).

x^

x^

x^

*3

*3

x3

X2

X2

x^

x\\

Wi

w->i.

'

1f

W-t

Wa

OUTPUT

Figure

12.18

Convolution

using

broadcast.

12.5

407

Computation

Systolic

w\\
\342\200\242

y*

^3

*2

1\342\200\224i\342\200\224' 1
I
I

.-*-_

\342\200\242

1
I
I

,-*-_ '

yi

y\\

X\\

'
I
I

I
I

,-*-- '

,_*__'
OUTPUT

y=y

Win

Wout

y
out
X'm

We leave
leaves Ph

data

12.5.3

12.19 Convolution using

the reader

to

value

of

is, of course,that

A =a\\

a2

an

find the minimal edit

the

and

The

of

right

the

in

Fig.

it

P,-

collects

can

move

array.
the value

out of

the

of
array

When

wk

through

the

yh

drawback to moving
as long.

The main

12.19.

takes twice

'''

bm

distance betweenA

proceeds

algorithm

and

computation

B\342\200\224b\\
b2

dynamic programming algorithm

first discuss general ways
to
solution.
bottom

each

that

verify

a bidirectional

\"in

data

at

Comparisons

Sequence
'''

to

y,- is computed,

the array

below

illustrated

path

half speed

Let

it

final

the

~ *in

*out
Wout

Figure

win-xin

table.

was

be
and

presented

parallelize

of characters. Our goal is to

Section 6.8 and Section
A
10.2).
in Section 6.8 to solve this problem.
We

two

strings

B (see

this algorithm,

then

by filling a table, such that

The value of the ijth entry

present

the

solution

depends

a systolic

array

appears at the
on three entries

l)th entries. Supposefirst that we

the
table
(see Fig. 12.20). Each processorP,j
\342\200\224
from
three
above
to
the
left
of it, and diagonally from it \342\200\22
it,
gets input
processors
and
sends output to three processors similarly.
Each
character
from B moves down
the
A moves
to the right.
Let's
see which of the algorithm's
table, and each character from
in Fig. 12.20 indicate the
The labels on the squares
steps can be performed in parallel.
level
of this square, such that
with the same level can be computed
in parallel.
squares
table in O (n +m) steps by computing
We can compute the whole
one level at a time. It
is not necessary,
The same processors that
handled
however, to have nm processors.
/ + 1, level / +2,
level
and so on. Therefore, min(\302\253, m) processors
level / can also handle
are sufficient. The problem is to arrange the data movement.
We use 2n processors arranged in a oneFor simplicity, we assume that n=m.
We have to ensure that every pair of
bidirectional array
dimensional
(see
Fig. 12.21).
characters a{ and bj meets at some processor at some time, and that the three values
l)yth, (/- \\)(j- l)th,
have nm processors arranged according to
surrounding

it,

the (/-

and

/(/-

408

Parallel

Algorithms

Figure 12.20Sequence
0
/

by a

comparisons

two dimensional array.

0
a

c
2

a
7

a
a

>

a
2

Figure12.21Sequence
needed to process these

2
a
a

3
3
3

comparisons

by a

2
1

3
a

one-dimensional

array.

This is achieved by
to the processor.
one
both
at
half
to
left,
right,
speed, and having
right
is
which
cell
used
use
half
for
We
the
connection.
memory
speed
per processor
diagonal
not only to ensure that the corresponding charactersmeet,
costs
but also to allow the
slots between two adjacent characters. For example,
so far to use the empty
let
computed
A =caa and B =aba (seealso Section
The first row (and column)
in the matrix is
10.2.3).
moving

A from

left to

are

characters
B

from

known

Bibliographic Notesand Further

123 (corresponding
the right side will

of all
are

in the

all

from

that

and

to 0.

characters

two

When

the two

numbers

to compute

needed

information

the

3,

beat,

next

or deleting those characters). The input from

the left side will be c \\a 2a3. The memories

of inserting

costs

the

a \\b2a

are initialized

processors

compared;

has

to
be

409

Reading

the

a{ and

following

bf

entry

corresponding

B[\\..j- 1] comes from

at, say, Pk, they

at Pk, and Pk now
in the matrix. The

meet

meet

them

the minimal cost of

and
comes from Pk+\\,
the minimal
cost of matching
A 11../ - 1 ] to B [ 1 .J - 1 ] comes
from Pk 's memory. This new
will
then be moved
entry
both
12.21
left and right in the next step, and so on. Figure
shows
four different steps:
The first row is the initial step, the second row is the third step, the third row is the fourth
and costs are shown
step. The values of the memories
step, and the last row is the seventh
leaves the last processor (in either
the step. When the last character
side), the
before
number that follows
it is the minimal edit cost.
of matching

cost

minimal

12.6

Summary

Since parallel
algorithms,

is a powerful

Pk-\\,

to B\\\\..j]

/\\[1../-1]

matching

A [1../] to

are

algorithms

we have to make
building

block.

even

more

good use of
It was

complicated to design

building

blocks.

than

sequential

The parallel-prefix

even suggestedas a primitive

machine

paradigm

operation

(for

is true for the

same
example, Blelloch [ 1987]). The
routing
through
permutations
which
of the suggested parallel architectures
(Section 12.4.5). It is still too early to know
in the future.
it is important to identify
Therefore,
(if any) will dominate
design
to
models.
We
have
seen four techniques
are
common
that
(and there
many
techniques
and other operations
that we have not seen): doubling
are, of course, others
(list
ranking
divide and conquer (addition,
on linked
lists),
parallel
prefix, sorting), pipelining
parallel
in general), and the Eulefs
tour
on the tree, and
A-smallest
systolic
computation
(finding
is
for
a
of
and
tree
Induction,
(which
again,
variety
graph
algorithms).
technique
helpful
plays a major role.

Bibliographic

Notes

and Further

Reading

of
as is evidenced by the number
rapidly,
computing is expanding
in the last few years. Books describing
books that have appeared
parallel
computing
and Parberry
include Akl [1985], which
is devoted
to parallel sorting;
[1986]
Hong
include
models of parallel computation
and their relationships;
theoretical
[1987), which
and Rytter [1988], which
are
books on parallel
Quinn [1987] and Gibbons
general
and
and
Malek
and Almasi and
[1984],
[1987],
Briggs
algorithms; Hwang
Lipovski
which
different
describe
Gottlieb
and
architectures
[1989],
parallel
parallel software;
with
which
deals
Ullman
and
VLSI;
Siewiorek,
[1984],
[ 1987],
mainly
Gehringer,
Segall
Cm
which describes experience with
describes
; Fox et al. [1987], which
experiments
which includes models, architectures,
with the hypercube; Reed and Fujimoto
[1987],
Bertsekas
and
which deals mainly
Tsitsiklis
several
detailed
and
[1989],
applications;
and
Misra
and optimization methods; and others.
with
numerical
[ 1988] present
Chandy
A
for
on
a general
(and
programs.
writing
parallel
methodology
sequential)
survey
The

area

of parallel

Parallel Algorithms

410

parallel graph

algorithms

by Moitra

algorithms

on parallel sorting.
Instruction

is given
and Iyengar

The

[1975] is

Valiant

from

these

for

algorithms

Borodin

an

machines

bibliography

to Single/Multiplealgorithms by circuits

Modeling

[1966].

Flynn

general parallel

a large

according

material

appear

in

Section

Shiloach

and

Vishkin

[1981],

ranking appear

in

and

merging,

finding,

Other parallel

is taken.

12.3.1.

in

for list

algorithms
Hopcroft [1985]. Additional
Snir [1985] (where parallel prefix

maximum

on parallel

work

early

problems

and on

[1984],

[1977].

of the

some

which

to

is due

is described, for example,in

Deo

and

of

classification

Single/Multiple-Data

sorting,

by Quinn

[1986]. Richards [1986]contains

and

in Borodin

and

Rudolph,

and

Kruskal,

[1987] (where a randomized

in
Vishkin
and
Cole
and
for
is
[1986] (where a general
presented),
technique
algorithm
to
deterministic
ones
is
randomized parallel algorithms
presented). Cole
converting
of mergesort which
has
an elegant
T(n, n) = 0(\\ogn)
[1988]presents
parallelization
the CREW and EREW models (although
the
CREW
both
is much
under
algorithm
involves
smaller
and
Goodrich
and
includes
Attalah,
Cole,
constants).
[1987]
simpler
in
to
several other applications of Cole's technique,
particular
parallel geometric
is

in Vishkin

used),

algorithms.

for

Algorithms

technique,

[1985]; see also Kruskal,

computing is discussed

Vishkin
parallel

Exercise 12.24 is from

and Dwork [1982].
It was

an

depth O(logAz).

(even though
Reif and

Ajtai,

Valiant

parallel

including

applications,

Steele

and

is

tree

multiplication

networks
multiplication

[1986].

on a

Nassimi,

by

are

[1985], and

given

Fox

Batcher

by

with

networks

exist

Section

for

it

3.2).

are

the fan-smallest

The

running

The algorithm for

and

Experimental
et al. [1988].

other

in

time is
matrix

interconnection

results

The

given

element

appeared

algorithm

whose

algorithm

on a hypercube

[1981]).

algorithms

finding

[1988].

such

Cook

O(logAz) sort.

(the regular

[1987]

in

various

sorting

[1987].

from

networks

\342\200\224
see

O0(\\ogn)

for

algorithm

to run

that

a randomized

with

Frederickson

Sahni

and

[1983] proved
impractical

A deterministic

be adapted

a hypercube

an

Manber

and

[1983]).

case is given

mesh can

(Dekel,
on

Szemeredi

by Hillis
The improved

Rodeh

and

worst

the

long

there exist

for

conquer

Stout

[1987]; Exercise 12.15is

whether

time

was definitely
exhibit a network

to Greenberg

due

Francez,

0(log3Az)in

Komlos,

and

and

in

mergesort, were introduced

the odd-even

is described

machine

Shrira,

and

Snir [1986]. Divide

Zorat [1983], and

and Snir

Rudolph,

problem

[1987]

and

Horowitz

for a

network

and

Rudolph,
in

including

open

their

connection

in Hillis

Kruskal,

networks,

Sorting

[1968].

on a

problem

and lower
[1980]. Fich [1983] providesupper
of gates in parallel-prefix
circuits.
Beame, Cook, and Hoover
describe parallel algorithms for division and related
The Euler's tour
problems.
as well as the algorithms
in
Section
are
from
and
12.3.5,
presented
Tarjan

addition, appeared in
bounds for the number
[1986]

the parallel-prefix
and Fischer
Ladner

routing

for

matrix

scheme

and Brebner [1981] (the

former
was actually developed earlier). (Valiant
to choose the order of
[1982]
suggested
the bits at random;
it was shown in Valiant
and Brebner
[1981] that the bits can be
considered in order, which is how we described it in Section 12.4.5.) Similar schemes
presented

in Section

12.4.5 is

from

Valiant

[1982]

and Valiant

Drill Exercises 411

networks

on

more

For

[1982].

Upfal

and

Karlin

in

appear

[1987].

Schemes
for emulating shared memory
in
and
and
Ranade
[1986],
[1987],
Upfal
Wigderson
Upfal
networks see, for example, Feng
interconnection
[1981],
Siegel

[1982] and

in Aleliunas

appear

[1985], or Ullman [1984].

More on
[1980],

and Kung

algorithm

(Section

array

The

difficult

more

significantly

Nevertheless, Megiddo
Cole

by

12.1

Draw

12.2

Design an

designing

parallel

technique to design
An

algorithms.

corresponding to the

to add k binary
k) using nk processors.

Draw

the circuit

12.4

Prove

that

algorithm

12.5

Prove

that

the rules

12.6

Complete
in

Figure

that corresponds to

the details

a tree

parallel
numbers,

algorithm

n log

12.3

some types

of

of this technique

improvement

algorithm

List Rank (Fig.

for constructing
of the

addition

each with

Parallel

12.3)requires

an Euler's

doubling algorithm

discussed

only

Section

the EREW
in

12.3.1.

The running time

Prefix 2 of Section

tour given
to find

bits.

in

Section

should be

12.3.3.

model.
12.3.5 are

the preorder numbering

correct.
of all

vertices

(Section 12.3.5).
is an example of a graph
an interconnection network
representing
in this chapter. Identify
and label the nodes appropriately.
the network,

12.22

discussed

that

we

uses
the proof of correctnessof the odd-even
sort. The proof
transposition
and shows that, when an additional processor (and input) is added, the number of
= 2, uses
steps is incremented
by 1. The base case, n
exactly one comparison. Hence,the
total number of steps seems to be n - 1 and not n. Show an example
that requires n steps,
and discuss why the proof does not show that n - 1 steps are sufficient.

Consider

induction,

12.9 a.

Draw

the full

circuit for merging

b.

Draw

the full

circuit for sorting

c. Draw the
12.10

usually

algorithms.

sequential

corresponding

an ingenious

found

algorithms is

parallel

designing
the

[1984].

the circuit

O(log

12.8

devoted to systolicarrays.

Exercises

Drill

12.7

[1983]has

Section

in

and

difficult,

than

by using

algorithms

sequential

is

in parallel\"

\"Thinking

was given

comparisons

Jagadish

of Computer is

issue

1987

July

in

Kung and Leiserson

multiplication
using Winograd's
and Kailath
[1989]. The systolic
and
12.5.3 is from Lipton
Lopresti
[1979],

Kung

matrix

for

are presented

9.5.1)

in

found

[1982]. Systolic arrays

for sequence

algorithm

[1985].

can be

computing

systolic

Complete

fuli

circuit

Fig. 12.21 by

two

of size

8.

16 inputs.

for sorting
showing

sequences

10 inputs.

all the

steps until

the two

sequences are separated.

412

Parallel Algorithms

12.11 Design

multiplication
algorithm for matrix
O (n3 / log n) processors.

an CREW

12.12Modify

[\\..n

array

an

array

an

b.

an EREW

12.14 Parallelize

EREW

We want

of values.

to copy

n)

and

with T(n, n) = 0

algorithm

algorithm

with

efficiency

(log

0(1)

rule to evaluate a polynomial

Horner's

O (log

time

n by n matrices

the

A [1]

value

to

all

other

locations

in

broadcast).

(i.e.,

a. Design
Design

two

multiply

the maximum
(Section 12.3.2) to work in the case
algorithm for finding
are not distinct. The parallel
time should still be O (1).
running

numbers
] be

that can

using

CRCW

the
the

where

12.13 Let A

n)

O (log

time

the

network.

interconnection

Exercises

Creative

in

An example of an

12.22

Figure

efficiency

n) for

and

this problem.

running

of degree n

time
under

O (log

n).

the EREW

model

in

0(1).

12.15 The

is the or function
of all of them.
of n Boolean variables. The output
is 1. It is
the output is 1 if and only if at least one of the variables
the or of n variables in flog2\302\253l steps
straightforward to compute
by an EREW algorithm
with
But
n processors.
It seems fairly
obvious
that doing better than |\"log2\302\253lis impossible.
this is not the case. Design an EREW algorithm whose running
time is less than
flog2\302\253l\342\20
You can assume an unlimited
is not
number of processors and space. (This algorithm
are not always so obvious.)
practical, but it shows that simple \"obvious\" lower bounds
In

12.16Solve

input

other

is an array

words,

Exercise

model.

12.15 (computing

or of n

Boolean

variables)

in time

O(l)

under

the

CRCW

413

Creative Exercises

12.17Draw

with 0(n

a circuit

O (log A7),

that the

such

12.19

Let

in time O

sequence
12.20

(log n).

assume

can

You

to

be a
x\342\200\236

X\\,x2

Design a CRCW algorithm

12.23 Let xlyx

the

depth
than 2 and

parallel-prefix

Design a CREW algorithm to sort the

an unlimited number of processorsand space.

assume

Let S be a set of size n and let k > 1 be a constant.

= 0(\\).
maximum
in S with T(n, n]+]/k)

12.22 Let

for

an EREW algorithm

with

numbers.

distinct

can

You

design

A and B into
to merge two sorted arrays
an unlimited number of processorsand space.

Design a CRCW algorithm

0(1).
12.21

sequence of

be a
x\342\200\236

, x2

jcj

in exactly

circuit
of Exercise 12.17
with T(n, n) = 0 (log n).

problem

parallel-prefix problem
processing element is no more
one operation.

of every

fan-out

and

every processing element is involved

12.18 Use the

that solves the

vertices

\\ogn)

fan-in

one

sorted

Design a CRCWalgorithm

distinct) integers in
sequence of (not necessarily
in the sequence with
to find the maximum
T(n,

array

in time

to find

the

range 1 to n.
n) = 0(\\).

the

real
let S be a subset
of {1, 2,..., n\\, and let* be
numbers,
The set S partitions
the sequence
on real numbers.
into
contains jcj, jc2
element in S,
xh where / is the smallest
groups, such
the second group contains ,v/+1
smallest element of S, and so on
jc;, where j is the second
the rest of the elements in case n does not belong to S). The group
(the last group contains
parallel-prefix problem is to compute parallel prefixes separately for all groups. Show that
the group parallel-prefix problem can be solved with the same resourcesas are used
for the
parallel-prefix
regular
problem.
an

be a
2, ...,*\342\200\236

associative

12.24 The cross product

such

if i

that,

sequence of

binary operation
that the first group

words, C is a row
and B [j].) Design

A[0..n-\\]

two arrays

of

= k-m+r%

of an n x

representation

and

B[0..m-

0<r<m-l,

0<k<n-\\,

matrix

then

C[i]

whose

ij

parallel algorithm to compute

use O (nm) space. (For simplicity,

an EREW

O (mnlp + log/?). You

p is a power of 2.)

can

1] is an array C[0..nm1]
= A [k] -B[r]. (In other

product of A [i ]
processors and time
you can assume that n, m, or
is the

entry

C with p

is an array of records. Each record contains

for
some data (which is immaterial
input
this exercise) and a Boolean variable
The goal is to pack all records whose Mark
Mark.
is 1 into the beginning of the array.
value
(For example, these records may be selected for
some purpose and some more work has to be performed on them; it is better to have them in

12.25 The

a contiguous

12.26

area.) The

order

an EREW algorithm

Design

among

for this

the records

problem

that

with T(n,

are

moved

n) = 0 (log

should not be

changed.

n).

is a linked list such that all of its elements are stored

(in an arbitrary order) in an
an EREW
Let F[i] be a Boolean flag associated
with the elements. Design
another linked list consisting of only
the
algorithm with T(n, n) = 0(\\ogn) to construct
elements with F[i]=\\ in the same order they appear
in the original list. (This algorithm
is
in
Section
needed, for example, in computing
12.3.5.)
preorder

The

input

array.

12.27 The input

array.

of the
sufficient

is a

Design
original

linked list such

an

efficient

reverse the
allocated to hold

list (i.e.,

space is

that

parallel

its elements are stored

(in an arbitrary order) in an
a linked list in the opposite
order
algorithm to construct
You can assume that
list), without moving any element.
the extra pointers.

all of

414

12.28

Parallel Algorithms
an EREW algorithm
obtained by the Euler's tour

with

Design

the number of
the text, and it

n) = 0(\\ogn)

T(n,

method.

the forward

rank

to

edges

the

in

list

find
words, for each forward edge you should
is
in
the
list.
number
denoted
that
it
follow
forward
(This
by / (/, j) in
edges
associated with the Euler tour;
see Section
is needed to compute the preorder
other

In

12.3.5.)
without
the extra
12.29 Suppose that a tree T is given in the regular adjacency-list representation
a fast and efficient
EREW
the two copies of each edge. Design
pointers connecting
can
assume
that
have
as
much
work
these
You
to
construct
you
pointers.
spaceas
algorithm
want.

you

12.30

tree
by adjacency lists in the same way as described in Section
represented
EREW
algorithm to find, for each vertex /, the number
Design a T(n, n) = 0(\\ogn)
of / in the tree (i.e., the length
of the path to the root).
of ancestors

Let T be a
12.3.5.

12.31

circuit: The input

of the number of

a counter

Construct

representation

binary

of n binary

consists

Is in

the

numbers,

the

and

is the

output

input.

of the odd-even transposition

sort.
12.32 Consider the following
attempt to prove the correctness
We have seen that the maximum
element gets to the right position. Without it there are
n - 1 numbers.
But the maximum
number
adds at most one idle
to each number. Since
step

we

to spare, the

a step

have

terminates

algorithm

on time.

Discusswhy

this

proof

is not a

valid.

12.33

the

Consider

sum

the

have
computed

12.34

12.35

xx

in one

topology and assume that each leaf / has a

this model. At the end of the computation,

tree

binary

parallel-prefix problem

number

\342\200\242
\342\200\242
\342\200\242
(where
\342\200\242jc2\302\253
\342\200\242*,

step). The running

\342\200\242
is an

time

associative

of the algorithm

operation

binary

should be O(log

a sorting

Design

algorithm
12.4.3.

Section

in

smallest element to

for processorsconnected
Each leaf contains a

the kih leaf.

Show

the

that

through
number,

running

and

time of

Solve

that

tree

described

algorithm

in
to

modifications

a complete binary
the goal is to get

your

the

should
can be

n).

Consider the parallel algorithm for finding

the fan-smallest
element on a
Section 12.4.3. Suppose that the elements are not distinct. Describethe
the algorithm required to deal with this case.

described

xt.

each leaf /

under

tree

as

the kih

is optimal

to

a constant.

within

12.36 The processorsare arranged

in a ring. Each processorPt holds the \302\253th
row of an n x n
A and the \302\253th
element
of the vector b = (bub2> ...,b\342\200\236).
We want to compute the

matrix
matrix

vector

product x-Ab

that

such

jc,

is stored

in P{ at the

end.

(Xi

^aik-bk.)

Design an

*=i

algorithm for this

O (n)

12-37

Solve Exercise
matrix

12.38 The

(and

problem.

12.36,exceptthat

transpose ofanxn

initially
compute

initially

each

processor

Px

holds

the

\302\253th
column

of the

b{).

stored

in an n xn

the transpose

matrix

mesh

A is

the

such

that

matrix

P[z, j]

AT

that A
such that a]j = ajh
Suppose
an
O(n)
algorithm
Design
aVj.

holds

is
to

of A.

12.39 Suppose that you are given a mesh of nxn processors,each holding
a pixel of an nxn
In other words, each
black-and-white picture.
processorholds one binary number where 1

Creative Exercises

415

to white.
We would like to find the connected
to black and 0 corresponds
Two black pixels belong
to the same component if there
is a
components of the picture.
or
black path
them
are
not
considered
(horizontally
vertically;
diagonals
connecting
should
be labeled with a unique component
connected). All pixels in the same component
label. Initially,
the (//)th
that
pixel is labeled i-n+j. Considerthe following
algorithm
labels each component
in it. In each step, each processor
with the smallest label of a pixel
If any of them is
a black pixel looks at the current labels of its black
holding
neighbors.
smaller than its own label, then it updates it own. Prove that this algorithm will eventually
corresponds

the components correctly, but

c is a constant). How would
(where

that

label

12.40 Let A

be

an n x

in the

second row.

to row c(i)
12.41 Consider

element.

that o

such

/?),

in the

mesh

is stored

to n

j should

algorithm to permute
with the rows).
(starting

move to

usual

some input

column

way.

be two
is stored
to o and

Let o and n

of the grid and n

the rows of A according

in the first

an O(n)

Design

column

and

row

other

In

words,

row

move

should

n(j).

hypercube
topology and assume that each processor holds one
(n) parallel algorithm to find the maximum of the 2\" elements.

A7-dimensional

the

an O

Design

fix problem on an A7-dimensional

Design a fast algorithm to solve the parallel-pre
hypercube.
Each processor holds one number. You can assign indices
to the processors in any way you
each processor should have
want.
At the end of the
the value of the prefix
algorithm,

associated with
12.43

steps for

you terminate the algorithm?

in an n x

stored

matrix

of A according

the columns

12.42

of (1,

permutations

\342\200\242
require c n2 parallel

it may

Let C = (V,

its index.

E)

be a directed

to compute

algorithm

number of vertices of

weighted

the costs of

G.

You

can

with

graph

shortest

all

paths

use as many

positive
weights. Design a
in G in time 0(log2/7)
where

processors as you

want,

CREW
n

is the

and any reasonable

graph representation.

12.44

parallel

Design

Use as many

a. Design

for the
algorithms
as you can

processors

an algorithm

problem

following
while

preserving

to determine whether

a given

when

an adjacency

an efficiency

matrix

is given.

of O (1).

undirected graph contains a triangle

as

a subgraph.

b.

12.45

to determine whether
an algorithm
Design
a subgraph (namely, a cycle of length
4).

undirected graph contains a

square

as

A common
Xj

= a

Design
when

12.46

a given

is to use the recurrence

to generate
integer random numbers
way
where a, b, and p are constants (p a prime), and jr, is the seed.
*,_! +b(mod/?),
the first p numbers from this sequence
an EREW parallel algorithm for computing

the seed

is given. The

running

time

of the algorithm

should be T{p,p) = 0(\\ogp).

a.

tree of height n > 1 (which

has 2\" - 1 nodes)
cannot be
that a complete binary
one graph G in
embedded in an /7-dimensional
(Embedding
hypercube (with 2\" nodes).
of H such that
another
vertex of G to a unique
vertex
every
graph H involves mapping
into two adjacent vertices of//.)
two adjacent vertices of G are mapped

b.

trees in hypercubes approximately, that is, some of

Find
ways to embed complete binary
the tree edges may be mapped to short paths rather than to edges. The goal is to be able
for complete trees to hypercubes
with little effort (and with
to map algorithms
designed

Prove

reasonable

efficiency).

416

Parallel Algorithms

12.47 Show

to embed a

how

two-dimensional

grid

of

size

2k

by

2m in the

(fc

+ m)-dimensional

hypercube.

12.48

12.49

is a parallel gossip problem. Supposethat there are n =2k persons, each with
a certain
can communicate
with another person
In each step, each person
and
item of information.
he or she knows (including
information
learned in previous steps).
share all the information
than one person in any step.
more
cannot
communicate with
A
Design a
person
communication (gossip)pattern such that after \\og2n steps, everyone knows everything.

This

that both the x vector and the

Each processorwill have two
The partial values are accumulated
at the
right.
different speeds.)

for the convolution problem such

Design another systolic array
w vector move in the same direction (say, from left to right).

inputs from the left and two outputs to the

processors. (Hint: Movethe two vectors at

12.50 Design

a two-dimensional

same as a

arrives

matrix

systolic
mesh.

two-dimensional

through

the

first

array

for matrix

column.

The

multiplication.

arrives from
result moves from

The first matrix

the
right

The

connections

are

the

top row, and the second

to left starting from the

last column.

12.51

there
are only four different
applications of sequence comparisons,
to
four
of
In
that
case, each character can be
types nucleotides).
(corresponding
Show
that
it
is
two
bits.
to
encode
the
associated with the
numbers
represented
by
possible
the characters
in the systolic implementation
costs that move following
in Section
given
12.5.3 with only two bits as well. (This short encoding is important,
since
it reduces the
In

some

biology

characters

required bandwidth

between

the processors.)

SOLUTIONS

OF

SKETCHES

EXERCISES

SELECTED

TO

You

know

my methods.

Sir Arthur

them.

Chapter

1.13

Apply

Conan Doyle

states
have 269 electoral votes overall:
possible ties. The following
Wisconsin,
Illinois, New Jersey, Massachusetts,
Maryland,
Rhode
Island, Washington DC, Ohio,
Connecticut,
Minnesota, Washington, Iowa,
North
Carolina, Oregon, and West Virginia.
Michigan,
well before the 1988 election.)
This is written
(Note:
are

There

a few

quite

New

California,

York,

Chapter 2

2.1

The
by

base

x-y

xn-yn

The

case of n = 1
for all

2.6

induction
x and

hypothesis states that

We try to write
(x*y).

=x(xn~x

-yn-x)

+ y{xn-x

-y'^^

= x(xn~l

-y\"-l)

+ y(xn-1

-yn-l) + xy(xn-2-yn-2).

jc,

+ x-y\"^

xn~x -yn~x
x\"-yn

is divisible
in terms

of

-yx\"-]

in the expression above are divisible

by the induction
by x-y
includes xn~2-yn~2. Therefore, we use the strong induction
is divisible by x-y for all natural
the hypothesis to state that xk-yk
changing
<
k
such
n.
and
that
k,
y (x *y),

two

first

the

terms

term

third

The base caseis k

k
The sum for

(_1)<*-ih

The

numbers

hypothesis;
principle by
numbers

is trivial.

natural

for k- 1.
that the claim is true
and indeed l2 = 1(1 + l)/2. Assume
the sum for k -1,
which by the induction
assumption is
_
This sum is
is (-1)*\"1*2.
(\302\243
i)\302\243/2,
plus the *th element, which
= 1,

is

(k -\\)k/2
(_1)*-i*2+(-l )(-!)*-\342\226\240

= (-\\)k-i(k2-k(k-\\)/2)

= (-\\)k-lk(k

+ \\)/2.

A\\n

418

Sketches of Solutionsto

2.8

The proof is by induction on n. The case of

that
n and consider n + 1. We have to prove

n = 1

start with the

for

as we

can

side.

right

induction hypothesis. We now rearrange

result (2n(an+l + bn+l)).

(a+b)2n-\\an + bn)
=

+ bn+l)

2n(an+l

The left
other

term

words,

The 2n~l

is what we want,
we need to prove

to make it

the expression

as close

ban + a bn

-bn+l).
that the

to prove

sufficient

ways to prove

are several

We say

is the

remainder

that

we

n is

divide

if
if

a < b,
a > b.

We present a proof of
congruent to k modulo 3, written
by 3. (k can be any one of 0,

theorem.

this

an integer

when

In

+ bn(a-b) = (b-a)(an-bn).

= an(b-a)

then the expression above is equal to 0. Otherwise,

factor
is negative, and vice versa
positive but the right
no
is
more
the proof is complete.
than
and
0,
expression

theorem.

most 0.

expression becomes

it the

without

a = b,

There

is at

term

right

that

ignored, and

-bn+l

-an+l

is

so it

+ abn)

+ban

bn+l

bn+l) < 0.

an+l

can be

factor

= 2n~l(an+l

+ 2n~\\ban+abn-an+l

2n~\\b an+abn-

2.9

is true

the desired

to

If

that the claim

Assume

+ b)(a + b)n<(a + b)2n-l(an+bn),

(a + b)n+l=(a
by the

is trivial.

+ bn+l)>(a + b)n+l.

2n(an+l
We

Exercises

Selected

then the left factor

In either case,

is
the

a slightly more general

n =k (modulo 3), if k

1, or

2.)

We prove the

following stronger theorem.

D

A.l

Theorem

A number n, given in its decimal representation,

only if the sum of its digits is congruent to k
Proof: The claim

is congruent

checked for

to k modulo 3 if

and

3.

modulo

Assume that it is true for

representations of n 1 and n is
in
the
last
-1
not 9, then
of
n
is
only
digit, unless the last digit of n 1 is 9. If the last digit
when we go from n -1 to \302\253,
we increment
both the number and the sum of its digits; thus,
the congruence remains
the same.
If the last digit of n -1
is 9, then it changes to 0 and 1 is
added to the second digit. The change of 9 to 0 does not change the congruence
of the sum
of digits (since both are divisible by 3), and the addition of 1 to the second digit has the
same effect on the congruence of the sum of digits as that of the addition of 1 to the first
to 0).
digit (e.g., it may cause another 9 to change

n -

2.15

1, and

The subtle

consider

hypothesis

is that,

point

of the number

in

be easily

can

difference between

n. The

we subtract

when

question

we
\302\253,

on n-at
cannot be used
But the requirement was

2.26 The only

that

tree with two

the theorem is true

vertices
for

n -

a single

is
1

vertices,

sure

numbers.

decimal

an element,

say

ah

from the list

to reduce the

size

<a{\\ otherwise, the induction

since it may require another use of a{ in its
from the list.
numbers
that the sum use distinct

must make

representation.

small

the

that

n -a%

in which
case both degrees are
and consider the case of n vertices.

edge,

1.

Assume

Since

all

Sketches of

Solutionsto

Selected

Exercises

419

and their sum is In -2, at least one of them, say dh must be equal
to 1
degrees are positive
and at least one of them,
1. We can now remove d( from the
say djy must be greater than
sequence and decrement dj by 1. There are now n -1 numbers whose sum is In - 4;
exists
a tree with these degrees.
We can add
therefore,
by the induction hypothesis, there
one additional
vertex to this tree and connect it to the vertex whose degreeis dj - 1 to obtain
the desired tree.

2.29

the

inside one box,

balls are put

two

If

is true for

principle

Assume the contrary; namely,

box
with one ball, and if we
induction hypothesis.

2.34 The claim

the box contains more than

one ball. Assume that
and consider putting
n + 2 balls inside n +1 boxes.
no box contains more than
one ball. But then, there exists a
remove both the box and the ball we get a contradiction
to the
then

balls

+1

true for n - 2. Assume that it is true for n and consider n + 2. Take any
The graph without them is Kn, and by the induction hypothesis it can
be partitioned into nil spanning
trees.
We need to extend
those spanning
trees to include
v
tree.
The edges included in Kn+2 that are not covered
and w, and to add one more spanning
are connected
to v and u. We partition the set of n vertices
into two
by Kn are those that
the edge (v, u) and all edges
equal-sized groups X and Y. The new spanning tree contains
trees
from v to vertices in X and from u to vertices in Y. Each of the previous spanning
we can extend
each of these spanning trees to span
includes all vertices in Kn. Therefore,
a vertex of X to u and an edge from a vertex of Y to v. We
Kn+2 by adding an edge from
will
different vertices of X and Y for each spanning tree.
choose
two

2.38

It

is easy

to remove the

2.

We

value of c.

We

+ Acn \\og2n

Both inequalities

clearly

that T(n)<cn

guess

and

c2(log22)2>4,

(1)

n/2

/i(/i + 1)/4+1/2.

+/i/2

power of

2cn (\\og2n)2

= T(n-2) + (n-\\)/2+

\342\200\242\342\200\242\342\200\242
=

+ 3/2+

Let n be a
3.5.1) that
for some

can

simplify

+ 2cn

hold

that

for

(2)

(\\og2n)2. We

have

>2cn

c(2n)(\\og2(2n))2

to verify
(\\og2n)2

(see Section
+4n

log2(2\302\253),

(2) to

> 2cn (\\og2n)2

+ 4n.

+ 4\302\253
\\og2n

c >2.

We have
\302\243/Mog2i

3.15

recursion:

= T(n-\\) + n/2

1+2/2

3.14

w.

in the proof: We took it for granted that a maximum

a subtle implicit
added
assumption
then there are finitely
many
possible sets
always exists. If all the x(s are positive integers,
if
some
of
the
exists.
whose sum is S, and so a maximum
However,
jc,s are real
always
then a maximum may not exist. Hence, the proof, as is, is valid only for integers.
numbers,

T(n)

3.8

v and

We

Chapter

3.6

is clearly

vertices

<

The functions f

xk\\o%2xdx =

(n) = 3n,

g(n)

jf]-[

= 2n, and

\302\260gX~

s(n) = r(n) = n

\\\\

form

=0(nk+l\\ogn).

a counterexample

to the

claim.

SelectedExercises

of Solutions to

420

Sketches

3.17

We

can

and

the other

the two

construct

one grow

functions

by

one of

letting

odd numbers.

for the

fast

We

them

/<1) = S(0=1.
f

- 1) +

(/?

^(\=1") f(n-\\) + (g(n-\\))2

fast for all even numbers

the functions recursively. Let

grow

define

odd

when

n is

when

a?

when

n is

even

when

n is

odd

is even

and
g (n

f
g(<n)=\\

all odd

3.19

+ (f(n-\\))2

g(n-\\)

/(n) is about
for

- 1) +

the square of g (n) for

numbers. Therefore, / (n)

We can use Theorem3.4,since

all even

m and

\302\261
O (g

numbers, and g (n) is about

O (f (n)).
(n)) and g (n) \302\261

c are constants.

The

result

the

square

of

/ (n)

is

S(n) = o[nlogmUmlogmi
3.21

is T(n) = 0(n).

The solution
to show

have

To

>

dn

it, we

prove

guess that

constant d.

for some

T(n)<dn

We

that
k

+ en

a-,(dnlbt)
\302\243
i=\\

=dn^

a\\lb\\

+ en.

/=!

3.25

Since

/=

If we

replace

> 0, the

a,lbt
\302\243
i = \\

above

inequality

holds for d>elf.

the
term
T{\\yn\\ ) with T(n/2), then we get the typical
divide-and-conquer
is no
relation, whose solution,
3.4, is 0(n log a). Thus, since
by Theorem
than n/2 for n >2, T(n) = 0(n logn). However,
it is possible that
is smaller
greater
T(n)
than
that.
Let's try T(n) = en. The basecase implies
that e>\\. Consider 2n. We have to
prove that

recurrence

2cn > en

3.29

a.

true for e-2

is clearly

which

If

+ n,

+ Vo?

power of 2,
we can substitute

n is a

case,

the

then

n >2.

and

upper part of the

for n, and obtain

recurrence
the

T(k) = T(2k).

r(*) = r(*-i)+i,
The

of this

solution

b. Consider now
replaced

by

n = 2k' - 1,
numbers

recurrence relation

will

and
always

will

T(k) = T(2k -

always

1),and obtain
r(k) = 2T(k-\\), r(i) =

The solution

of

this

be used.

recurrence

In

relation

this

for

is clearly

T(k) = k + 1.

use the bottom part of the recurrence, then T(n) is

words, if we start with
(n -1)/2 = 2*~' -1. In other
use the bottom part of the recurrence relation, becausethe

If we

n=2k-\\.

then we

always

no)=i.

7((a7-1)/2),

involved

will

following

be of the form 2J -1
the recurrence relation

for

some

j.

We can

now

define

i.

recurrence

relation

is

clearly

T(k)

= 2k~].

The

recurrence

relation

of Solutions to

Sketches

421

Selected Exercises

of 2, because its definition

for powers of 2 and for nonpowers
is
differently
and odd numbers.
different between even numbers
We can restrict
ourselves
to
is a function that behaves
powers of 2 if the solution of the recurrence relation
\"nicely.\"
In particular, the expressions for the running times of most algorithms
are monotonically
increasing functions. If the difference between the value of T(n) and T(2n) is no more
than a constant and if T(n) is monotonically increasing, then T(m) for n < m < In is no
more than a constant times T(n). Therefore, the analysis for powers of 2 is sufficient
in
that case.
behaves

vastly

Chapter

4.5

16 is

Since

4.12

use an

will

We

occupied, 8 (16's

easy.

be occupied. By the same argument, 4, 2,

of heap elements in an array of size

must

number

minimal

n and store element / at

complicated, since / is not

of size

array

is more

removal

parent)

Thus, 5 is the

be occupied.

must

location
specified

and

16.

/. Therefore,
insertions are
and we do not know which

locations are occupied. We handle removals by linking all the occupied locations in a
list. An insertion of / thus involves marking the /th location and inserting / to the list.
A removal deletesthe first element from the list and unmarks that element.
linked

4.13

We

first

and

rearrange

an arbitrary element from

one of the heaps (e.g., the top of one of them),
this heap. We then insert the removed element as a new root such that the
two heaps are connected to it as its children. The new
root may not be in its
place, but it takes O(log (m + n)) steps to move it down the combined heap to a

remove

roots of

the

appropriate

correct place.

4.15

4.17

We
use AVL trees so that
insertions
and deletions can be done in 0(\\ogn)
To
steps.
for jc, except
that we branch right
we use the regular search algorithm
perform Find_Next(x)
if we meet x and x is not a leaf. If x is not a leaf, then the search leads us to the desired
element. If x is a leaf, then the element for which we are looking is the last node on the path
from the root to x where
we branch to the left (if we always branch to the right, then x is the
We can, during the search, store a pointer to the last node from
in
element
the
tree).
largest
which
we branch left and use it if x is found to be a leaf.

in O(log n) steps. We add a

so that insertions
and deletions can be done
of v
to every node v, denoted by v.D, which contains the number of descendants
node
v
the
number
of
We
can
use
this
field
to
find
the
of
a
rank
(namely,
(including v).
elements in the tree that are smaller than or equal to v.Key) during the search. This is done
as follows.
We assume that we know the ranks of all nodes on the
by the use of induction
of w.
path from the root to some node u\\ and show how to find the ranks of the children
Denote the rank of node v by v.Rank.
use

We

AVL trees

new field

w.Left.Rank = w.Rank

- 1.

w.Left.Right.D

w.Right. Rank = w.Rank + w.Right.Left.D + 1.

(If

any

of

Find_Next(x,

the

is unsuccessful).
according

introduced

children

k) we

first

We

does
find

not

v such

exist,
that

then search for the

as it is according
a new field, we must

to ranks

show

the

v.Key

node

to keys.
how to

value

is

taken

as

0.)

To

perform

=x (if there is no such node, then the

with
rank v.Rank + k. It is as easy to

search
search

We are not yet done, however. Since we have

maintain this field when insertions or deletion

422

Sketchesof

take

Here is

4.23

in

another question:

did we

Why

search for

a key

along

(delete)

we must redo these

add a descendant

that,

(Notice

path.

if the

changes.)
field

with a specified rank

we increment

element

an

the search

instead

of a rank field,

which

easier?

much

the smaller
elements) be /*, and the
(i.e., the tree with
Assume
that
other
case
is similar). First, delete
(the
hl>h2
right
height
the maximal element from the left tree. Denote this element
by r. Then, use r as a new root
for the right
tree, and insert this root in the appropriate
place on the right side of the left
More precisely, traverse the left tree, taking only right branches, for h, -h2 steps. Let
tree.
concatenated
tree will have r in place
node at that place be v, and its parent p. The new
the
of v as the right child of p, v as the left child of r and the root of the right tree as the right
of r (the right tree remains below its root on the right side of r). It is easy to verify
child
search
tree. This insertion may invalidate
the AVL property,'
that this is a consistent binary
case we can use the usual remedy of a rotation.
in which

Let

height

of the

of the

tree

the

tree

\"left\"

be h2.

worst AVL tree of height

h (namely,
the tree with the fewest nodes). 7, is
with a root and one child. T2 is a tree with a root and two children
such that
one of the children
has one child of its own, and the other child
is a leaf. In general, we
assume that we know how to construct
Th and consider Th+l. Th+l must have a child that is
a root of a subtree
of height h. Since this subtree must
the AVL property and we
satisfy
want a tree with
fewest
since the AVL property
nodes, we should use Th. Furthermore,
must
also be satisfied at the root, the other child
of the root must be the root of a subtree of
at least h-\\.
The subtree
with fewest nodes for this child
is thus Th.x. Therefore,
height
a
consists
of
root
connected
to
and
The
of
number
nodes N(h + \\) in Th+i
Th+l
Th
Th_x.
satisfies

Let

be the

Th

a tree

clearly

N(h +

\\)

+ N(h-\\)+\\.

= N(h)

Fibonacci recurrence relation.

in an AVL tree. This
the height of an AVL tree with

is the

This

number of nodes
on

bound

4.27

nodes

all

is unsuccessful,

deletion

we insert

When

follows.

fields

descendant

have made the

would

4.20

is done as

This

place.

(decrement) the
insertion or the

Exercises

to Selected

Solutions

Its
lower
n

nodes

leads to

solution

the

be used
in the theorem.

can then

bound

given

on the
the upper

bound

lower

to prove

sums
of numbers in a special way. Let's
by S, to store
from 1 to
k. We store in S [2k~x ] the sum of all numbers
into two subproblems of half the size \342\200\22
so, we have divided the problem
the left side of the array (from 1 to 2*\"' -1), and the right
side of the array
maintaining
The first subproblem
(from 2*\"' + 1 to 2k - 1). Each subproblem can be solved
by induction.
We

use

the extra

assume first that

2*\"'. By doing

array,

n = 2k -

solved independently,
in S
appropriate change

denoted

1 for

some

the
is performed,
Add operation
can also be solved
The second subproblem
sum.
that we add the value of S [2*_l] to the returned value of Partial
independently,
except
If we look beyond
the recursion,
what happens is that S [i ] holds the sum of the numbers
from A[j] to A [/], where j is the closest
index to / that is divided by a higher degree of 2
than
i. For example, S [ 12]holds
the
the sum A [9] +A [ 10] +A [ 11 ] +A [ 12], and S [6] holds
value of A [5] +A [6]. We leave the exact details
to the reader.
can be

4.29

We can

that every time an

except

[2*_l

must

be made.

all the operations in

AVL trees. To support
Scale, with the tree. Add_all{y)
simply
value
associated
with x. Insert(x, Value)
way

by

implement

using

the exercise,

except for Add

Add_all(y),

we associate

all,

in a straightforward

one more

global

adds y to Scale. Find_value(x)

adds
sets the value of x as Value
-Scale.

variable,

Scale

to the

of Solutions to

Sketches

Selected Exercises

423

Chapter 5
is the

loop invariant

The

5.4

Only one
that

following:

person
is i

person

next

the first

among

= j,

if next

orj

persons

can be the

candidate,

and

= i.

if next

maximum consecutive subsequence

problem,
except
even further.
We assume that we know (1) the
hypothesis
maximal
(2) the subsequence that ends at the end with the
product,
with the minimal
maximal
and (4) the
product, (3) the subsequence
negative
product,
that ends at the end with the minimal negative product. We leave it to the
subsequence
reader to handle
all these cases.

The

5.12

5.14

is similar

solution

to

of the

that

that we strengthen the

with the
subsequence

induction

The

for a tree

be
left

is trivial

solution

solvethe
an

for a tree

problem

arbitrary

distance between w

Let v

with

7\\ and let w be the

tree T. The distance

and

vertices. Suppose that we know how to

and consider a tree T with n + 1 vertices.
Let v
only vertex connected to v. If we remove v, we are
v and any vertex
between
u in T is 1 plus the

one

or two

n vertices,

leaf in

a smaller

with

with

u.

arbitrary vertex in G, which we designate as the root of the tree. If we remove v

tree, we obtain several smaller trees whose roots are the vertices that were adjacent
to v. We can repeat
each time obtaining new roots and smaller trees, until all
this process,
are removed. We will solve the problem by induction
on the number of times
we
edges
until all edges are removed. If the path corresponding to the
have to perform this process
diameter
of the tree does not contain
the root, then
the diameter
of the tree is also the
of one of the smaller
If
diameter
which
we
find
induction.
the
root is contained in
trees,
by
the path corresponding to the diameter,
then this path connects
two vertices in two separate
smaller trees that are farthest away from the root. This observation
suggests the following
is stronger than the straightforward one).
induction hypothesis (which

5.15

be an

from the

Induction

hypothesis:

n vertices,

and

how

We know how
to find

the

maximal

to find
distance

of subtrees
a fixed root.

the diameter

from

<

with

The base caseis trivial.

Given
a tree with n vertices, we designate v as the root, and solve
for
all
the subtrees
rooted at the children of v. Notice that the
induction
problem
by
to find the
distances we find are those to the children of v, and not those to v. However,
distances to v, we need only to add 1 to all the distances. After doing that, we compare the
with the sum of the two maximum
maximum
diameter found among the subtrees
distances
from v. The larger of the two is the diameter.
the

We assume that a subset whose sum is k exists.

Use the black box
one of the elements. If the answer is yes, then
is a solution
there
the answer is no, then the element is necessary. In either case,
reduced, since the status of one element has been determined.

5.21

without

input set
this

without

if

element;

the problem

has

been

Chapter

6.3

for the

the text in half and running the program on one half. If

can use binary search by cutting
an error occurs, then we know that it is in that half and we can continue in the same manner;
it is in the other half.
if no error occurs, then
(We assume that the offending string has not

We

been

cut.)

424

Sketches of

6.11

We

Selected

Exercises

Partition (Fig. 6.9) such that, if the pivot is not equal to X [Left ],
Let X[l] = 1, X[n]= 2, and let all other
with X[Left].
will
In
2.
this
the
than
be
elements greater
case,
clearly be X [n ]. Algorithm Partition
pivot
as
mentioned
then
with
will
first exchange
above,
X[n]
X[n] with X[2]
X[\\]
exchange
(sinceX [n ] is now < 2 and X [2] > 2). Since all other elements are greater than 2, no other
exchanges will take place. The result of the partition is to put 1 and 2 at their correct places
will then recursively sort the array from the third
of the array.
in the beginning
Quicksort
this pattern, we need to put the third smallest element in
to the last one. To continue
index
in the last position in the array after the partition
the third position, and, since the element
to put the fourth
in the second position, we need
smallest
element in the
was originally
if n = 8, then a bad order will be 1, 4, 3, 6, 5, 8, 7, 2.
For example,
second position.
first

algorithm

modify

the

then

6.14

Solutionsto

is

pivot

first

exchanged

sizes different
two equal-sized subsets
(or into two sets with
by 1 if the
and merge the results.
set is odd). Solveeach subproblem
recursively,
the maximal
and minimal
element.
recursive
solution produces two results,
Each
Only two
to
the
two
one
to
the
two
are
and
solutions;
maximals,
merge
compare
required
comparisons
The following recurrence relation
is obtained:
one to compare the two minimals.

Divide

the

set into

size of

the

original

T(n) = 2T(n/2) + 2,

7(2)= 1.

that T(n) = 3n/2-2: T(2) = 3\342\200\242

2/2-2=
1, and T(2n) =
+ 2 = 3n -2.
The savings come from
the base case! It takes
only
one comparison to find both minimum and maximum
two numbers. We do it (at
between
the final unfolding
of the recursion) for n/2 pairs, so we save about
n/2 comparisons.
It is
to
note
more
than
if
is
that
3n/2-2
be
needed
n
not
a
interesting
comparisons
may
power of
2. For example, since 7(3) = 3, we have 7(6) = 8, 7(12)= 18,and so on. The algorithm
It

is easy

2T(n)+ 2

to

by

verify

induction

= 2(3/i/2-2)

suggested in

Section

6.5.1 is

better in

this

case.

6.19

Consider w =A [n/2] (we assume without

loss of generality that n is even). If w>z,
then z
must appear in the array between A [1] and A [n/2]. This is true because
the difference
between consecutive
in the array
is at most 1, and
elements
therefore
no number can be
If
w
<
a
similar
z
must
between
A
then,
z,
skipped.
[n/2+ 1] and A [n ].
by
argument,
appear
In either
number
case, we cut the search space in half with each comparison. The maximal
of comparisons is thus [log2\302\253l.

6.20

We can

6.22

a. Sort the

set S\\

b. We

induction.

use the

bound. The number

of possible
answers to the
information-theoretic
so
Exercise
6.19
is
n
A
A
and
can
be
to
on). Therefore,
question
(z
[1],
[2],
posed by
equal
leaf
decision
tree that solves this problem
must have at least n leaves
any
(where the \302\253th
to the answer \"z is equal to A [/]\.") The height of such a decision tree must be
corresponds
at least flog2\302\253l.

use

simple

Induction

of <n

for each

then,

element z of S, perform

hypothesis:

We

know

how to

binary

solve the

search

problem

for the

for a

number x -

z.

sorted set

elements.

in increasing
and let
that S is sorted
order. Consider 5[1] and S[n],
S [n J. If y =*, then we are done. If y > jc, then clearly S [n ] cannot be part of
the solution because S[n] + S[i]>x for all /. Therefore,
from
we can eliminate
S[n]
consideration and solve the remaining
<
a
similar
If
induction.
then,
jc,
by
problem by
y
argument, we can eliminate S [ 1 ].

Assume

=S

[ 1J +

of Solutions to

Sketches

Selected Exercises

425

6.26

the largest,
the set, and pair the smallest element with
the second smallest
with the
second largest, and so on. This procedure is correct for the following reasons. Suppose that
there is another partition with
a better bound, in which x, and xn are not matched. Let x, be
matched
in this partition with
But then, we can change
the partition
so
jc, and xn with Xj.
that
jc i is matched
with xn and jc, is matched with xjy without
the
maximal
sum,
increasing
which
is a contradiction.

6.28

The

6.29

Sort

a total order on xux2

an defines
a2, \342\200\242\342\200\242\342\200\242,
xn. That is, we can
and
of
elements
jc,
a, to o;. We want to rearrange the
4'compare\"
any pair
Xj by comparing
in order. Therefore, any sorting
that the o,s will
that sorts
XjS such
appear
algorithm
to the values of the o,s and moves both jc, and at together will lead to the desired
according
outcome. Sincewe want an in-place algorithm, we can use, for example, heapsort.
o,,

permutation

all the

put

Initially,

remove the

of the

elements

minimal

in the

element

minimal

heap

and

insert

sequences
the next

in

a heap.

Then,

element from

in

each

step,

the corresponding

sequence.

6.31

a.

into a balanced binary

search
tree (e.g., an AVL tree). Each
node of
and a pointer to a linked list of all the elements with the same key.
Since the number of different keys is 0(log\302\253), the number of nodes in the tree is
the tree's height is 0 (log log a). After we construct the tree we
0(}ogn) and therefore
can append the different
linked lists to one list by performing
inorder traversal of the
element

each

Insert

the tree

a key

has

tree.
b. The

proof

lower bound for

assumed that

6.33

permutations

to only

the possible inputs

restrict

The sum of heights

is at

does not

sorting
all

those

apply

to this

are possible
with 0(}ogn)

problem, because the lower bound

In this problem, we
inputs.

valid

different

values.

most

r(/o

To evaluate

7(7)=
= 3

+ i)//)l.

2xriog2((/i

start with a small

T(n)v/e
log28

+ log22 + riog28/5l +riog28/6l+[log28/7l

+ log24+riog28/3l

+ 2 + 2+1

+1+

concrete example. Consider7(7):

+ 1.

1, two terms of size 2, and one term of size 3. Let's

of
of (n + l)/2 terms
to see that T(n) will consist
is exactly the
of size 2, and so on, up to one term of size k. (This
size 1, (n + l)/4 terms
heights of all nodes in a complete binary tree.) So, overall

There

are 4

assume

that

(=(n+ l)/2) terms

n +

=2*.

It is

of size

not

difficult

it

6.36

T(n)

This

sum

be

most

at

Suppose

Zi2k-i.
was

that

comparisons

smaller
being

than

to be O(n) in Section 3.4. If n + 1 is not a power of 2, then

contribution of 1 to each term, adding no more than n.

shown

an extra

A finds that x, is the kth largest element by making

algorithm
results cannot determine whether
whose
a certain
element Xj
jc,-.

equal to a

This
value

largest element, which

there

a series

can

of

or
that
the
is
outcome
of
the
consistent
with
implies
comparisons
Xj
z <Xj. But both cases lead to a different
kth
y >x, and to a value
is a contradiction.
is greater

426

Sketches of Solutionsto

6.38

The idea is to

consider

their median.

All

2k elements

elements

consisting of computing

6.41 Thereare exactly

the

greater

and do

next k elements

same.

the

the median

n + 1 possible

must have at

problem

We start with the first 2k elements, and

at a time.
We now look
than the median can be eliminated.
This

process requires

of 2k

elements.

answers

least n

is discussed in

Exercises

Selected

+ 1 leaves.

to this problem, so every decision

The claim follows immediately.

This problem

6.46

the next table for the part of the pattern

The basic idea is to construct
it as more characters of the pattern
become known. This is not
extend
to the regular pattern matching
of the next table is similar
construction
Takaoka[1986]).

6.51

b serves

Part

(and the

as a good

hint

for

each

tree that

solves

10.2.

6.44

Section

the

phases,

nlk

approximately

find
at

the solution

of part
T and

We find the

a.

hard

to do,

problem

minimal

edit

(see also

distance

the assumption that no

can be used,
are allowed (the same algorithm
that the cost of a
except
replacements
is set to 2). The LCS of the two sequences
is the set of characters
that are
replacement
matched in the minimal edit sequence (i.e., those that are not involved in the insertions or
We leave the proof to
deletions). The SCS is the LCS plus all the insertions and deletions.
edit

minimal

sequence)

between

and to
since the

is known,

that

under

the reader.

The easiestway

6.54

other

this problem is to use the regular sequence comparison

algorithm,
of C (/, 0) (seealgorithm
Minimal
Edit Distance) are 0. In
which
is exactly what
we
inserting the beginning of A does not cost anything,

that the

except

words,

to solve

initial

values

want.

The

6.60

(not

answer

surprising)

algorithm

Vegas

that runs

in

is

that

expected

it is

better to be correct

time T(n) and

always

than

to be fast.

produces

Given a Las

the correct

result, we

it into a Monte Carlo algorithm

termination of the algorithm
by enforcing
after 4T(n) steps, if it has not already
terminated.
Since its expected running
time is T(n),
the probability
that it does not terminate
after AT(n) steps is no more
than 1/4. When we
enforce termination,
we
an arbitrary
answer.
The algorithm now runs in
output
time 0(T(n)) and no more than 1/4 probability
of error.
(guaranteed)
A little harder
Show that it is sufficient to run the algorithm for 2T(n) steps.
question:
can

6.61

transform

Each
candidates.
algorithm, except that we keep up to three different
is compared to the candidates,
the
then
and if it is equal to one of them,
from
is
is
different
if
new
element
incremented.
the
Otherwise,
corresponding multiplicity
the existing three candidates,
of all candidate (by the
then we can decrement the multiplicity
same argument as in the majority algorithm \342\200\224
from
we can eliminate four distinct values
If there
are less than three candidates,
we add the current element as a new
consideration).
candidate. At the end, all remaining
are compared to all the elements.
candidates

We

use

the majority

element

Chapter

7.4

Let T be

a tree that satisfies the conditions

the rest of the graph.
of the problem, and consider
be an edge not in T. Since T is a DFS tree, e must be a back edge; that is, e must
connect a vertex to its ancestor. However, since T is also a BFS tree, e must connect two
at the same level or at neighboring
vertices
the only
levels. This is a contradiction;
hence,
graphs that satisfy the constraints of the problem are trees.

Let e

Sketches of

7.8

The claim is wrong.

7.24

The conditions
and its

7.27

Perform

in
7.29

7.34

that

of the
traversal.

problem

(a BFS
satisfies the

a DFS
tree

will

imply that G

do just

is Eulerian.

as well), and

conditions of the

Solutionsto

Selected

C in

an Eulerian circuit

Consider

construct the

427

Exercises

DFS

tree.

Prove

that

any

G,

leaf

problem.

The solution is by induction.

The base case is simple. We want to reduce the problem to
one with n -1 numbers, and still have the same constraints.
all d,s are positive
Since
and
their sum is 2n -2, there exists at least one k such that dk = 1. There also exists
j such that
dj> 1. So,we can remove dk from the list and subtract 1 from dj. This is a valid problem
of size n-\\ (everything
is still positive and the sum is 2\302\253-4), which
we solve it by
induction.
We find a tree T with the corresponding
degrees. We need only to add a leaf to
the vertex
to dj in T to find the tree that solves the original problem.
corresponding
use

We

the KMP pattern-matching algorithm (Section 6.7) in combination

with a DFS-like
traversal
of the tree. During
the traversal,
we maintain
the status of each vertex

(preorder)
in
the

to current potential matching.

regard
maximal

prefix in the pattern

that

That is, we

matches

the

define

suffix

v.match

to be

of the path from

equal

to the

the root to

v.

size of
(This

is exactly the same information

that is maintained by the KMP algorithm.) preWORK for v
is defined by including
v's character as the next step in the KMP algorithm,
and computing
\342\200\224
v
the
status
of
that
vmatch. postWORK is defined
is,
by restoring
by continuing the
in the number
time of this algorithm is linear
matching from v using v.match. The running

of vertices in
7.35

Without the cycle, the graph would have been a tree for which the problem is easy (pick a
root
and direct all edges away
from it). We can discoverthe cycle
the first
by DFS when
is found.
to a previously visited vertex
The cycle can be directed
so
(and only) edge leading
1. All other edges should
that it becomes a directed
and all its vertices have indegree
cycle
another DFS from any
be directed
away from the cycle. This can be achieved by starting
all edges in the direction they are visited
the search.
vertex
on the cycle and directing
during
The single back edge will point to the root of the search which will have indegree 1 (as will
all other

7.37

the tree.

vertices).

DFS on the graph, and build

a DFS tree T. Sinceall the other edges in the graph
of T to point away from the root and all
are back edges relative
to 7\\ we can direct all edges
vertices that are closerto the root (i.e., toward the ancestors).
other edges to point toward

Perform

7.40

of indegree 0 are removed together

so that all vertices
Modify the topological sort algorithm
handles
from the queue and are put in one group. The algorithm
these vertices as before
the corresponding indegrees), and
(i.e., it removes their emanating edges by decrementing
then
0 are removed.
Prove that the groups satisfy the
again all vertices of indegree
conditions of the problem, and that there are no more than k + 1 of them.

7.42

Use BFS starting at v. Each edge (jc, y) encountered

through the search, such that y is a new
the count of
vertex, is part of a shortest path from v to y. A common error is to increment
there
(jc, y) is encountered. However,
may already be several
and
one
of
a
shortest
to
each
of
them
is
a
to y (ending with
jc,
path
part
paths
to x
is to add the number of shortest
the edge (jc, y)).
Hence, the correct solution
paths
to 0.
to the count of shortest
known)
(which is already
paths to y. All counts are initialized
shortest

known

paths

shortest

to y

when

428

Sketches of

7.43

The

no heap

time (since

Add to
compare

possible
small

be

7.48

V \\)

find a new

we

Instead,

but now the

time,

minimum

update for

each

path by looking at all

edge takes only constant

is maintained).

Use the same algorithm, except

edge a weight of \\u where u stands for a unit.
The units are used to
plus the sum of units.
length of a path is its regular length
artificial
two paths with the same regular weight. Instead of using
units, it may be
to add a very small number x to the weight of each edge. The value of x must be

each

the

that

O (|

takes

This

Exercises

Selected

to use a heap.

is not

idea

main

the vertices.

7.45

Solutionsto

that it

enough
taken

as m/( |

then

m -

1).

We

use

induction

does not

on k,

paths with

affect

V | + 1), where

find

and

different

the precision of

m is

not only the

we discuss
vertices. For simplicity,
of k = 1
themselves.The
case
paths

For example, x can

weight.

if they

(e.g.,

weights

are

integers,

to w, but also the paths

to all other
the lengths of the paths, rather than the
Denote by w.SPk the length of the shortest

path

only

finding

regular
the

is obvious.

that we know the shortest paths,

path from v to w which contains exactly k edges. Assume
That is, we know the values
which
contain exactly k - 1 edges, from v to all other vertices.

of w.SPk.l for all

to x.SPk_x + length
0<|EHV|).

7.50

well.

(x, y)

for a

(Hint

it is

if

consider

smaller

than

all edges, and for

current y.SPk.

the

each edge(jc,

y) set

The

time

running

Shortest Paths (Fig. 7.16)works

for negative-cost
a vertex is considered, all the paths leading to

Acyclic

Improved

Algorithm

need to

w. We now

Whenever

proof:

y.SPk
is thus

edges as
it

from

the

source have been compared.)

7.52

The

must be Eulerian, since each cycle adds a degree of 2 to each of its vertices.
On
the edges of an Eulerian
can
a
set
of
be
into
always
partitioned
cycles.
graph
is enough
to note that there exists at least one cycle in an Eulerian graph, and that
graph

the other hand,

It

removing

a cycle from an

Eulerian

graph

gives

a set

of

(possibly

one)

connected

Eulerian

graphs.

7.55

is an excellent example for a solution

will be on the
by induction. The induction
number of vertices.The basecaseis simple.
like to
Let v be any leaf in the tree. We would
remove v, to solve the problem
for the remaining
a value of S{y) that
tree, and to find
satisfies
the requirements. If the edge connected to v points toward v, then the reduction is
simple:Let the edge be (w, v); then, S (v) can be set to S(w) \\^j {X(w, v)J, and it is easy to
see that the requirements are satisfied. The difficult
case is when the edge is (v, w). In that
all the other subsets
case, we set S(v) to be equal to S(w) and change
by adding the label
^(v, w) to them. It is easy to verify that all the requirements are satisfied.

7.60

Start with F

as

algorithm

Exercise

7.64

This

in

the

current

\"minimal\"

spanning tree

and

add

edges

to

it

according

to the

7.59.

For
the MCST, call it T. (The answer for all edges of T is obviously
T itself.)
v
v and w, find the maximum-cost
in
the
connecting
(unique)
path
edge
to w in T. This
is a preprocessing step. We may
for all pairs, but it
not need the information

First

find

each pair

the edge
tree containing
them. A minimum-cost spanning
the
the
maximum-cost
to
then
7,
edge
edge (jt, y)
by adding
removing
the unique cycle that is formed (or the second largest,
if (.v, y) is the maximum). This
is the maximum-cost edge on the path from x to y in 7\\ whose identity we obtained in
DFS on T for | V \\
step. The preprocessing
preprocessing
step can be done by using

is simpler

U, y) is
from

edge
the

of vertices

to compute

obtained

it

for all of

Sketches

7.67

time starting

each

times,

vertex.

all

order

of the

costs, it

also

does not

change the

result.

we notice that we can

First,
(This is just a sketch of a rather complicated algorithm.)
concentrate on T and ignore the rest of the graph.
Since T has already
been found to be an
a cycle with
the edges of T in which
it is a
MCST,
every other edge in G completes
maximum-cost edge. Adding
another
vertex does not change
that
fact. We will
also
in T (e.g., we can add dummy
assume, for simplicity, that v is connected to all vertices
on the number of vertices in T. Consider
T as a
edges with infinite cost). We use induction
and
an
rooted
tree (with an arbitrary
look
at
internal
vertex
w
connected
to
several
root),
leaves. Each pair of leaves x and y connected to w defines a cycle of four edges with v and
w (containing
the edges (w, jc), (jc, v), (v, y), and (y, w)). The maximum-cost
edge of this
cycle should be deleted, which creates a new leaf in the new graph. This leaf is guaranteed
We are not quite done,
to be in the new MCST, which will reduce the size of the problem.
of (w, z),
since w may have only one leaf z. In that case, the cycle is a triangle
consisting
in
is
is
maximum-cost
this
and
The
difficult
case
when
the
(z, v),
(v, w).
(v, w)
cycle
edge
no new leaf is found.
and needs to be removed. This case is difficult
because
we
However,
*
w. We replace the two edges (w, z) and
can still reduce the problem by 'compressing\"
of w in the rooted tree) by one edge (p, z), and set the cost of
(w, p) (where p is the parent
it to the reader to
leave
this
new edge as the maximal
cost of the two old edges. We
this argument into an algorithm. Oneneedsto prove that the compression is valid,
complete
and to design the appropriate data structure
to maintain the necessary information
the
during

running of the

algorithm.

in the way the

vertices
backward.
Instead of deleting
data
in the opposite order. We use the union-find
in the graph. We start
the set of components
structure (Section 4.5) to maintain
currently
We then consider
vertices
with vertex n and add it as one component to the data structure.
n -1, n - 2, and so on. For each vertex
z, we first add it as a component, then check all
z and j belong to
(/', j) such that j > i. For each such edge (/', j), we check whether
edges
same component
the
and, if not, we join the two corresponding
(using the find operation),
We stop when one component contains more than
components (using the union operation).

The

main

problem

idea is to solve the problem

is stated, we add vertices

vertices.

nil

7.71

Exercises 429

costs

the

change the

7.70

to Selected

an MCST, because only the relative order

costs by a constant does not change
is important for the algorithm.
Whenever costs are used in the algorithm, they
are compared
to each other with a simple comparison ( \"<\" or \">\.") They are not used (or
a constant
to all costs does not
manipulated) in any other way. Therefore, since adding

Increasing

of

7.68

from another

of Solutions

Since every
E-F cannot

cycle

in G

an
must contain
We want
The largest

a cycle.
the size of E-F.
contain

edge from the feedback-edge

the size of F,
to minimize

set F, the
which

set of

is the

edges
same as

no cycle in a graph corresponds

set of edges with
if the graph is not connected).
tree (or spanning
Hence, the minimumforest,
tree of the
size feedback-edge
set in an undirected graph is a complement of a spanning
that the graph is connected). The
graph, and its size is | E \\ - \\ V | + 1 (assuming
for directed graphs is NP-complete (see Chapter
11).
problem
corresponding
maximizing

to

7.74

a spanning

Check
w.SP

all the edges not in

(where Jt.SP is the

obviously not a shortest-path

tree is indeed a shortest-path

the

tree. If there is an edge (v, w) such that v.SP + cost(v, w) <

of the shortest path
to x via the tree), then
the tree is
tree. It remains to prove that, if no such edge exists, then the

length

tree.

430

7.79

connected in the
the other hand,
vertices according
7.84

order, and

a topological

Find

We proved

to their

in

that

labels, if they are

a Hamiltonian path. On
will sort the
then any topological order

path,

run

the biconnected

will

a slightly

prove

the claim

path.

stronger theorem

(and

leave it

There is

Proof:

is not

one

components

to show

reader

are

in

graph and to
to the same

it implies

that

A.2

and w are two vertices of the same biconnected

Ifv
path between v and w is contained in B.

that

to the

if they

the exercise).

in

D Theorem

from

gives

7.9 that two edges are contained in a cycle if and only

component. Hence,we need only to remove c from the
a and b belong
whether
components algorithm to determine
in the graph without c.

same biconnected

path

consecutive

in Section

the

We

place

vertices with
topological order

for all

check,

then

graph. If they are, then the

if there exists a Hamiltonian

biconnected component

7.85

to SelectedExercises

of Solutions

Sketches

then
\302\243,

in

and to
component
can be connected

the existence of a

cycle

one simple

at least

obviously

contained

it has

come back
only

involving

more

path

v and

between

to leaveB and
through

through

component B, then

to come

another

(since

one articulation
than

back.
is

it

point).
one biconnected

But

simple

any

is
B
and two

in B.

If there

has

to leave

It

simple,
would

that

imply

which

component,

is

impossible.

7.87

Pick

articulation
point v. Remove v and its incident edges from the graph and
connected components of the resulting
graph. Then put back the removed
Consider
now each component separately,
and
edges into the appropriate components.
and so on.
perform the same procedure
by picking another articulation
point,
find

7.89

an

the

arbitrary

different

to
may be to try to modify the strongly connected component
algorithm
the
values
more
For
instead
of
DFS
we
numbers,
update
High
accurately.
example,
using
can use the currently
known High values. In other
words, when an edge (v, w) is
considered
for the purpose of computing
we use w.High instead of w.DFSin the
v.High,
However, w.High may be updated later (e.g., one of its other children may
computation.
have a path to a higher vertex). Updating
of
all the vertices that
used the previous value
A

first

w.High
Denote

same

approach

may be too costly. We use another

the largest

concerned. That
Hence,

it is

connected

DFS number

connected

strongly

is,

sufficient

component.

of

from v by v.RealHigh.
have the same characteristics
can reach a vertex w, then all

reachable

component

if any

approach.

them

for
v.RealHigh
compute
We
first use the strongly

to

All

in the
reachability is
them can reach w.
the

vertices

as far as
of

one representative
connected component

of

each

strongly

algorithm

to

of the root
connected
(this is the vertex with the highest DFS
component
number
in the component).
SG must be acyclic, so we have reduced the problem to that of
the
of this reduction is that it
The advantage
values of acyclic graphs.
finding
RealHigh
solves the problem with repeated
SGis acyclic, there
Since
we
before.
had
that
updating
are no back edges. Therefore, at the time the edge (v, w) (in SG) is used to update
the value of w.RealHigh
We leave it to the reader
is the correct final value.
v.RealHigh,
for example,
to prove this fact \342\200\224
induction
is
not
trivial).
by
(the proof

construct the SCC graph SG.

of the corresponding strongly

Each node

in

SG

is marked

with

the

DFS number

Sketches of

7.92

Solutionsto

431

Exercises

Selected

to a vertex basis B. If v is not on a cycle and v has a nonzero

a vertex w such that (w, v) is an edge and w is not reachable
w belongs
from v. For B to be a vertex basis, either
to B or there is a path from some
vertex in B to w. In either case, however, there is also a path from a vertex in B to v, and
v can be removed from B. This
contradicts
the minimality of B.

a. Supposethat

v belongs

indegree, then

b. Sinceno

exists

there

is on a

cycle in an acyclic graph, it follows from part a that only the

0 can be in a vertex basis. But, it is also true that each vertex of
indegree 0 must be in any vertex basis, since there is no path leading to it. Hence, the
0 (we proved in
unique vertex basis of an acyclic graph is the set of vertices of indegree
Section
7.3 that this set is not empty).
vertices

vertex

of indegree

find the SCC graph

(see Section
Any vertex in a strongly connected
all other
vertices in this component. Hence,it is sufficient to choose
in each component. Thus,
a vertex basis consists of one vertex
(any
to a node of indegree
0 in the SCC graph.
component that corresponds
Given

a general graph

the SCC

shown,

7.95

now,

we

first

is acyclic.

tree with an
the tree as a rooted
both from
w.
Remove
parent

Consider

it to

graph

arbitrary

root.

7.9). As we have
component covers
one of the vertices

leaf

Take an arbitrary
*

from each

vertex)

and

match

become
(and
disconnected). Solve the resulting problem by induction. We need to prove that the edge
If M does
Let M be a maximum
(v, w) indeed belongs to a maximum
matching.
matching.
to w). If (w, w) is in the
not
contain
(v, w), then v is not matched (it is connected only
then simply replace it with (v, w). It is still a matching, and it has the
maximum
matching,
same cardinality.
7.99

its

a. A graph

must not

b.

include

we

vertex,

arbitrary

may

a vertex

tree

of degree0 to have
component that

up with a

end

the

all

other

'brothers\"

such a cover.
consists

that

When we remove an

of only

one vertex.

the graph contains only

vertices
of degree > 0, and we make sure
Only vertices of degree 1 pose any
graph satisfies this condition.
one such vertex
problem. So, if the graph contains any vertex of degree 1, we remove
one
to
the
cover.
with
its edge and add this edge (which is a star graph
itself)
Only
by
is reduced
by 1. It is easy to see that this is now a valid
degree of one other vertex
reduction.

assume

We now

that

reduced

the

that

in combination with the maximum-matching

search
algorithm described in
Let's solve first the problem of verifying whether there exists a maximum
such that all of its edges have weights <x (for some given x). This problem can
matching
be solved by removing from the graph all edges of weight > x and applying the maximumin the remaining
has the
to see whether the maximum
matching
graph
matching
algorithm
in the original graph. The cost of this
of edges as the maximum
same number
matching
is
is dominated by the cost of the maximum-matching
verification
algorithm, which
for weights.
0(^fn m). There are m edges in G, so there are at most m different
possibilities
We can now use binary search to find the smallest x such that x is a weight of some edge
and
is a maximum matching
there
such
that all of its edges
have weights <.v. (For an
and Tarjan [1988].)
with a lower time complexity, see Gabow
algorithm

7.100 We use

binary

7.10.

Section

7.104 We use divide

construct a chain
same

way

as

conquer.

as required

regular

procedure. The

10.18.

and

similarity

We divide the players into two equal-sized

groups and
each group.
We then merge the two chains in exactly the
We leave it to the reader to verify the correctness of this
is not incidental; see Exercise
this algorithm to mergesort

for

merge.
of

432 Sketchesof
7.106

Clearly,

to Selected

Solutions

at least

k colors

Exercises

are necessary, since

all

edges

incident

differently.

We

will show

that

k colors

are also
such that

to the same

vertex

must

be

use divide
the subproblems
the problem
are the same as
but we have to divide
and conquer,
we have to ensure that
the
If we divide into two smaller subgraphs,
the
problem.
original
that
are
a
2.
in
are
and
of
This
is
each
vertices
of
all
they
power
subgraph
equal
degrees
a circuit exists, since all degrees
circuit of the graph (such
achieved by finding an Eulerian
the circuit and divide the edges into two groups by alternating
are even). Then, we traverse
between the two groups (i.e., the first edge goes to the first group, the second to the second
defines a subgraph with all the
group
group, the third to the first group, and so on). Each
of
the
The
and
one-half
vertices
original edges.
degree of each vertex in each
original
a
since
whenever
an
vertex is put in one group, the
is
fc/2,
entering
edge
exactly
subgraph
is leaving
the vertex) will be put in the second group.
next edge (which
The rest can be
We now have graphs with fewer edges and, more important,
smaller
done by induction.
but we use distinct
We color all the edges of the two subgraphs
colors
separately,
degrees.
The complexity
of
for each group. Therefore, we can just put the two colorings together.
the
for
Eulerian
which
takes
linear
this algorithm is dominated
circuits,
by
algorithm
finding
but which has to be performed
for each subgraph. The number
of stages of this
time
algorithm is log2fc (since k is halved at each stage); hence, the total running time is
is k can be
O (| E | log k). (It is interesting to note that every graph whose maximal
degree
edge-coloredwith at most k + 1 colors; see, for example, Chartrand and Lesniak [1986].)
colored

sufficient.

7.108 A

We

want to

common
from students'
a
examinations) is to consider
approach to this problem (judging
root of the tree (any vertex can serveas the root), to find minimum vertex coversfor all the
subtrees below the root, and then to include
the root in the vertex cover unless
all its
in the vertex coversof the subtrees.
This seems like a good
children are already
included
since the root must indeed
to the vertex cover if any of its children does
approach,
belong
not
vertex
covers (see Fig. A.l).
However, there may be many different minimum
belong.
For a given subtree, it is possible that there is a minimum vertex cover that includes its root
and one that does not include the root. Therefore, this approach
does not necessarily lead to
minimum
vertex covers (unless more precautions
are taken).
A different
can be
reduction
achieved by considering
a leaf first. The edge from a leaf to its parent must be covered, and
can be coveredby either the leaf or its parent. There is no advantage
the leaf,
to choosing

Figure A.l

tree

with two

minimum

vertex

covers.

of Solutions to

Sketches

and

the

incident
to have

guaranteed

More precisely,

else.

if we

Therefore,

parent.

its

all

433

there is a minimum
vertex
cover that
choose the parent (of any arbitrary
remove it
leaf),
edges, and then solve the remaining
problem
by induction, we are
a minimum vertex cover. It is not difficult to implement
this algorithm

not cover anything

it does

since

includes

SelectedExercises

in linear time.

7.109

is easy to see that the weight

number of edges(sinceall edges
the cover). A cover that includes

It

We

cover.

weight

Solution 1: Perform
level number according

(even level

numbers

must be

that

show

will

cover in this case must be equal to at least the

covered and each contributes at least a costof 1 to
exactly one vertex from each edge is thus the minimumsuch a cover can always
be found.

of a vertex

v as the root, and assign each

with an arbitrary
vertex
distance from v. Now add all vertices with odd level
will do just as well)
will be
to the vertex cover. The result
connects
vertices
from
two
so
them
one
of
levels,
adjacent
edge
a BFS

to its

cover, since every

level number.
Furthermore,

each

edge is covered by

exactly

one vertex,

so

the

vertex a

numbers
a vertex

has odd
cover

is

minimum.

Solution 2: The induction

< n vertices.

trees with
the

Let the

tree.

problem

for

resulting

vertex

have

v and

be w,
again

is still valid;
include v in
otherwise,
cases the cover of G is minimal.
But, the
of G must be at least 1 more than the minimumextra
edge must be coveredby either v or w (and it costs 1
of w is increased). We achieve
this bound, so the
degree

to prove that
vertex
cover

since the
case, since the
cover is minimal.

of G',

in either

more

7.114

minimum-weight

cover

weight

leaf

vertex
adjacent to v
induction.
Consider
by
vertex cover of G', then
We

know how to solve the problems for all

remove it (and its incident edge) from
and let G' be the remaining
tree. Solve the
the original
tree G. If w is used in the

states that we

arbitrary

only

G'

minimum-weight
the cover for G.

size of a

hypothesis

Pick an

this

cover

in both

vertex
of G. Denote the set of vertices adjacent to v by N(v). If v
arbitrary
to the vertex cover,then none of the vertices in N(v) can belong to the vertex cover
all the vertices
Furthermore,
(since the vertices in the cover should be independent).
is
to
vertices
from
must
to
the
since
that
the
cover,
N(v)
only way to cover
adjacent
belong
if any two vertices of N(v) are adjacent,
then the edge between
those edges. In particular,
is continued until either it
be covered, and the procedure fails.
This procedure
them
cannot
is found. All the
case v cannot belong to the vertex cover, or a vertex
cover
fails, in which
the vertex cover that is
steps of the procedure are determined
by the choice of v; hence,
v that satisfies the conditions
of the problem. If this cover
found is the only one containing
v cannot belong to the vertex
is not too large, then we are done; otherwise,
cover. But, if we
vertex
all
must
determine that v cannot
to
the
then
its
vertices
cover,
belong
adjacent
belong
to the cover, and we can apply the same procedure.

Let v be an
belongs

7.116

represented by two numbers (/,, r7) (the left and right endpoints). Sort the
in the sorted order
to the r,s. Denote the intervals
(i.e.,
by /,, /2,..., /\342\200\236
is a maximum
the right endpoints). We claim that
there
r, is the minimum
among
To prove it, take any maximum independent set and
/,.
independent set that includes
the interval /, in it such that t'j is minimum among all other right endpoints in the
consider
with
But
that implies that /, does not intersect
set. Since r, is the global minimum,
r} >r,.
in the set (except possibly
the
with /,); hence, /, can replace /,, and
any other interval
follows
set. The algorithm
modified set is still maximum
independent
directly by induction,
because we know how to handle /1.

Each

interval

intervals

/, is

according

Sketchesof

434

7.119 a.

Let

v,,

matrix

Exercises

to Selected

Solutions

4 (in that order). Considerthe adjacency

v2, v3, and v4 be a cycle of length
is 0 (i.e., a vertex
is not
that the main
of the graph, and assume
diagonal

Is
to itself).
The rows corresponding to v, and v3 both contain
some
a
of
is
characterization
a
of
to
and
This
sense,
is,
v4.
v2
cycle
corresponding
is symmetric
since the graph is undirected.)
We can
4. (Notice that the matrix
length
look at all pairs of rows, and for each pair check its intersection. In other words, for each
There
pair of vertices, we check all the other vertices that are adjacent to both of them.
contains
at least two vertices. Any row
is a square if and only if any such intersection
a row and operation) can
be performed
in linear time.
intersection
(which is basically
There are O (| V \\2) pairs of vertices, so the overall running time is O (| V |3).

considered

b.

adjacent

representation. First sort all the edges in all the lists. Then
between two rows described in part a can be performed
in
number
of edges incident to the two corresponding
vertices.

Use an adjacency-list
the intersection

that

proportional to the

7.120 Compute
vertices,

squares

the

and

maximum

number

show

it may

may take more than

Chapter

8.6

counterexample

The

that

scan even though

is given
it

is on

the

of squares that
be more than
\342\200\242
(| V | | E |) time.

in

Fig.

A.2.

be contained

can
0(\\

The circled

\\

\342\200\242

point

\\E |).

will

in

show

time

with | V

a graph

Therefore, just

listing

be removed

Graham's

by

hull.

vertices of the polygon in a cyclic order according

to the angles of the line segment
they make with q (and a fixed line). Then scan these vertices in that order. Start with an
of edges of the polygon
that intersect with the
arbitrary vertex v, and determine the number
half-line
8.2. Then,
passing through q and v using the algorithm presented in Section
whenever a vertex w is visited, one can in constant time determine whether
the line through
q and w intersects one more edge,one lessedge, or the same number of edges.

8.11

Sort the

8.16

Divide
the plane into vertical columns of width
d starting
among the points and ending with the largest x coordinate.

Figure A.2

counter

example

with the
Find

for Exercise

8.6.

x coordinate

minimal

the points

with

\\

all

the minimal

Sketches of Solutionsto

435

Exercises

Selected

in each of the columns.

This can be done in time 0(n +Xld)
contains a given point can be found in constant time. Apply
a
similar to Graham's scan to all the minimal
and then to all the maximal
points
procedure
can easily be connected at both ends to
in two convex paths, which
points. This results
form a convex polygon
the convex hull of the points,
C. C is not necessarily
but it is easy to
show that the distance of any point from the set outside
the hull to the hull is at most d.

and

y coordinates

maximal

since

column

the

8.17 The gift-wrapping

can be

algorithm

observation

The necessary

points.

that

the

Thus,

here

requires

algorithm

comparisons to

applied to this problem without

each point is compared against

additional

is that

to find the

comparisons

all
first

complexity.

other

vertex

remaining
of the

hull,

and so on. After the hull is discovered, all its vertices

are marked with the appropriate depth, and removed. The same algorithm
continues.
The
first vertex on it requires
n-k comparisons (where k - 1 is the number of vertices removed
so far), and so on. Overall, O (n2) steps are required.
n-2

8.22

8.26

the

find

The only difficult

the segments with

case is

We strengthen

induction

second,

intersections between all vertical line segments and

or can be handled
(the other casesare either
symmetric
of
8.6). We use the same approach as the intersection algorithm
by the algorithm in Section
Section 8.6. The segments
are sorted according to the x coordinates
of their endpoints. The
is performed
in the same way.
A segment
with 45-degree angle
is
line-sweep algorithm
is meeting the sweeping line, and is deleted when
inserted when its left endpoint
its right
line. We now have to find intersections between a new
the sweeping
endpoint is meeting
vertical
line and several candidate
with 45-degree angles. This is done as follows.
segments
For each segment
of the full line
with
45-degree
angle, we compute the intersection
the segment with the x axis. We use this value when we perform the range
overlapping
is the intersection between
The range is set as follows.
The left point of the range
queries.
the line with 45-degree angle
that contains
the top endpoint
of the vertical line and the x
between the line with 45-degree
axis. The right point of the range is the intersection
angle
contains
line and the x axis. We have converted
the
that
the bottom endpoint of the vertical
the same.
problem to a one-dimensional
range query, and the complexity remains
the

of finding

that

angle

45-degree

hypothesis a

We know how to

Induction
other

hypothesis:
intervals among

endpoint among

a set of < n

intervals,

the intervals

all

mark

and how

to find

contained

the largest

in

right

them.

interval;

left endpoints. Assume that we solved the

their
If its right
intervals, and consider the nX\\\\ interval.
it is not contained in any
is larger than
the largest right endpoint so far, then
we update the largest
Otherwise, it is contained in another
right endpoint.

interval;

we mark

We first sort the intervals

problem
endpoint

8.28

little.

The

for

the

according

first (leftmost)

to

-1

it.

of two rectangles whose edges are parallel

time. Furthermore, either this intersection is empty
solve the problem in linear time by intersecting one

intersection

constant
can

to the
or it
rectangle

in
axes can be computed
is a rectangle. Hence,we
after the other in any

order.

8.31

We give
slabs and

only a rough sketch of a solution.

to intersect the slabs separately.

idea is

each polygon into

according to their x
This sorting can be done in linear time, because the cyclic order of the vertices
coordinates.
in each
polygon is known. We associatewith each vertex a vertical line, and we use these
The

First,

basic

we sort

all

to

divide

the vertices

of Solutions to

Sketches

436

slabs. An
because each slab
into

intersections

slabs, as in shown

in

Fig.

a polygon

in

Since we know

A.3.

only to

compute intersections
two slabs can be computed
intersection
between
then
assemble
has at most four edges. We can
need

between

disjoint
in constant time,

the corresponding

time.

linear

two triangles in constant

is a convex polygon. We can intersect
An intersection of triangles
in linear
time
time, and we can intersect two convex
(Exercise
8.31). Thus, a
polygons
that divides the set of triangles
into two sets, computes the
divide-and-conquer
algorithm
intersects
the two resulting
of all triangles in each set recursively, and then
intersection
has a running time of O (n log n).
convex polygons

8.34

Chapter

One example

The method

is w = 15.

(jc15 = (((x2-x)2)-x)2-x).
(jc15 =

9.10

into

order of the vertical

sorted

pairs of

9.2

two polygons
lines, we

to divide the

lines

vertical
the

Selected Exercises

It

((jcjc)2jc)3). See

We use the

notation

the following

types of

of Section
which

introduced

presented

in

Section

to

compute

x]5

possible

p. 443]

1981,

[Knuth

definitions,

products

is

for a detailed

9.2 requires 6
with only 5

discussion of this

multiplications
multiplications

issue.

(B +C +D + F +G)X. We make
in the four
to
the
seven multiplications
correspond
directly
in Section 9.5.2: z, = b(e+f\\ z2 = c(g+h),
z3 =

9.5.2. We need to

compute

We
z4 = (a-h)e, z5 = (a -c)(g -e\\ z6 = (d-c)h, and z7 = (d-b)(f-h).
now
B contributes [z,, z,, 0, 0] (we write it
look at the contribution of each of the matrices.
as a row matrix instead of as a column matrix for convenience); C contributes [0, 0, z2, z2];
D contributes [0, z3, -z3, 0];F contributes
[0, z6 + z7,
[z4, 0, z4 +z5, 0]; and G contributes
= Z\\ +
we have p = Cu = z, +z4, r = C2J
0, z6]. So,overall,
z3+z6 + z1, s = C,i2 =
+ Z5, and t = C2,2 =z2 + z6. There
are 18 additions and 7 multiplications.
Z2+Z3+Z4
(c-b)(e+h),

9.16 The hard

that

part

it does

of this

problem is proving

by induction.

It

is easy

to

that
verify

such a representation always

exists.
We prove
the base case. Let n > 2 be an integer, and let

not larger than n (i.e.,F(k)<n

number
F(k) be the largest Fibonacci
claim that F(k)>n/2,
since otherwise
F(k+
\\) = F(k)+F(kcan
be
as
a
sum of at most
hypothesis, n-F(k)
represented

Figure

A.3

Intersection

of two

convex polygons

and F(k +

1)<n.

By

log2(w -F(k))

by the slab

method.

1)> n).

the

We

induction

Fibonacci

Sketches of

numbers.
To
that

9.19

since

But,

the numbers

keeps
find

the

We denote

furthermore,

parts of the

four

the

437

Exercises

of n-F(k)
still
adding F(k) to the representation
- F (k)) + 1 <
it is easy to see that
\\og2(n
log2w.
all Fibonacci numbers until we reach one
we need, we compute
to n\\ we can then follow the proof given
here.

than or equal

second

the

Selected

n-F(k)<F(k),

distinct;

representation

is larger

Solutionsto

by a2,

polynomial
as shown in Fig.

first

by a,, b,, c j,
We can illustrate

polynomial

b2, c2, and

d2.

and

d,,
the

and the four

parts of

using a 4x4

problem

A.4. This table is similar to the 2x2 table that was given in Section
9.4. Each entry in the table corresponds to a product of two parts of the polynomials.
We
of each entry, only
do not have to compute the value
the sum of values in each
of the
in the figure. Each of these
to the coefficient of a
corresponds
diagonals shown
diagonals
in the product of the polynomials.
The following 9 products are sufficient
certain
to
degree
\342\200\242
\342\200\242
(1) a, \342\200\242
(a 2 + b2), (4)
a2, (2) b, b2, (3) (a j + b,)
compute the values of all the diagonals:
+ rf2), (7) (a 1\"+c1)-(tf2
+ 6-2), (8) (/>, +</,)\"(*2+</2),
c, -c2t (5) dx -d2, (6) (c,+rf,)-(c2
and (9) (a\\ +b\\ +c'i +d\\) -(a2+b2+c2+d2).
We leave it to the reader to verify that all
from these 9 products. The corresponding
recurrence
relation is
diagonals can be computed
T(n) = 9T(n/4)+ 0(n), which implies that T(n) = O(wlog9') = 0(wlog3), which is the same as
of the algorithm that divides
each polynomial into two parts.
Is that
the running
time
In other words, what
a good reason for it?
are the
equality coincidental, or is there
similarities between the two algorithms?
table

9.23

assumes
of multiplication
that
(i.e., it assumes
algorithm
commutativity
If
we
use
as
the
base
of
the
we
have
to
be
able
recursion,
x).
-y =y
Winograd's
algorithm
matrices for elements, but we cannot do that because
matrix multiplication is
substitute

Winograd's
x
to
not

9.29

commutative.

on n. The case of n = 1 is trivial. Assume that we know

\342\200\242
\342\200\242
and consider n + 1. Denote by M[i.J] the product M, x \342\200\242
xMj.
+
+
is
first
and
\\..n
to compute
to
A/[/+
1], for
M[\\..n
1]
compute M[\\..i]
to multiply
the two products. We can find the value of /
some (as yet unknown) /, and then
to
leads
to the minimum
cost
We know the best way
that
by trying all possibilities.
However, we need to know how to compute M [i + 1 ..n + 1 ].
compute M [ 1../ ] by induction.
To do that, we strengthen the induction
hypothesis.

We solve the
the solution
The best way

problem

for

Stronger

by induction

n > 1,

induction

for all 1 <i

hypothesis:

We

know

the best

way

to compute

M[i.j]

<j<n.
ax

bx

cx

dx

ai

b2

Cl
d2

Figure

A.4

Finding

the product of two

polynomials

by dividing them into four

parts.

438

Sketches of

to Selected

Solutions

Exercises

To extend

+ 1] for all 1 <i<n.

to n + 1, we need to compute
this hypothesis
We
M[i..n
solve this problem by yet another (nested) induction, this time on / in a reversed order. For
to compute
the product M[n..n + \\]\\ hence, the problem is
is only one way
there
/' = \302\253,
Assume
that we know the best way to compute M[i..n + 1]and consider
/ - 1. We
trivial.
now perform
the reduction. We check for each j, / <j < w + 1, the cost of computing
can
we know by the original
induction), the cost of computing
(which
M[i-\\..j]
and the cost of multiplying
we know by the nested induction),
\\..n + 1] (which
both
M[j+
minimizes
the j that
this cost. Overall, the two induction
We then choose
products.
processes correspond to two loops, the first of size w, the second of size n -/', and the inside
n

steps. The

of n-i

consists

loop

number

total

is

of steps

thus

which is

\302\243
XX#0),
k=\\ i=k j=i

0(n3).

10

Chapter

10.3 We show

the interval-containment

that

each

For

algorithm.

coordinate is -L,
interval is contained
maximal.

10.5

that corresponds to a university into two vertices. Connect

students who were admitted
to that university. This is a
problem.
vertex

each

Split

and

problem can be solved by using the maximal-points

interval /y = (Ly,/?7), we define a point in the plane such that its x
its y coordinate is Rj. We leave it to the reader to verify
that an
in another
interval if and only if the corresponding point
is not

vertices to
matching

all the

both

of those
bipartite

regular

10.12 The simplest

solution
to this problem is by reduction.
We construct a new graph H with two
w of G. We call w{ the tail vertex of w, and w2 the
w{ and w2 for each vertex
head vertex of w. The edges of H are the following.
For each edge (w, w) in G, we add the
0
m
with
cost
to
H.
In
other
we
make
all edges go from head vertices to
words,
i)
edge (vv2,
tail
vertices.
In addition, we add an edge (u\\,u2)
with cost c(u) for each vertex u in G.
The problem
becomes
the regular single-source shortest-paths
from v2 to all tail
problem

vertices

vertices.

10.15 The

involves only one

counterexample
to
is
maximize,
jc, and the only
try
simply
since there is no maximum
problem has no solution,
simplest

we

10.17

The

linear

solution

10.20 To compute
its

size.

changed

every organization to

easily,

donate

but

two

The objective function,

are x<0
smaller than 0.

constraints
number

it is much

easier

to

the

and

which

x*0.

that the

notice

This

best

its maximum.

the square of a matrix, we indeed need only

five multiplications
of two arbitrary
matrices,
However, these are multiplications

matrices of half
10.24 We show
numbers

can be

program
is for

x.

variable

of matrices
not

squarings

half

of

size.

such an algorithm
to sort. Given a sequence x^x2
xn of distinct
sort, we associate a point pt with each jc, such that all points lie on
a fixed line and the distance of point
pt from a fixed origin is jc,. It is easy to see that the
minimum-cost
on the line \342\200\224
namely,
spanning tree must connect each point to its neighbors
the tree is a chain.
The tree has two leaves, which
elements
are the maximum and minimum
of the sequence. If we know the minimum-cost spanning
tree, then we can find the sorted
order of the points in linear time as follows.
the points,
First, we find the minimum
among
how

that

to use

we

wish to

Sketches of

Solutionsto

Selected

439

Exercises

say xh The chain defined by the tree gives us the sorted order, and we can follow it in linear
time. Thus, we have a lower bound of Q(n \\ogn) for this problem under the decision
tree
for sorting was proved).
model (which is the model under which the lower bound

11

Chapter

11.3 O

(nk)

(and
11.11

function in
is a polynomial in w, but it is an exponential
is
not
a
this
as
be
as
w),
polynomial-time algorithm.
may
high

k. Sincek

is part of

the

input

the regular vertex-cover problem. Let G =(V, E) be an arbitrary

from
let
and
U be the set of vertices of odd degree in G. We modify
G by
graph,
are connected
to each other (in a triangle). We
adding three new vertices, x, y, and z, which
also connect
x to all vertices in U. It is now easy to prove
that the modified graph
has a
vertex cover of size K if and only if G has a vertex cover of size K - 2.

We use a

reduction

undirected

11.14

The problem
subgraph in

is obviously

in NP,

time.

polynomial

since

we

can

We use a reduction

guess
from

the subset and

Let E-C

3SAT.

check
\\

C2

its induced
Cnbc

a graph G=(V, E) with

instance of 3SAT. We construct
4w + l vertices as
four vertices, one associatedwith the clause itself,
follows.
For each clause C,, we include
and the other three
associated
with the corresponding variables.
The
four vertices
associated with a clause are fully connected
to one another. We also connect
any two
vertices in the graph that are associatedwith x and x for any variable x. Finally, we add one
r which is connected
to all vertices associated with
additional
vertex
variables
(but not to
with the clauses themselves).We claim that the graph has a subset
vertices
associated
the
of size In + 1 that induces an acyclic graph if and only if the expression is satisfiable.
an arbitrary

then we can find a consistent

truth assignment satisfying
the expression is satisfiable,
of the vertex r, the n vertices corresponding to the
clause. The subset will consist
and one vertex for each clause corresponding to the variable
that satisfies the
clauses,
an
clause (ties are broken
This
subset
induces
arbitrarily).
acyclic
graph since r is
connected
are not connected if they belong
to a
only to variables, and two variables

1. If

every

consistent truth

assignment.

exists a subset

S with In + 1 vertices
that induces an acyclic
graph, then we claim
include
r and exactly two vertices from every clause. Indeed, S cannot
more than two vertices from one clause, since such vertices are connected
(and thus form
to x
a triangle). Also, since S includes
r, it cannot include two vertices that correspond
for E.
and x for some variable x. Therefore, it is possible to obtain a truth assignment

2. If there
that

S includes

from
3SAT.
Let C =(x+y+z) be a clause in an arbitrary 3SAT
11.16 We use a reduction
We
with
the
three clauses (the o,s are all new variables):
C
problem.
following
replace
(x + a\\ + a2), Cy + 0 3+tf4), and (z+a5+a6). In the l-in-3-SAT
problem, exactly one of
We want to guarantee that
in each of the three clauses above must be satisfied.
the variables
that
We do that by adding
more clauses that guarantee
at least one of jc, y, or z is satisfied.
one of a2, tf 4, and a6 is
and no more than
no more than one of a \\, o3, and a5 is satisfied,
are (fl,+fl3+fl7), (a^-\\-a5-\\-a^),
satisfied.
The
clauses
(a2 + a4+a\\0),
(a5 + a\\+a9),

(a4+a6+au), and

problem

We ^eave tne

verification

problem. Let G = (V,

to the

reader.

E) and k be an arbitrary instance

problem to this
clique
R such that the clique
We need to convert
G into a regular graph
problem.
add
a clique problem for R. We cannot simply
for G can be solved by solving
to G until G becomes regular, because this would potentially increase the sizes of the

11.19 We reduce the

of the clique
edges

(a6+a2+an)'

440

let

We must

of G.

cliques

d be the

Let

Selected Exercises

of Solutions to

Sketches

maximal

add

vertices

degree

of G

it is

number of vertices of G.
and connect each of
vertices

be the

d-d(v)

to G

and edges
if

For
them

even or
each

the

vertex

to v.

no new

that

such

maximal

v of G

with

cliques are

plus

degree

degree

The total number of

formed.

and

otherwise,

we add

d{y)<d,

is

vertices

additional

= dn

dn-^d(Vj)

2\\E

This number

|.

is even since

we

d to

chose

be even.

that

Notice

vertices now have the same degree d, and that no new cliques were added
to only one vertex). The only
is that the new
each new vertex is connected
problem
more
vertices have a degree of 1. We can change their degrees to d without
introducing
in
between
them
the
We
divide
set
of
new
the
following
way.
cliques by adding edges
vertices to two equal sets. We then connect each vertex of one set to exactly d - 1 vertices
thus induce a bipartite graph, which
does not contain
of the other set. The new vertices
this
cliques of size > 2. We leave it to the reader to verify that it is possibleto construct
all the original
(since

bipartite graph

11.25

If

we

then

of

in

way.

determine whether there exists a Hamiltonian

path
determine whether there exists any Hamiltonian

could

could

we

vertices.

Therefore, a

polynomial-time
in

Section

11.2, but

algorithm

polynomial-time

for

algorithm
algorithms

polynomial-time

defined

the desired

the

Hamiltonian

for all NP
it

problems.

path
Notice

for

with specified
path by just

this

problem,
that

trying

all pairs

problem leads

which

this is

end vertices,

in

not a pure

turn

to

leads to

reduction

as

is sufficient.

11.29 We use a

reduction
from the vertex-cover problem. Let G =(V, E) and K be an arbitrary
instance of the vertex-cover
We construct a directed
G' by replacing every
problem.
graph
vertex v in G with two vertices v, and v2 connected by a directed edge (v,, v2). We replace
each edge (v, w) of G with two directed
(w2, Vj) and (v2, h^). We now claim that
edges,
G' contains
a feedback-edge
set of size K if and only if G contains a vertex cover of sizeK.

11.33We

sketch
a reduction from SAT. Let v be a vertex in G and let the edges adjacent
to v be
the edges of v,
(in the cyclic order) e,, e2,..., ek. The Eulerian tour defines a pairing
among
such that the consecutive edges in the tour used to enter
and leave v are paired. By the
conditions
of the problem, an edge et can be paired
either
with e,-_j or with ei+] (additions
and
subtractions
are done modulo k). The main observation
is that, if e,- is paired with e/+1,
then
the pairings
for all other edgesof v are fixed (ei+2 must be paired
with e/+3, and so on).
The same is true if e,- is paired with e,-_i. In other words, there are only two ways of pairing
the edges of v. We will associate
such
a vertex with each variable in the SAT expression,
that the value
of this variable in the truth assignment will correspond
to the way the pairing
is done for this vertex in the tour. We have to make sure that the truth assignment is
consistent
(i.e., the values of x and J are complementary), and that every clause is satisfied.
We will have one vertex
connect
vv for every variable jc, and one vertex v^ for J, and we will
them such that a pairing for vt forces a certain
values
the
truth
associate
for
We
Vx.
pairing
three vertices
Then, for each clause (x +y + z), we connect the corresponding
accordingly.
such that at least one of the pairings that is associated with one the variables being true must
be taken. We leave the details
to the reader. The planar
(which are not straightforward)
case, which is more complicated, is proved
in Bent and Manber [1987].

Chapter 12

12.13

We

simply

use

the doubling method. In the first step, only

one processor
Px
A [ 11 to A [2]. In step /, there are 2'\"1 participating
processors,

copies

participates: P,
and

they

copy

of Solutions to

Sketches

the

12.16

array

This

' to A
]

A [1..2'

To improve the

+ 1..2'].

be solved

can

problem

[T

in

exactly

Fig.

A.5.

the

same

use Brent's

efficiency,
as

way

441

Selected Exercises

lemma.

problem of

the

the

finding

maximum.

12.17 The circuit

=8 is given

forw

in

12.20 Assign one processorPti

to every pair of elements A [i ] and B[j\\. Pti first compares

A [i ] to
+
B
1
If
is
A
to
A
between
B
and
B
+
1
then
[i
]
].
[i
]
[k
]
[k
],
compares
[j
Pik will
that out and conclude that A [/ ]'s place in the final array is in the (/' + ifc)th position.
The

B [j], then
find

same procedure can

12.25 We

to B.

be applied

an elegant solution using

parallel
prefix (the problem can also be solved directly).
on
the
the
Mark
compute
parallel prefix
array with the + operation. The prefix
value
of each record whose
value is 1 will be equal to its place in the compacted
Mark
We can then complement
to do the same for the other elements.
the Mark array
array.
Once all the indices are computed, the actual movement can be done in one (parallel) step,
sketch

We

first

since there
12.27

If

we

are no

associate

index in the

a processor

appropriate

12.32 We

cannot

each

record,

successor.

of its

then

in

step each

one

Each processornow

processorsif

root,

and

for

If the

RL and

height of
it down

sends

height

binary

a particular

but a delay

step,

the numbers

We use induction.
root, which then
algorithm
complete

idle

It is true
number separately.
for one processorcan
do not arrive there fast enough.

on each

concentrate

most one more

in

with

field

processor can

knows

write

its predecessor

its
in

list.

the

12.33

conflicts.

h,

the

it

is 2, then the
can

be added

left

leaf

each

more

sends

incurs at
delays at other

number

its number x]

to jc2. Suppose

that

we

to

have

the

an

height h + \\. (We will consider only the case of

the algorithm to the general
case.) Let R be the
modify
and right children. We call the sum over all the leaves

and consider
It is

easy to
root's left
subtree the sum of the
trees.

tree

where

that

cause

RR be the

Figure A.5

subtree.

parallel-prefix

A straightforward

circuit.

solution is

to

solve

the

442

Sketches of

Solutionsto

Selected

Exercises

then to broadcast the sum of the left

problem separately for the left and right subtrees,
Each leaf in the right subtree simply
adds the
to all the leaves of the right subtree.
subtree
so far. The problem with this solution is
sum of the left subtree to the prefix it computed
relation is T(h + \\) = T(h) + h. We
time is 0(h2), becausethe recurrence
its running
that
there
is
no
need
to wait until the left subtree has
that
this
solution
can
by
noticing
improve
The right subtree needs to obtain the sum of the left subtree, and
its computation.
finished
from RL is that it
this sum can be available at the root at step h + 1. Thus, the requirement
from RR
receives the sum of all its descendants and sends it up to the root. The requirement
This leads
the sum from the root and sends it down to all its descendants.
is that it receives
starts
to the following rules: (1) each leaf
by sending its value
up (we have to
directly
2
since
the
root
needs
to
know
the
solution
for
the sum); (2)
height
slightly,
modify
simple
from below, add those values
and send them up;
internal
the
nodes, when receiving values
(3) the internal nodes, when receiving values from above, send those values down to both
children; and (4) the internal nodes also act as the roots of their own subtrees and send the
to their right children.
We leave it the the reader
value they receive from their left children
and that its running
time is 2h.
to verify that this is a correct algorithm
first
that we are interested only
in computing
a 12 b2,
x{. Processor P2 computes
is its contribution to the value of jcm and sends it to P3\\ processor
P3 adds to the
value it receives a^-b^
and
forwards
that value to P4, and so on. After n - 1 steps, P, will
from P\342\200\236
the value of jc, -au
receive
-bly and it will be able to completethe computation.
to
all
the
we
Now,
jc,s,
compute
pipeline this process. In the first step, each processorP{
\342\226\240
index
calculation
are done modulo n), and sends it to Pi+l. In the
(all
b,
computes
a{i.\\y,
7-1
adds to it o(/_y)/
y'th step,
P, receives the value
bh and sends it to Pi+l
\302\243
a{i_j){i_k) \342\200\242\302\243,_*,

12.37 Suppose
which

k = \\

(again,

all index

calculations are modulo

n).

12.40

This problem can be solved directly, but it is easier to use a reduction

to matrix
We
first
use
the
the
to
o
build
a
S
such
matrix
that,
multiplication.
permutation
permutation
for each column z, only the entry at the o(z )th row has a value of 1, and all other entries at
that
column
have a value of 0. We can easily build this matrix
it to the
and distribute
in n steps. Permuting all the rows according
to the permutation o is
appropriate
processors
the
same as computing the product S A, which
we already know how to do in O(n) steps.
the columns
can be done in the same way with the permutation matrix
Q, except
Permuting
that the product is A Q.

12.42

The straightforward induction solution is to compute

the parallel prefix in each half-cube
as the
separately in parallel, then to broadcast the largest prefix in one half-cube (designated
smaller one) to the other. (The base case of one dimension
Broadcast in a dis trivial.)
dimensional cube takes
d steps.
time
of this algorithm satisfies the
Thus, the running
recurrence
that T(d) = 0(d2). We
relation: T(d+ \\) = T(d) + d, which
following
implies
can
this algorithm by strengthening
the induction hypothesis. We assume that
improve
the sum of
its corresponding prefix, but also computes
every
processor not only computes
all numbers
in the cube. The base caseof one dimension
is still easy: The two processors
their values. Given a (d+ 1 )-dimensional
cube, we divide it into two dsimply
exchange
dimensional cubes, and solve the problem by induction
But now, we do
in both of them.
the
not
have to broadcast the sum of the left cube.
Since each node in the left cube knows
sum, it can send the sum in one step to its neighbor in the right cube. All the prefixes can
thus be computed
in one more step. Furthermore,
the sum of everything can also be
in one more step if the nodes in the right
subtree
send their (global) sum to their
computed

of Solutions to

Sketches

neighbors
implies

in the

left

subtree.

that T(d) = 0(d).

The

The

relation is thus T(d+\\)

indices to processor should

recurrence
of

assignment

SelectedExercises

443

= T(d) + 2, which
be clear from this

description.

12.48 Thereis a rich

subject of gossip as

to computer networks;
see,
The exercise
[1988] for a survey.
presents a relatively
simple
gossip problem. It can be solved by the pattern shown in Fig.
is the parallel
which
has
most
notable
A.6, called butterfly,
many other uses (the
=
shows
the
for
n
A.6
solution
8.
of
the
FFT).
Figure
computation
for example

Figure

literature

on the

Hedetniemi,

A.6

and

Hedetniemi,

The solution

of the

gossip

it

is related

Liestman

problem

(Exercise 12.48) for

n =8.

BIBLIOGRAPHY

Add'son-Vel'skii G. M.,
Aho

A.

\"An

of

the organization

for

algorithm

(1962), pp. 1259-1262.

A.

Aho

The

Data Structures and

E. Hopcroft, and J. D. Ullman,

MA, 1983.

and E. Szemeredi,\"An
M., J. Komlos,
ACM Symposium on Theory of Computing,
Parallel

Aleliunas

Sorting

Almasi G.

S., and
RedwoodCity,
and

D.,

Boston

Academic

Algorithms,

sorting

logn)

0(n

Addison-

Algorithms,

New York,

Press,

A.

G. Valiant,
Journal

matchings,\"

Parallel

Highly

1-9.

1985.

ACM

on

Symposium

pp. 60-72.

1982),

Benjamin/Cummings,

Computing,

1989.

CA,

L.

and

Gottlieb,

First

Annual

15th

network,\"

1983), pp.

(April

\"Randomized
communication,\"
parallel
of Distributed Computing, Ottawa (August

R.,

Principles

Angluin

of Computer

Reading,

Wesley,

S. G.,

and Analysis

Design

MA, 1974.

Reading,

Addison-Wesley,

V., J.

D. Ullman,

Hopcroft, and J.

J. E.

V.,

Algorithms,

Akl

Dokl., 3

A. V., and M. J. Corasick, \"Efficient

An aid to bibliographic
string
matching:
the
18
333-340.
ACM,
search,\" Communications
1975),
(June
of
pp.

Aho

Ajtai

M. Landis,

Y.

and

Math.

Soviet

information,\"

for Hamiltonian
\"Fast probabilistic algorithms
and System Sciences, 18 (April
of Computer

circuits
1979),

pp.

155-193.

Apostolico A.,

Z.

and

1985.

York,

E.

V. L.,

Arlazarov

of

construction

1970),

(May

pp.

M. D.,

Atkinson

multiplication,\"
Attalah

Algorithms on Words, Springer-Verlag,New

Combinatorial

Galil,

M.
technique

J.,

A.

the

closure of

transitive

I.

and
directed

A.

\"On economical

Faradzev,

graph,\"

Soviet

Math

Dokl, 11

1209-1210.
and

N.

\"A

Santoro,
Processing

Information

R. Cole,
for

A. Kronrod,

M.

Dinic,

and

designing

M.

T.

parallel

practical
Letters,

Goodrich,
algorithms,\"

for

algorithm

29 (September

\"Cascading
28th

Boolean

matrix

1988), pp. 37-38.

divide-and-conquer:
Annual

Symposium

on

445

446

Bibliography

S.

L., and

Auslander

Science, Los Angeles (October1987),pp.

of Computer

Foundations

Mech.,10(May

V.

\"On

Parter,

1961),

Communications

\"Sums of
15

divisors,perfect

(November

Schwab,

\"A

of the ACM,

20 (May

B.

and

the

in

and

J. Math,

plane,\"

pp. 517-523.

J. Shallit,
and
Bach E., G. Miller,
SIAM Journal on Computing,

Baer J.-L.,

graphs

imbedding

151-160.

and

numbers

of tree-balancing

comparison

factoring,\"

1143-1154.

1986), pp.

algorithms,\"

pp. 322-330.

1977),

for the weighted

Bar-Yehuda R., and S. Even, \"A linear time approximation
algorithm
2 (1981), pp. 198-203.
vertex cover problem,\" Journal of Algorithms,

BatcherK. E., \"Sorting

their applications,\"

and

networks

Spring Joint Computer Conference,(1968),pp.

J. L.,

Bates

L.

R.

and

\"Proofs as

Constable,

Programming Languages and Systems,

BavelZ., Math
Virginia,

Beame P. W.,

S. A.

Beckenbach

E.,

Bellman,

House,

R. E., Dynamic

S.

Bent

R.

and

Random

Library,

Bellman

New

An Introduction

Princeton
the

\"Finding

on Theory of

Symposium

pp. 113-136.

for division

circuits

Reston,

Company,

Publishing

1986),

and

related

pp. 994-1003.

to Inequalities,

New

Mathematical

1961.

York,

Programming,

and J. John,

W.,

ACM

Science, Reston

on

Transactions

ACM

1985),

Cook, and H. J. Hoover,

\"Log
depth
SIAM Journal on Computing, 15 (February

problems,\"

32nd

AFIPS

Proceedings

programs,\"

7 (January

Computer

for

Companion

1982.

in

307-314.

median

Press, Princeton, NJ, 1957.

University

In comparisons,\"

requires

Computing,

RI

Providence,

17th

(May

Annual

1985), pp.

213-216.
Bent

S. W., and U. Manber, \"On non-intersecting

Mathematics, 18 (1987),pp. 87-94.

Bentley

J. L.,

Eulerian

Programming Pearls, Addison-Wesley,

Reading,

circuits,\"

Discrete

1986.

MA,

F. P. Preparata, \"Approximation
and
Bentley J. L., M.G. Faust,
algorithms
hulls,\" Communication of the ACM, 25 (January 1982), pp. 64-68.
Bentley

J.

L.,

conquer
Bentley

J. L.,
1(1988),

D. Haken,

recurrences,\"
and B.

and

J.

B. Saxe,

SIGACT News,

W. Kernighan,

\"Tools

Applied

\"A general

method for

for printing

indexes,\" Electronic

solving

for convex

divide-and-

(Fall 1980),pp.36-44.
Publishing,

pp. 3-17.

for
Bentley J. L., and T. Ottmann,
\"Algorithms
intersections,\" IEEE Transactionson Computers,

reporting

C-28

and

(Sept.

counting

geometric

1979), pp. 643-647.

Bibliography 447
Berge C,

The

and Its Applications, John Wiley

of Graphs

Theory

and

New

Sons,

York,

1962.

Berge C, Graphs and Hypergraphs, North

Bertsekas D.

P., and

J.

Bitner

Englewood Cliffs, NJ,

Prentice-Hall,

Methods,

R.,

M.

E.

and

Communications of the
Blelloch

Parallel

Processing,

parallel operations,\"1987International

R. W. Floyd,
selection,\"Journal

V. R.

Pratt, R.

L. Rivest,

and System

of Computer

B.,

Bollobas

B., Combinatorics, CambridgeUniversity

Bondy J. A.,

S. R.

U.

and

on

Conference

Tarjan, \"Time bounds for

7
Sciences, (1972),pp.448-461.
R. E.

and

Course, Springer Verlag, New

An Introductory

Theory:

Bollobas

Graph

techniques,\"

pp. 355-362.

1987),

(August

Blum M.,

programming
1975), pp. 651-656.

18 (November

ACM,

Numerical

Computation,

1989.

\"Backtrack

Reingold,

as primitive

\"Scans

G.,

Parallel and Distributed

N. Tsitsiklis,

J.

1973.

London,

Holland,

Press,

Murty, Graph Theory with

1986.

Cambridge,

New York,

Elsevier,

Applications,

1979.

York,

1976.
time
and
Borodin A., \"On relating
6
1
977),
(December
Computing,
pp.

J. E.

A., and

Borodin

computation,\"

Borodin A.,

and

Problems,

Boyer R.
the

Brelaz

D.,

Burge

W.

System

Fourier

The Fast

the

Transform,

Combinatorics,

A., Introductory
Recursive

H.,

string

Complexity of Algebraic
New

York,

Communications

algorithm,\"

searching

Numeric

and

1975.

of

vertices

graph,\" Communications

of a

of the

1979), pp. 251-256.

22 (April

R.

Brualdi

to color

methods

\"New

\"A fast

30 (1985),

Sciences,

1977), pp. 762-772.

20 (October

E. O.,

Brigham

S. Moore,

J.

on

parallel models of
pp. 130-145.

and sorting on

merging

Computer and

I. Munro,
The Computational
Elsevier Computer ScienceLibrary,

S., and

ACM,

ACM,

of

Journal

SI AM

depth,\"

733-744.

\"Routing,

Hopcroft,

Journal

to size and

space

Programming

Prentice-Hall,
North

Holland,

Techniques,

Englewood

Cliffs, NJ,

New

1977.

York,

1974.

Addison-Wesley, Reading, MA,

1975.

Bussey
24

W. H.,
(1917),

Capobianco M.,
J. L.,

and

mathematical

induction,\"

American Mathematical

Monthly,

199-207.

and

J.

C. Molluzzo,

New

North-Holland,
Carter

of

\"Origin

pp.

M.

N.

York,

Examples and

Counterexamplesin

Graph

Theory,

1978.

Wegman,

\"Universal classes

of

hash

functions,\"

Journal

of

Bibliography

448

Computer and System

R.,

ACM,

17

M., and

K.

Chandy

\"An
S. Kapur,
algorithm
(January 1970), pp. 78-86.

143-154.

for convex

and S.

D.

Chand

1979), pp.

18 (April

Sciences,

J. Misra, Parallel Program

polytopes,\" Journal

A Foundation,

Design:

the

of

Addison-Wesley,

Reading, MA, 1988.

G.,

Chartrand

and

D.,

Computing,

Group,

minimum
R. E. Tarjan, \"Finding
5 (December 1976),pp.724-742.

S.T. Klein,

Fraenkel,

of

Proceedings

manipulation,\"

Montreal, Canada(1985),pp.
Christofides

&

Wadsworth

Monterey, CA, 1986.

A.S.

Y.,

Choueka

International

Wadsworth

Graphs & Digraphs, SecondEdition,

L. Lesniak,

and

G.,

Brooks/Cole,
Cheriton

Models,

CA, 1977.

Belmont,
Chartrand

Mathematical

as

Graphs

N., Graph

Y.

and

the

\"Huffman

Perl,

on

Journal

coding

bit-

without

ACM-S1G1R

Annual

Eighth

SIAM

trees,\"

spanning

Conference,

122-130.

Theory:

An

Press, London,

Academic

Approach,

Algorithmic

1975.
Christofides

\"Worst-case

N.,

problem,\" Technical

Cohen J.,

M.

and

down

\"Slowing

Annual

1984),

Foundations

on

Symposium

Industrial

Administration,

of Strassen's

fast

multiplication

6 (1976),pp.341-355.
networks

sorting

of

School

salesman

the traveling

PA, 1976.

the implementation

\"On

Roth,

Acta lnformatica,

algorithm,\"

ColeR.,

Pittsburgh,

for

heuristic

new

Graduate

Report,

University,

Carnegie-Mellon

of a

analysis

to

faster

obtain

of Computer

sorting

25th

algorithms,\"

Science, Singer Island

(October

pp. 255-259.

Cole R., \"Parallel

merge sort,\"

SI AM

on Computing, 17

Journal

1988),

(August

pp.

770-785.

Cole R., and U. Vishkin,

\"Deterministic
coin tossing and accelerating
cascades:
and macro techniques for designing parallel algorithms,\"
18th
Annual
Symposium on Theory of Computing,
(May 1986), pp. 206-219.
Berkeley

micro

Cook S. A.,

ACM

\"The

complexity

of theorem

Symposium on Theory of Computing,

Cook S. A.,

\"An

26 (June
Cook

overview

New

of computational

1983),pp.400-408.

S. A., and C. Dwork, \"Bounds

ACM
functions,\" 14th Annual
(May 1982), pp. 231-233.

on

the

Symposium

proving

York

proceudres,\"

(1971),

Annual

Third

ACM

pp. 151-158.
of the

complexity,\" Communications
time of parallel RAMs
on Theory of Computing,

to

compute
San

ACM,

simple
Francisco

Bibliography 449
CooleyJ. M.,and J. W.
Fourier series,\"Math.

\"An algorithm for the machine

19 (1965), pp. 297-301.

Tuckey,
Comp.,

Coppersmith D., and

\"Matrix multiplication via arithmetic

S. Winograd,
New
York
ACM Symposium on Theory of Computing,

Annual

19th

of complex

calculation

progressions,\"

1987), pp.

(May

1-6.
Culberson

on Theory

Symposium
G.

Dantzig

Princeton, NJ,
Dekel

route

and

Programming

10

with

C. Pang,

14

M. E.

and

IT-22

and A.

S. E.,

New York,

Dvorak, S., and

Journal,
Ebert

J.,

Science,

Computer

and annotation,\"

Networks,

two

IEEE Transactions on

in cryptography,\"

directions

\"New

Boston, 1983.

Birkhauser,

1976), pp. 644-651.

in connexion

problems

of Programming,

with

Prentice-Hall,

Dobkin D. P., and J. I. Munro, \"Determining

12 (1980),pp.255-263.
Press,

and

Engineering

Numerische

graphs,\"

1 (1959), pp. 269-271.

A Discipline

E. W.,

Dreyfus

Cliffs, NJ, 1982.

Englewood

Taxonomy

algorithms:

(June

\"A note on

W.,

Mathematik,
Dijkstra

SI AM

algorithms,\"

graph

1974.

of Programs,

Hellman,

Theory,

Information

E.

to

Applications

and

657-675.

Prentice-Hall,

\"Shortest-path

Dershowitz N., TheEvolution

Dijkstra

Press,

University

pp. 275-323.

(1984),

Diffie W.,

matrix

1981), pp.

Prentice-Hall, Englewood Cliffs,NJ,

N., and

Science, 6

Management

Princeton

Extensions,

\"Parallel

Sahni,
(November

Programming,

Theory

Graph

S.

and

DenardoE. V.,Dynamic

Deo

ACM

Annual

1985), pp. 205-212.

a newtwork,\"

through

17th

trees,\"

RI (May

1963.

E., D. Nassimi,
Journal on Computing,

Deo N.,

search

binary

187-190.

B., Linear

G.

of

in

Providence,

Computing,

\"On the shortest

B.,

(1960),pp.
Dantzig

of updates

effects

\"The

J.,

B.

M. Law, The
1977.

Durian,

and

Art

the

Theory of

linear time

\"Unstable

31 (1988), pp. 279-283.

\"Computing

Eulerian

mode,\"

trails,\"

Information

Cliffs, NJ,

Englewood
Theoretical

Computer

Dynamic Programming,

0(1) spacemerging,\"
Processing

Letters,

1976.

Science,

Academic

The

Computer

28 (June

1988),

pp. 93-97.
Edelsbrunner

Edmonds
network

H.,

J., and
flow

Algorithms

R. M. Karp,
problems,\"

in Combinatorial
\"Theoretical

Journal

of

the

Geometry, Springer-verlag,Berlin,

1987.

in algorithmic efficiency
improvements
19
ACM,
(1972), pp. 248-264.

for

450 Bibliography

Elliott

D.
F., and K. R.
Academic Press, New

J. L.,

Eppinger

\"An

A.

and

P.,

Szekers,

P., and

in geometry,\"

problem

J. Spencer,Probabilistic

Academic Press, New

in Combinatorics.

Methods

L., \"Solutio problematis ad geometriam situs

Academiae Scientiarum Petropolitanae, 8 (1736),pp.

Even

S., Graph

Even

S.,

and

space,\"

R. E. Tarjan,
Journal of the

\"A

Press, Rockville,

Science

Computer

Algorithms,

combinatorial

in

polynomial

pp. 710-719.

23 (1976),

ACM,

MD, 1979.

is complete

which

problem

Commentarii

pertinentis,\"
128-140.

N. Pippenger, and H. R. Strong, \"Extendible hashing

method for dynamic files,\" ACM Transaction on Database

\342\200\224
a fast

J. Nievergelt,

R.,

Fagin

Compositio

463-470.

1974.

York,

Euler

search trees,\"

in binary

deletion

and

(September 1983),pp.663-669.

combinatorial

\"A

Mathematica, 2 (1935),pp.
Erdos

of insertion

study

ACM, 26

of the

Applications,

Analyses,

Algorithms

Transforms:

1982.

empirical

Communications

Erdos

Rao, Fast
York,

access

Systems,

(September 1979), pp. 315-355.

Feng

of interconnection

\"A survey

T.,

networks,\"

14 (December

Computer,

1981), pp.

12-27.
Fich

F.

bounds for parallel

E., \"New

Theory of Computing,

Boston

FischerM.J., \"Efficiency
R. E. Miller and
Fischer M. J., and

of
J. W.

IEEE 12th Annual

on
Symposium
(October 1971),pp. 129-131.
M.

Fischer

Fischer M.

and

J.,

M. O.

in Complexity

arithmetic,\"

J.,

and

S.

L.

3 (1982),

Algorithms,

Rabin,

Floyd

R.
American

W.,

97:

Shortest

Mathematical

meanings
Society

and

majority

analysis of
August

transitive

Theory,

Ed.,

Computations,
153-168.

pp.

East

closure,\"
MI

Lansing,

of

complexity

R. M. Karp
a

1972,

York,

Automata

\"Super-exponential

Salzberg,
\"Finding
pp. 375-379.

and

Complexity

multiplication

and

of Computation,

'Assigning

matrix

Switching

Flajolet P., and J. S. Vitter, \"Average-case

Technical Report 718, INRIA,
France,
Floyd R. W., \"Algorithm
1962), pp. 345.

in

algorithms,\"

eds., Plenum Press, New

\"Boolean

R. Meyer,

A.

ACM Symposium on

Annual

100-109.

1983), pp.

equivalence

Thatcher,

15th

circuits,\"

prefix

(April

Presburger
1974.

SIAM-AMS,

among

algorithms

Journal

n votes,\"

and

data

of

structures,\"

1987.

paths,\" Communications

of the

to programs,\" Symposium on Appllied

(1967), pp. 19-32.

ACM,

5 (June

Mathematics,

Bibliography 451

the

M.

J.,

L.

systems,\" Proceedings of

the

54

IEEE,

1901-1909.
The

flow theory,\"

\"Network

R.,

Communication of

for selection,\"

bounds

computing

high-speed

\"Very

(1966), pp.
Ford

\"Expectedtime
165-172.

1975), pp.

18 (March

ACM,

Flynn

R. L. Rivest,

W., and

R.

Floyd

Rand

Monica, CA

Santa

P-293,

Corporation

(1956).
D. R.

R., and

L.

Ford

L.

Ford

Ford L. R.,

M.

S.

and

66 (1959),

Monthly,

J.

and

S.,

problem,\"

Franco J.,

Hopcroft,

the

\"On

problem,\"
Frederickson

Rabin's

on

note

\"A

nearest-neighbor

M.

Journal

optimization,\"
H.

on

Galil, T.

spanning

(1986),pp.

for selection in

algorithms

of

the

for

algorithms

Journal

sets,\"

34 (July

ACM,

in

searching

satisfiability

of Computer

and

uses

R.

undirected

in

network

1987), pp. 596-615.

E. Tarjan,

for

maximum

algorithms for finding

graphs,\" Combinatorica,

\"Efficient

directed

and

109-122.

H. N., and R. E. Tarjan, \"Algorithms for two bottleneck

9 (September
Journal of Algorithms,
1988), pp. 411-417.
\"On

their

and

and

Gabow

Galil Z.,

the

1986), pp. 103-106.

1988), pp. 337-348.

H. Spencer,and

trees

Walker,
and

Techniques

implementation of Edmonds's algorithm

Journal
of the ACM, 23 (1976),pp.221-234.

graphs,\"

N., Z.

H.
minimum

23 (August

D. W.

\"An efficient

N.,

matching

of

Letters,

E. Tarjan, \"Fibonacciheaps

and R.

L.,

performance

Processing

37 (December

Sciences,

Fredman

Gabow

probabilistic

\"Distributed

algorithm,\"

8 (1979), pp. 20-23.

Letters,

Information

G.,

System

Gabow

Mathematical

American

pp. 387-389.

M. A.

C,

Press,

University

S. W. Otto, J. K. Salmon,
and
Johnson, G. A. Lyzenga,
1: General
Solving Problems on Concurrent Processors,Volume
Cliffs, NJ, 1988.
Regular Problems,Prentice-Hall,
Englewood
G.

Journal

Canadian

network,\"

Princeton

Networks,

\"A tournament

Johnson,

Processing

Information
Fox

in

Flows

Fulkerson,

through

NJ, 1962.

Princeton,

Fortune

D. R.

and

R.,

flow

\"Maximal

Fulkerson,

8 (1956),pp.399-404.

of Mathematics,

improving

algorithm,\"

the

worst

case

Communications

time

running

of

the

optimization

of

ACM,

22

problems,\"

the

Boyer-Moore string
1979), pp.
(September

505-508.
Galil

Z.,

\"Efficient

algorithms

Surveys, 18 (March

for

finding

1986),pp.23-38.

maximum

matching

in graphs,\" Computing

Bibliography

452

Galler

B. A., and
of the ACM,

\"An improved

J. Fishcer,

M.

7 (1964),pp.301-303.

equivalence algorithm,\"

D. S. Johnson, Computersand Intractability,

W. H. Freeman, San Francisco,
CA,
NP-completeness,

D. P.

E. F.,

Gehringer

GibbonsA., Algorithmic

W. Rytter,

and

A.,

Gibbons

Parallel

Efficient

of

Theory

Parallel

The

Processing:

Cm

University Press, Cambridge,1985.

Cambridge

Theory,

Graph

Segall,

the

1979.

1987.

Press, Bedford, MA,

Digital

Experience,

Z.

and

Siewiorek,

to

A Guide

R., and

M.

Garey

Communications

University Press,

Cambridge

Algorithms,

Cambridge,1988.
A.

Goldberg

V.,

Journal of

the

and R.

E. Tarjan,

ACM,

35 (October

new

\"A

all
S., and J. Killian, \"Almost
ACM Symposium on Theory of Computing,

maximal-flow problem,\"

can be

primes

Berkeley

(May

18th Annual

certified,\"

quickly

pp. 316-329.

1986),

programming,\" Journal

\"Backtrack

L. Baumert,

to the

1988), pp. 921-940.

Goldwasser

GolombS.,and

approach

of the

ACM,

12 (1965),

pp. 516-524.

GolovinaL. I.,and
Golumbic

M.,

I. M.

Induction

Yaglom,

in

Geometry

(translated

from Russian),

D. C.

1963.

Boston,

Heath,

and Perfect

Theory

Graph

Algorithmic

Graphs, Academic Press,New

1980.

York,

M., and M.

Gondran

Minoux,

Graphs

and

Algorithms,

John Wiley &

Sons, New

York,

1984.

Gonnet

G. H., Handbookof

Graham

R. L.,

R.

Processign

L., D.

Reading, MA,
F., Pulse

Gray

tree machine,\"
D.

Boston,

E.

H., and

U.

Manber,

IEEE

planar

set,\"

D. E.

Patent

US

2632058

(March 1953).

\"A probabilistic pipeline algorithm

for ^-selection
on Computers, C-36 (March 1987),pp.

Transactions

Mathematics

Knuth,

for

the Analysis

of Algorithms,

on the

359-362.

Birkhauser,

1982.

Gries D., TheScienceof

GrotschelM.,L. Lovasz,
in

of a

Concrete Mathematics, Addison-Wesley,

O. Patashnik,

and

Knuth,

hull

1989.

Code Communication,

GreenbergA., and
Greene

Addison-Wesley, Reading,

for determining the convex

algorithm
Letters, 1 (1972),pp. 132-133.

efficient

\"An

Information
Graham

Data Structures,

and

Algorithms

1984.

MA,

combinatorial

Programming,
and

Springer-Verlag,

A. Schrijver,

optimization,\"

New York,

1981.

its consequences
\"The ellipsoid method
and
1 (1981) pp. 169-197.

Combinatories,

453

Bibliography

L. J.,

Guibas

Gusfield

D.,

L.

and

Pitt,

D. Wood,

Guting R. H., and

Combinatorial

M.,

10 (1935),pp.

Coding and

R. W.,

F., Graph Theory,

Hedetniemi
gossiping

and

Hillis W. D., The

W. D.,

Hillis

and

K., J.

sorting,\"

Connection
G.

L. Steele,

Practice

Hoffman C.

P.

broadcastingand

with

to

applications

pp. 13-28.

Cambridge, MA,

1985.

of the

Communications

solves the closest

\"Plane-sweep
26 (1988), pp. 255-261.

Schorn,

ACM,

pair

problem

Letters,

for

algorithm

of the

ACM, 18

common

maximal

computing

(June 1975),pp.341-343.

results

\"Recent

S.,

in

Time

Wraps,

String

1983.

R., \"Quicksort,\" TheComputer

M., Group

Computer

Science,

Probabilistic

J.-W.,

Hong,

trees
1962),

\"Data parallel algorithms,\"

linear-space

Wesley, Reading, MA,

C. A.

Prentice-Hall,

on the
of common-subsequence
complexity
and
Edits, and Macromolecules: The Theory
D.
Sankoff
and
J.
B.
AddisonKruskal,
eds.,
Comparison,
of Sequence

D.

problems,\"

Hofri M.,

Edition,

1170-1183.

Processing

\"A

S.,

9 (January

MIT Press,

Machine,

subsequences,\" Communications
Hirschberg

Society,

1969.

certain

of

properties

of the ACM,

Journal

Information

D.

Hirschberg

Hoare

Second

Addison-Wesley, Reading, MA,

Nievergelt, and

elegantly,\"

1986.

York,

Mathematical

the London

of

New

1986.

29 (December1986),pp.
Hinrichs

by divide-and-conquer,\"

and Sons,

Wiley

Theory,

Information

N., \"Some combinatorial

searching

cover,\"

pp. 771-775.

S. T., S. M. Hedetniemi, and A. L. Liestman, \"A survey of

to appear
in communication
networks,\" Networks,
(1989).

Hibbard T.

node

26-30.

Englewood Cliffs,NJ,
Harary

John

Edition,

Journal
representatives of subsets,\"

Hall P., \"On

Hamming

Second

Theory,

intersections

(July 1984),

C-33

Computers,

for

algorithms

pp. 291-294.

1986),

rectangles

\"Finding

IEEE Transactionson
Hall

Letters,22 (May

Boyer-Moore

pp. 672-682.

9 (1980),

approximation

\"Equivalent

Information Processing

of the

linearity

SI AM Journal

algorithm,\"

searching

string

proof of the
on Computing,

\"A new

M. Odlyzko,

A.

and

Theoretic

and Graph

Algorithms

136, Springer-Verlag, New

Analysis

Computation:

of Algorithms,
Computability,

(1962),

Journal,

York,

10-15.

Isomorphism, LectureNotes

and

New
Duality,

York,

1987.

Pitman, London,

1986.
Hopcroft

J. E.,

graphs,\"

and R. M. Karp, \"An n5'2

SI AM Journal on Computing,

algorithm

for maximum matchings

1973), pp. 225-231.

2 (December

in

1982.

Springer-Verlag,
Similarity

pp.

in

bipartite

454 Bibliography
J.

Hopcroft

SIAM

R. E. Tarjan,

\"Efficient

J.

J. D.

and

E.,

Ullman,

\"Divide

A. Zorat

and

E.,

Transactionson
C,

of

the

21

ACM,

of Computer

and

Programming

Integer

and

for

Network

Science

Computer

Algorithms,

conquer

(June 1983),

C-32

Computers,

Journal on

294-303.

E., and S. Sahni, Fundamentals

Press, Rockville, MD (1978).

T.

components,\"

SI AM

algorithms,\"

\"Set-merging

Horowitz

Hu

Journal

testing,\"

planarity

1973), pp.

2 (December

Computing,

Horowitz

into triconnected

graph

1973),pp. 135-158.

1974), pp. 549-568.

(October
Hopcroft

\"Dividing

2 (September

on Computing,

E., and

J.

Hopcroft

E. Tarjan,

and R.

E.,

Journal

IEEE

processing,\"

parallel

pp. 582-585.

Flows,

MA,

Reading,

Addison-Wesley,

1969.
and M. A. Langston, \"Practical

B. C,

Huang

in-place merging,\"

ACM, 31 (March 1988),pp.348-352.

D.

Huffman

\"A method

A.,

Proceedingsof
J.

Hunt

G.

and T.

W.,

for

\"A

Szymanski,

subsequences,\"
New

Incerpi J.,

A.,

R.

\"Practical

Sedgewick,

T.

and

Parallel

and

Processing,

variations of shellsort,\" Information

circuit

minimum

1978), pp.
Kailath,

IEEE

multiplication,\"

\"Finding

7 (November

Jagadish H. V.,

for computing longest common

20 (May 1977),pp.350-353.

algorithm

McGraw-

Processing

pp. 37-43.

M. Rodeh,

Computing,

fast

Briggs, Computer Architecture

26 (1987)

and

codes,\"

redundancy

1984.

York,

and

Letters,
Itai

F. A.

and

K.,

Hill,

minimum

1952), pp. 1098-1101.

of the ACM,

Communications

Hwang

of

construction

the

IRE, 40 (September

the

of the

Communications

in

SI AM

graph,\"

Journal on

413-423.

\"A

efficient

new

of

family

on Computers,

Transactions

C-38

arrays

for

matrix

1989),

(January

pp.

149-155.

Johnson D.

S., A.

B., \"Efficient

D.
the

24 (January

ACM,

Jones D.

\"An

W.,

A.,

Akad.

and

Nauk

SSSR,

of

\"Multiplication

the

in

paths

of

comparison

empirical

Yu. Ofman,

in Sow

translated

algorithms for shortest

1977), pp. 1-13.

Communications

implementations,\"
Karatsuba

case

SIAM

on Computing, 3 (1974),pp.299-325.

Journal
Johnson

and R. L. Graham, \"Worst

Ullman, M. R. Garey,
for simple one-dimensional packing
algorithms,\"

J. D.

Demers,

bounds

performance

ACM,
of

networks,\"

sparse

priority-queue

and

Journal of
event-set

29 (April

1986), pp. 300-311.

multidigit

numbers

Phys. Dokl., 7 (1963),pp. 595-596.Originally

145 (1962), pp. 293-294.

on

appeared

automata,\"

in Dokl.

Bibliography 455
\342\200\224
E. Upfal, \"Parallel hashing
An efficient
18th
Annual
ACM Symposium on Theory of

R., and

A.

Karlin

memory,\"

\"A

N.,

Combinatories,
Karmarkar

polynomial time
4 (1984),pp.373-395.
new

R. M. Karp,

and

N.,

Computer Science,(November
*

R. M.,

R.

Karp

M. Saks,

M.,

bound
Toronto

M.,

R.

L.

K.

N.,

Klee V., and

B.

L. Minty,

linear

in

algorithm

\"An

14 (May

The

D. E.,

simplex

York (1972),

New

R. Manber,

Art of Computer

Art

of

\"Estimating

29

Computation,

Knuth

D. E.,

\"Big

(1975),

omicron

1976),pp. 18-24.
Knuth

Computer
MA,

Reading,

E.,

of
New

of

the

algorithms,\" IBM

programming,\"

Soviet Math. Doki,

pp.

algorithm

for

sink-finding,\"

109-111.

algorithm?\"

in Inequalities

111,O.

pp. 159-175.

\"Signsolvability revisited,\" Linear Algebra

and

59 (1984), pp. 131-157.

Addison-Wesley,

Knuth D.

Press,

1987),pp.249-260.

optimal

1982),

Volume

Programming,

Second edition, Addison-Wesley, Reading,MA,

Knuth

Complexity

Plenum

Communications

pattern-matching

31 (March

\"How good is the

and

its Applications,
E., The

randomness,\"

randomized

Development,

Press,

Klee V., R. E. Ladner,

D.

and

Smith-Thomas,

Shisha,ed.,Academic

Knuth

in

19-28.

Processing Letters,

G.

of

191-194.

and

Information

Foundations

search problem related to branch-andon Foundations

of Computer Science,

\"On a
Symposium

\"Efficient

and

\"A polynomial

G.,

problems,\"
W. Thatcher, eds.,

J.

and

complexity,

Research

20 (1979),pp.
King

pp.

one-

the

1986), pp. 98-109.

M. O. Rabin,

M., and

Journal of
Khachian

Annual

\"Combinatorics,

29 (February

ACM,

Karp

1986),

E. Miller

Wigderson,

27th

(October

Karp R.

A.

and

procedures,\"

on

Symposium

combinatorial

among

Computer Computations, R.
York (1972), pp. 85-103.

for

scheme

approximation
Annual

programming,\"

pp. 312-320.

1982),

deducibilities

linear

for

algorithm

efficient

\"An

problem,\" 23th

bin packing

dimensional

Karp

(May

Berkeley

160-168.

1986), pp.
Karmarkar

of shared

implementation
Computing,

the

Programming,
1973b.

efficiency

II Fundamental

Algorithms,

1973a.

Volume 31 Sorting

of backtrack

and Searching,

programs,\" Mathematics of

pp. 121-136.
and

big omega

and

big

theta,\"

Volume
D. E., The Art of Computer Programming,
Second edition, Addison-Wesley, Reading, MA, 1981.

S1GACT News,

21 Seminumerical

(April-June
Algorithms,

Bibliography

456

Knuth D. E.,
163-180.

Huffman

\"Dynamic

D. E., J. H. Morris,
Journal on Computing,

Kronrod, M. A.,
J.

Kruskal

\"An

Nauk.

(1985),

pp.

SIAM

in strings,\"

of

without a field
pp. 1256-1258.
graph

the

and

Mathematical

American

the

Algorithms,

matching

pattern

of a

subtree

spanning

Proceedings of

problem,\"

186 (1969),

SSSR,

of

323-350.

algorithm

ordering

optimal

\"On the shortest

B.,

1977), pp.

6 (June

Dok. Akad.

Russian),

Pratt, \"Fast

V. R.

and

Knuth

Journal

coding,\"

(in

operation,\"

salesman

traveling

71 (1956),

Society,

pp.

48-50.
C.

Kruskal

M.
L. Rudolph, and
on Computers, C-34

P.,

Transactions
C.

Kruskal

L. Rudolph,

P.,

1986
problems,\"
pp. 869-876.

\"The power of parallel prefix,\"

Snir,
(November 1985),pp.965-968.

M.

and

International

\"Efficient

Snir,

parallel algorithms
Processing,

for

on Parallel

Conference

H.

T.,

\"Let's

H.

Kung

T.,

and

U.

Ladner

time

of polynomial

Fischer,

\"Parallel

prefix

L.,

Combinatorial
New

York,

A., S.

Even, and

Theory of Graphs:

intersection

and

of the

Journal

reducibility,\"

ACM,

of the

Journal

ACM, 27

Optimization:
A.

H. G.

Wiley

and

Networks

International

Rinnooy Kan,

& Sons, New

I. Cederbaum,
An

Mathematical

Cambridge

Discovery,

Matroids,

Holt,

Rinehart &

1976.

L., J. K. Lenstra,
SalesmanProblem,
John

Lawler E.

Lempel

for set
267-282.

831-838.

and Refutations: The Logic of

University Press, Cambridge,1976.
E.

in

eds., Addison-Wesley,

Conway,

computation,\"

I., Proofs

Winston,

L.

and

Mead

processor arrays,\"

155-171.

pp.

1975),

E., and M. J.
(October1980),pp.

Lawler

for VLSI

\"Algorithms

Systems, C.

structure

the

Caltech

the

pp. 37-46.

1982),

\"A probabilistic distributed

algorithm
Theoretical Computer Science, 49 (1987),pp.

Ladner R.

Lakatos

15 (January

Manber,

R. E., \"On

22 (January

of

Proceedings

systems,\"

sparse

pp. 271-292.

(1980),

its analysis,\"

of

pp. 65-90.

Leiserson,

to VLSI

Reading, MA,
Kurtz T.,

E.

C.

and

Introduction

VLSI

for

algorithms

systlic architecture,\" Computer,

T., \"Why

Kung H.

design

on VLSI, (1979),

Conference

graph
1986),

(August

and M. Snir, \"Techniques for parallel manipulation

Kruskal C. P., L. Rudolph,
matrices,\" IBM Research report RC-13364,(December1987).
Kung

IEEE

York,

and

D.

B. Shmoys,

The Traveling

1985.

\"An algorithm for planarity

Rome (July
Symposium,

testing

of graphs,\"

1966), pp. 215-232.

457

Bibliography

L. A.,

Levin

\"Universal

115-116.

English

Problems

in

Peredaci Informacii, 9 (1973),pp.

Transmission, 9 (1973),
Information

Problemy

problems,\"

sorting

translation

of

pp.265-266.
Lin

B. W.

and

S.,

Operations Research, 21 (1973),pp.

problem,\"
G. J.,

Lipovski
Wiley

Parallel

Malek,

New

the

salesman

traveling

498-516.

Theory and

Computing:

Comparisons,John

1987.

York,

Elements of Algebra

J. D.,

Lipson

M.

and

& Sons,

for

heuristic

effective

\"An

Kernighan,

and

Bejamin Cummings, Menlo

Computing,

Algebraic

Park,CA, 1981.
R.

Lipton

Litwin

of the

\"Linear

W.,

Sixth

the

D. Lopresti,

and

J.,

Proceedings

\"A

hashing:

Conference

A new
on Very

on

L.,

colorings of

and

\"Coverings

Southeastern Conferenceon
Mathematica,

LovaszL.,Combinatorial
Lovasz

M. D.

and

L.,

Problems

and

S., \"Sometechniques
(December1980),pp. 419-436.

for

time,\"
Lynch

North

G.

Manacher

Manber

U.,

A. Plotkin,

Information

K.,

\"On

Letters,

dynamic

maintaining
15

U.,

and

(1988),

tree

spanning

as

Reinhold,

path-finding

1988), pp. 291-293.

is not

L. McVoy,
pp. 263-272.

\"Efficient

in a

information

1986), pp.

(November

to design
Manber U., \"Usinginduction
1988), pp. 1300-1313.
(November

18

Nostrand

optimal,\" Journal

of the

1979), pp. 441-456.

Journal on Computing,

Manber

polynomial

pp. 42-65.
Van

26 (January
algorithm

sorting

tested in

can be

Applications,

\"Minimum-cost

Processing

Computing Surveys, 12

25 (1982),

Sciences,

System

\"The Ford-Johnson

26 (July

ACM,

S.

and

M.,

problem,\"

valence

bounded

T. J., Data Compression Techniques and

New York, 1985.

Maggs B.

Amsterdam, 1986.

Holland,

recurrences,\"

solving

of

1979.

1882.

Paris,

E. M., \"Isomorphism of graphs

Journal of Computer and

Luks

North Holland, Amsterdam,

Exercises,
Theory,

Mathematiques,

Lueker G.

of the Fourth
and Computing, Utilitas

Theory,

Graph

pp. 3-12.

Plummer, Matching

LucasE.,Recreations

363-376.

Proceedings

hypergraphs,\"

Combinatorics,

(1973),

Winnipeg

comparison,\"

string

(\\9S5\\ pp.

Databases,

Large

212-223.
Lovasz

rapid

VLSI,

and table addressing,\" Proceedingsof

Canada (1980) pp.
Montreal,

for file

tool

for

array

systolic

1985 ChapelHill Conference

algorithms,\"

storage

concurrent

environment,\" SIAM

1130-1142.
Communications

of nonadaptive

of

the

ACM,

routing tables,\" Networks,

31

Bibliography

458

Manber U., and

E.

\"Suffix

Myers,

arrays:

Z., Lectures on

Manna

Manna Z.,

R.

and

the

Masek W. J., and

Time Warps,

Reading,

MA

Sankoff

D.

N.,

tree construction

suffix

Journal

K., Data Structures and Algorithms

Computational Geometry,Springer-Verlag,

Mehlhorn

K.,

Menger

A.

Meyer

\"Zur

requires

Automata

Theory,

A.,

\"A

and

J.,

Moffat A.,

D. Gries,

and

T.

and

pp.

S. S.

all

Myers

\"Problems
ed.,

expression

regular
on

Switching

with

and

Marcel Dekker,New

E. W.,

related

\"An

O(ND)

(1986), pp. 251-266.

subsequence

problem,\"

elements,\"

Science

of Computer

matrix

Computing

diagrams,\"

Technical Report

York,

algorithm

(June

Surveys,

in Computational

multiplication,\"

Press, New
difference

time
expected
path algorithm with
16 (December 1987),pp. 1023-1031.

shortest

parallel algorithms,\"
Science, Cornell University

of Computer

to

stuttering

of

\"Discussion

Iyengar,

Algorithmics

pair

on Computing,

Journal

B. M. E., \"Decisiontrees and

1982),pp.593-623.
I.,

for

71-75.

repeated

Moret

Rustin,

and

143-152.

\"An

Takaoka,

SIAM

longest

26 (1987),pp.

\"Finding

2 (1982),

TR-86-759,Department

Munro

of serial

Searching

Symposium

and Graphs,

Networks

for the

technique

halving

0(n2\\ogn)\"
A.,

design

852-865.

10 (1927),pp.95-115.

Fund. Math.,

for

Processing Letters,

Programming,

Moitra

the

in

of

125-129.

pp.

Algorithms

Optimization

Information
Misra

(1972),

Journal

algorithm,\"

1978.

York,

Mirzaian

in

1984.

L. Stockmeyer, \"The equivalenceproblem

exponential
space,\" 13th Annual

R., and

of

quickly,\"

Multi-Dimensional

3:
Berlin,

Kurventheorie,\"

allgeminen

squaring

Minieka E.,

B.

J.

and

distances

string-edit

parallel
computation algorithms
the
30 (October1983),pp.
ACM,
of

\"Applying

algorithms,\"

Science

paradigm,\"

pp. 337-349.

(1983),

McCreight, E. M., \"A space-economical

the ACM 23 (1976),pp.262-272.
Megiddo

1980.

PA,

The Theory and Practice of

Kruskal,
eds., Addison-Wesley,

Macromolecules:

and

Edits,

String

CBMS-NSF Regional

search

binary

\"How to compute

S. Paterson,

M.

1990).

pp. 37-83.

9 (1987)

Comparison,

Sequence

origin of

'The

Waldinger,

Computer Programming,

searches,\"

string

(January

Programming,
of Computer
Mathematics, SIAM, Philadelphia,

Logic

in Applied

series

Conference

for on-line

method

new

Discrete Algorithms

first Annual ACM-SIAM Symposium on

1986).

14 (December
Complexity, R.

1971.

and

its

variations,\"

Algorithmica,

Bibliography 459
E.

Myers

\"A four-Russians

W.,

Technical

October

pattern

expression

regular

matching,\"

of

Arizona,

1988.

Nishizeki T., and N. Chiba,

Mathematics, 32, North
Ore O.,

for

algorithm

Department of Computer Science,University

#88-34,

Report

on

\"Note

Graphs, Theory and

Planar

Hamiltonian

of Discrete

Annals

Algorithms,

1988.

Netherlands,

Holland,

circuit,\"

American Mathematical

Random

House,

67

Monthly,

(1960),

p.

55.

Ore O., Graphs and Their

Uses,

Pan V., \"Strassen'salgorithm

of Computer Science}Ann
Pan

V.,

How

is not
Arbor,

129, Springer-Verlag,Berlin,

I., Parallel

M.

C,

Sons,

York,

A.

\342\200\224

H.

and

Communications of the
G.,

Polya

Induction

New York, 1987.

John Wiley &

Framework,

Transformation

Search

Intelligent

for Computer

Strategies

Problem Solving,

MA, 1984.

Reading,

Itai,

Recursion

& Sons,

Wiley

and

Algorithms

1988.

Addison-Wesley,

Perl Y.,

Design:

Algorithm

New

Volume

Science,

Computer

1984.

ComplexityTheory, John

J., Heuristics

Pearl

in

K. Steiglitz, CombinatorialOptimization:
Prentice-Hall, Englewood Cliffs,NJ, 1982.

Complexity,

Paull

Lecture Notes

on Foundations

Symposium

pp. 166-176.

C. H., and

Papadimitriou

Parberry

optimal,\" 19th Annual

MI (October 1978),

Matrices Faster,

to Multiply

New York, 1963.

Avni,
ACM,

and Analogy

in

\342\200\224
A

search

\"Interpolation

21 (July

search,\"

loglog/V

1978), pp. 550-553.

Princeton

Mathematics,

University

Press, Princeton,

NJ, 1954.

Polya G., How

to

It, Second

Solve

Edition,

Princeton

University

Press,

Princeton, NJ,

1957.

Polya G., Mathematical

Discovery,

Combined

Edition,

John Wiley &

Sons, New

York,

1981.

G., and G. Szego, Aufgaben

Springer, 1927, p. 7.

Polya

G.,

Polya

and

G. Szego,

und Lehrsatze

Problems and Theorems

aus der Analysis,

in Analysis

I, Berlin,

Volume

I, Springer-Verlag,

Berlin,

1972.
and M. I. Shamos,Computational
New
York, 1985.
Verlag,
F. P.,

Preparata

Prim

R.

C,

\"Shortest

TechnicalJournal,

connection
36

(1957),

networks

p. 1389.

Geometry:

and

some

An Introduction,

generalizations,\"

SpringerBell System

Bibliography

460

York,

New

The Analysis

and C. A. Brown,
1985a.

P. W.,

Purdom

Purdom P. W., and C. A. Brown, 'The pure

14 (November
SIAM Journal on Computing,
M.

Quinn

1985b), pp.
Parallel

for

Algorithms

average time,\"

polynomial

943-953.
McGraw-Hill, New

Computers,

1987.

York,

M.

Quinn

Efficient

J., Designing

and

rule

literal

& Winston,

Holt, Rinehart

of Algorithms,

J.,

N.

and

\"Parallel

Deo,

graph

16

Surveys,

Computing

algorithms,\"

(September 1984), pp. 319-348.

M.

Rabin

and

\"Probabilistic

O.,
New

algorithms,\"
Traub, ed.,

J. F.

Directions,

Raghavan P., \"Probabilistic construction

packing integer programs,\" 27th

Science,Toronto
A. G.,

Ranade

M.

R.

and

Processing,MIT
Reif J. H.,and

L. G.

ACM, 34

of the

Rivest

J. Hansen,

W.

28th

memory,\"

Multicomputer Networks:
MA, 1987.

Message-BasedParallel
size networks,\"

Journal

Shamir, and L. M. Adleman, \"A method for

cryptosystems,\" Communications
of the

and public-key

1983.

18 (Summer

SIGACT News,

bibliography,\"

on

Symposium

pp. 185-194.

Data Structures, Little, Brown, Boston, MA,

\342\200\224
A

sorting

Annual

1987),

(October

\"A logarithmic time sort for linear

1987),
pp. 60-76.

Valiant,

D., \"Parallel
R. L., A.

Cambridge,

(January

Reingold E. M., and

Richards

Fujimoto,

Press,

pp. 21-39.

1976,

York,

algorithms:
Approximating
on Foundations of Computer

Symposium

of Computer Science,LosAngeles

Foundations

Reed D. A.,

deterministic

Recent Results

Complexity,

10-18.

shared

emulate

to

\"How

of

pp.

and

Algorithms

Annual

1986),

(October

in

AcademicPress,New

obtaining

1986).

signatures

digital

ACM, 21 (February

1978),

pp. 120-126.
F. S.,

Roberts

Rodeh

M.,

RonseC,
Philips
Rosenberg
News,
Runge

C,

on digital

bibliography

Research

distributively,\"

A. L., \"On

1984.

of Computer and

and computational convexity,\"

Laboratory, Brussels, (February

(1973), pp.

Journal

NJ,

System

pp. 162-166.

24 (1982),
\"A

median

the

\"Finding

Sciences,

Prentice-Hall, EnglewoodCliffs,

Combinatorics,

Applied

time

the

required

Manuscript

Ml85,

1987).

to recognize

properties of

graphs,\"

SIGACT

15-16.

and H. Konig,

Die

der

Grundlehrn

mathematischen

Wissenschften,

Springer,

Berlin, 1924.
Ryser

H.

J.,

\"Combinatorial

Journal of Mathematics,

properties
9 (1957),

of matrices of

pp. 371-377.

zeros

and

ones,\"

Canadian

461

Bibliography

\"Fast

W.,

Rytter

Sankoff

Theory and Practice

1983.
Savitch

W.

J.,

Schaback R.,

the

\"On

17

Theory of

A.,

N.

and

the

Linear

Pippenger,

and

Sedgewick

R., Quicksort, Garland,

New

Sedgewick

R.,

Edition,

Second

Algorithms,

SedgewickR., and J. S.
(1986),pp.31-48.

Journal

of

184-199.

John

& Sons,

Wiley

Vitter,

1978.

York,

\"Shortest

and D. Hoey,
on
Foundations
Symposium
M.

I.,

Reading, MA, 1988.

Addison-Wesley,
in

paths

Euclidean

M.
I., and D. Hoey, \"Closest-pointproblems,\"
Foundations of Computer
Science,
(October
Berkeley

Shamos

the median,\"

\"Finding

1986.

Chichester,

Shamos

SI AM

algorithm,\"

Boyer-Moore

Programming,

Integer

tape

pp. 177-192.

648-658.

13 (October1976),pp.

System Science,

and

of

1988), pp.

(August

deterministic

and

4 (1970),

Sciences,

System

sublinearity

expected

SchonhageA., M. S. Paterson,
Schrijver

nondeterministic

between

Reading, MA,

Addison-Wesley,

Comparison,

The

Macromolecules:

and

Edits,

String

Wraps,

Journal of Computer and

Journal on Computing,

Computer

Time

\"Relationships

complexities,\"

languages/'

pp. 12-22.

Sequence

of

context-free

and

automaton

pushdown

67 (1985)

Control,

J. B. Kruskal,

and

D.,

of

recognition
and

Information

\"Geometric intersection
Science,
of Computer

Annual

16th

Algorithmica,

graphs,\"

on

Symposium

1975), pp. 151-162.

Annual

17th

problems,\"

Houston

1976), pp.

(October

208-215.
Shell D. L., \"A
1959), pp.
Shiloach

Y,

sorting

high-speed

U. Vishkin,

and

N. Francez,

L.,

\"Finding

algorithm,\"

Computing,

Ottawa

SiegelH. J.,Interconnection
Studies,
D. D.,

Sleator

G.

and

and

32 (July

ACM,
Smit

and M.

distributed

Case

maximum,

Algorithms,

ACM

1983), pp.

Networks

for

merging,

3 (March

the

2 (July

ACM,

and

in

sorting

a parallel

1981), pp. 88-102.

From a sequential to a
on
Principles of Distributed
Symposium

Rodeh, \"Distributed

Second

(August

the

k selection:

143-153.

Large

Scale Parallel

Lexington Books, Lexington, MA,

R. E. Tarjan,
\"Self-adjusting
1985), pp. 652-686.

\"A comparison of three string

12 (1982), pp. 57-66.
Experience,

V.,

of

30-32.

computation model,\" Journal of

Shrira

Communications

procedure\"

matching

Processing:Theory

and

1985.

binary

search

algorithms,\"

trees,\"

Journal

Software

of

\342\200\224
Practice

the

Bibliography

52

test for

R., and V.

Strassen, \"A fast Monte-Carlo

6
(March 1977),pp.84-85;erratum
Computing,

olovay

(ominskii I.

'The

E. H.,

Jpafford
TR-823,

average

1988).

(November

in a

paths

graph of

positive arcs in

pp. 28-32.

2 (1973),

MA., 1980.

Reading,

Addison-Wesley,

Techniques,

Technical Report CSD-

University

0(AZ2log2Az),\" SIAM Journal on Computing,

Constructive Combinatorics,Springer-Verlag,

D. White,

and

D.,

Purdue

for finding all shortest

algorithm

Standish T. A., Data Structure

Stanton

An analysis,\"

program:

Science,

of Computer

new

time

worm

Internet

Department

M., \"A

D. C.

Induction, (translated from Russian),

of Mathematical

Method

1963.

Boston,

Heath,

Spira P.

S., The

p. 118.

1978),

(February

on

Journal

SIAM

primality,\"

New

York,

1986.

R,

Stout Q.
of

Strassen

Computing, 4

Distributed

and

is

elimination

\"Gaussian

V.,

algorithms

divide-and-conquer

\"Supporting

Parallel

not

for image

(1987),pp.95-115.
Numerische

optimal,\"

processing,\" Journal
13 (1969),

Mathematik,

pp. 354-356.
Takaoka

\"An

T.,

22 (May

on-line

Tarjan R. E., \"Depth

Computing, 1 (June
Tarjan R.

E., Data

Tarjan R.

of

E.,

J.

and

E., and

Journal

U.

van

graph

linear

not

SIAM

algorithms,\"

set union

on

Journal

algorithm,\" Journal

of the

complexity,\"

des

\"Almost

PA, 1983.

Philadelphia,

SIAM Journal

\"Worst case analysis

Leeuwen,

Vishkin,

SIAM,

Algorithms,

on Applied and

pp. 306-318.

1984),

labyrinths,\"
Geometry,

of set

union

algorithms,\"

pp. 245-281.

\"An efficient

on Computing, 14 (November

\"Leprobleme

J. S.,

Network

computational

parallel

1985),

Nouvelles
North

all /:-colorable graphs

(1988),pp.63-82.
Tutte

but

6 (1985),

Toussaint G., Computational

Turner

a good

of the ACM, 31 (April

Tarjan R.
Tarry G.,

146-160.

Structures and

E., \"Amortized

Journal

linear

and

pp.

1972),

DiscreteMathematics,
Tarjan R.

Letters,

1975), pp. 215-225.

22 (April

R.

Processing

Information

algorithm,\"

matching

search

first

E., \"Efficiency

ACM,
Tarjan

pattern

pp. 329-330.

1986),

algorithm,\"

biconnectivity

SIAM

pp. 862-874.

Ann. de Math,

Holland,

are easy to

W. T., Graph Theory, Encyclopedia of Mathematics,

Reading, MA, 1984.

(1895),

p. 187.

Amsterdam, 1984.
color,\"

Volume

Journal

of Algorithms,

21, Addison-Wesley,

Bibliography 463
Uhr

Parallel

L.,

Ukkonen

E.,

J.

for

\"Algorithms

1987.

York,

string matching,\"

approximate

and

Information

Control,

pp. 100-118.

64 (1985),
Ullman

Vision, Academic Press,New

Computer

Aspects of

D., Computational

VLSI,

Press, Rockville,

Science

Computer

MD,

1984.
for parallel

schemes

\"Efficient

E.,

Upfal

Principles of Distributed
and A.

E.,

Upfal
the

Vacca

the first discoverer

\"Maurolycus,

P. M.,

\"Geometry

helps

of Computing,

Chicago

(May

L.

Valiant

\"Parallelism

G.,

L.

a distributed

of the

of mathematical

principle

and

20th

422-425.
SI AM

problems,\"

comparisons

for fast parallel

1982),

Journal on

Computing,

on

Journal

SI AM

communication,\"

pp. 350-361.

J. Brebner,

G.

induction,\"

ACM Symposium on Theory

Annual

matching,\"

1988), pp.

in

\"Universal schemes for

ACM Symposium on Theory of Computing,

L. G.,

Annual

Journal of

system,\"

348-355.

scheme

\"A

G.,

Computing, 11 (May
Valiant

in

1975), pp.

(September
Valiant

in

Mathematical Society, 16 (1909),p. 70.

American

of the

Bulletin

Vaidya

(August

to share memory
Wigderson, \"How
1987), pp. 116-127.

on

Symposium

1982), pp. 55-59.

34 (January

ACM,
G.,

Computing,

ACM

First

communication,\"

Ottawa

parallelcommunication,\"
Milwaukee

13th

1981),

(May

pp.

263-277.
van

interpolation search,\" Communications

Nat M., \"On

der

1979),p. 681.
van

von

Leeuwen

J., \"Graph

Science,

Neumann

J., Collected

U.,

Vitter J.

J.

S.,

Neumann

Wagner
the

R. A.,
ACM,

W. C.
New

Press,
J.,
and

analysis

York,

list

ranking,\"

Collected
M.

of dynamic

of Parallel

and

Huffman

&

Analysis

OR

coding,\" 26th
(October

Annual

1985), pp.

of

Coalesced

York,

1963.

Symposium

293-302.

Hashing,

Oxford

1987.

Works, 5,

Macmillan, New

\"The string-to-string
1974), pp. 168-173.

J. Fischer,

21 (January

Journal

91-99.

pp. 319-333.

Chen, Design
York,

1963, pp.

New

of Computer Science,Portland,

and

University

von

4 (1987),

S., \"Designand

Macmillan,

speedups for

parallel

Computing,

on Foundations
Vitter

Works, 5,

\"Randomized

Distributed

(December

of
algorithms,\" Technical Report RUU-CS-86-17,Department
The Netherlands,
Utrecht,
(October 1986).
University of Utrecht,

Computer

Vishkin

ACM, 22

of the

correction

problem,\"

Journal

of

464

Bibliography

modification of Warshall's algorithm

of the ACM, 18 (April

H. S., \"A

Warren

for

Communications

relations,\"

Warshall S.,

matrices,\" Journal

on Boolean

theorem

\"A

of

closure

transitive

binary

pp. 218-220.

1975),

of the

9 (January

ACM,

1962),

pp. 11-12.
and applications
of hash functions,\"
classes
N., and J. L. Carter, \"New
1979),
Symposium on Foundations of Computer Science, (October

M.

Wegman

Annual

20th
pp.

175-182.

Weiner, P., \"Linear

and

Automata

Whitney H., \"Congruent

Mathematics, 54
A.,

Wigderson

Wilf H.

S.,

Iowa City, Iowa (October

the performance guarantee for

1983),
pp. 729-735.

An

\"Backtrack:

W.

J.

0(1)

expected

Processing

Letters,

\"A new

S.,

C-17 (1968),pp.
S.,

Winograd

of

coloring,\"

\"On

time

algorithm for

inner

IEEE

product,\"

the

for

algorithm

coloring

graph

pp. 119-121.

18 (1984),

the ACM,

of

Transactions

7 (June

on Computers,

693-694.

of

the number

to

necessary

multiplications

Pure

on

Communications

functions,\"

approximate graph

232: Heapsort,\"Communications

J., \"Algorithm

1964),p. 701.
Winograd

Journal

American

graphs,\"

ACM, 30 (October

problem,\" Information
Williams

of

the connectivity

and

graphs

Switching

(1932),pp. 150-68.

\"Improving

of the

Journal

14th IEEE Symposium on

1973),
pp. 1-11.

algorithm,\"

matching

pattern

Theory,

and Applied Mathematics,

certain

compute

23 (1970a),pp.

165-170.

S., \"The algebraic complexity of functions,\"

Proceedings
3 (1970b),
Congress of Mathematicians,
pp. 283-288.

Winograd

Winograd S.,
Sequential
New

York

\"Someremarks
(1973),

pp.

Winograd S., Arithmetic

Wirth

N.,

Yao

A.,

Algorithms
\"An

0(\\E

Information
Yao

A.,
Annual

Symposium

Ziv

Algorithms,

of

Press,

log

\\V

SIAM, Philadelphia, PA, 1980.

of Computations,

|)

Prentice-Hall, EnglewoodCliffs,

algorithm

for

finding

minimum

1986.

NJ,

trees,\"

spanning

Letters, 4 (1975),pp.21-23.
\342\200\224
toward

computations
on Foundations

a unified

of Computer

measure of complexity,\" 18th

Science,Providence,

RI

(October

222-227.

and A. Lempel, \"Compression of indiviual

IEEE Transactionson Information
Theory,

J.,

of polynomials,\" in Complexity
J. F. Traub,
ed., Academic

multiplication

& Data Structures,

| log

International

181-196.

Complexity

Processing

\"Probabilistic

1977), pp.

fast

on

and Parallel Numerical

of the

via variable-rate

coding,\"
(September 1978), pp. 530-536.

sequences

IT-24

INDEX

347-349, 351-354,356-363,369,

41

O 39

414-415,426-434

370-374,

00 41-42,174,316,410

greedy 210-211,235,237,363

0 41

for

ft

networks

interconnection

378,

389-403

41

co 41

for

notation

numeric 293-320,379-380,398-^01,

Aanderaa, S. O.
data

Abstract

Ackerman's

404-407

113

types

402-403

collection of sets 372

see Directed acyclic graph
graph,

Acyclic
Acyclic
Addition,

84
Adjacency matrix representation
Adleman,L. M. 295,316
A. V. 85, 174, 175,305,316,336
Aho,
Ajtai,

Angluin,

R. 411
293-320,436-438

37-60
of

definition

distributed

Arborescence

357, 363-367

approximation

208, 376
158,

169, 172,

176, 183,264,278-281,286, 287,

289, 299-304, 311-314, 316-319,
394-396,

409-410,

431

of 341
efficient, definition
geometric 102-104, 265-291,331-333,
434-436

graph

95-96,98-101,

357, 363-367

186

66,

24-26,

35-36

divide-and-conquer 103-104,113-114,

379-380,383,

289

Arithmetic geometric mean theorem

131,137,141,144,

175

A.

Approximate convex hull

Approximation
algorithms

293-320, 436-438
of

369

D.

Apostolico,

analysis

322

Anti-Gray codes 33

Algorithm

algebraic

119-183

Amortized complexity 55, 83,86

of algorithms 37-60
Analysis
Ancestor
(in a tree) 193

M. 410

Algebraic algorithms

42

236

path

Alternating

Akl, S. G. 409
Aleliunas,

sets

systolic 404-409,411,416
All shortest
paths problem 212-214,
G. S. 409
Almasi,

84

list representation

Adjacency

sequencesand

G. M. 75

Aderson-VePskii,

of an

time

running

379-380,411

parallel

158-164, 175,287,320,

probabilistic

86

function

214,216,304,375-416

parallel

62

185-264,325-326,

Arithmetic sum

11-12,31,53

V. L. 317

Arlazarov,

of 63
Array, definition
Articulation point 218,223,260,430
ASCII

Assembly
Assignment

146

line 404
problem

330-331

Asymptotic complexity, definition

Atkinson, M. D. 317

of

38

466 Index
Booleanmatrix

410

M.J.

Attalah,

Augmenting

path

L. 248

Auslander,

A.

Borodin,

130
AVLtree 75-77, 104,114-115,

174

Avni,H.

multiplication

316,410

338
Bach, E. 297

Breadth-first-search

369, 373

358-363,

Baer, J. -L. 86

75, 104-106,248

Balance factor

R. 369

Bar-Yehuda,

Batcher, K. E. 394,410
L. 369

Beame,P. W.

410

begin 5
Blelloch, G.

409

Bentley,
Berge,

31, 114

217

Binary

357, 363-365,
conversion
numbers,

Binary

search

370, 371, 374

to 26-28, 36

120-125,285,342,431

molecular

149,155

matching

117,187,372

235-238,

Bisection 125
R.

Blum, M.
Bohr,

N.

55

Bolzano

175,247,248

method

Bondy,J. A.

247

125

409

K. M.

Chandy,

247,432

31

Chen,CW.

86

D. 247

Cheriton,
Chiba,

50

equation

Chavey, D.

Christofides,

174

79

Chand,D.R.286

N.

Choueka,Y.

369

Bollobds, B.

337, 357

98-101,113-114

Chaining, separate

Chartrand,G.

71-77, 86,285

Bipartite graph 111-112,

248

problem

Characteristic

34,263-264

Biology,

24

A. L.

Celebrity

34, 115

search tree

L. 86

Cederbaum, I.

edge-biconnected259
packing

175

L.

Carter, J.
Cauchy,

Binary

411

Capobianco,M. 247

259

complete

113

Capacity 238
search

component 217-223,249,259-260,430
definition of

142,164

127-129,

network

Butterfly

Carrol,

Binary tree

55,369

Bussey,W.H.30

Biconnected

Bitner,J.

sort

Bucket

409
BFS,seeBreadth-first

Bipartite

C. A.

Burge,W.H.

Bertsekas, D. P.

Bin

network 256

Brualdi,R.A. 55,336

174,440

extension

317

396-398,406,412

Brown,

J. L. 55, 106,287
C. 247

graph,

Briggs,

Bre1az,D. 369

E. 31

R. E.

in a

F. A.

Broadcast

Beckenbach,

Bellman,

graph 34, 220

409

Bridge,

Broadcast

Bavel,Z. 55

Bent,S. W.

D. 369

Brent's lemma 381,383,384,441

Brigham, E.O.

Bates,J. L. 106
Baumert,

246, 249,426,427

Brebner,G.J.410
Brelaz,

87

155,174

358-363, 370
95, 189, 198-199,238,

Branch-and-bound

b-Matching

Bag

317,

Bottleneck problems 254,257, 261,338

155, 174
Boyer,R.S.

Boyer-Moore algorithm

Backtracking

304-309,

319-320,325-326

239

247

175
N.

248,370

Circle 34,270,271,290,332
186
in a graph
Circuit,
Eulerian
187; see also Eulerian
graph
Hamiltonian
243-244; see also
Hamiltonian

cycle

467

Index

Circuit, model of parallel

378, 410
computing
356, 361, 370,
348, 351-352,
Clique problem

Conversion

binary to decimal

371,372
Closed

Convex hull

in the plane 266

278-281,

path,

Closestpair

287, 289

problem

Coalesced hashing
J. 304,317

Cook,S. A.
78-80

(in hashing)

175, 252,

14, 161-164,

Cooley,J. M. 317

262,432

Cover

edge 262

146, 256, 329

graph 348

set 355

network
of a

Complement

Completebinary

tree

Complete graph

96,

Complexnumbers

CREW 378,382,410,412,415
Critical
node (in an AVL tree) 75
Crossedge, in depth-first search 194

38

complexity

344

Crossproduct

413

Cryptography

295-297

Cube-connected-cycles 390

42

Component

Culberson,

217-223, 249, 259-260,430

biconnected

187, 192,415

connected

strongly connected

86

241

Cycle(in

186

226-233,247, 250,

a graph)

even-cycle problem
Hamiltonian

230

248, 372

odd-cycleproblem

Compression

data

145-148,

path

82,86

DAG, see Directed acyclic graph

B. 247,336
Dantzig,G.
Data compression 145-148,174

344

complexity
186

security

Concurrent-read

88
see CRCW

concurrent-write,

Concurrent-read exclusive-write,
Condensation graph 226
form

normal

Conjunctive

see

346

187,

192,415

Connected graph 186

376, 410

machine

co-NP 368
R. L.

CREW

Data

structures

Context-free language

317

62

252

Deadlock 6

31

representation

Decimal to binary

conversion

26-28,

36

Decision tree 141-143,

174, 177,332-333,
335, 424, 426,438-439
Decoding

106

61-90

Data types, abstract

de Bruijn sequence
Decimal

Connected component
Connection

246

233-234,

174, 177, 180,334-335

Concatenate operation

Constable,

J.

Cut (in a network)

251,260

Computational

363,

CRCW 378,383,410,412,413

space 42

Computer

348, 355-356,358,

371,373,432-433

55, 83

computational

372

3-sets

vertex 262-263,

318

asymptotic

by

34, 263-264
244, 263, 356

Complexity

amortized

414

circuit

Counter

14

352-354,370,371,373

3-coloring

Communication

316

D.

Corasick, M. J. 174

369

plane coloring

time

344,346,368,410

Coppersmith,

k-coloring

266

theorem 347

Cook's

369

edgecoloring

of

polygon, definition
Convolution 405-407

Cole,R. 410-411
Coloring problems

266,276

path

Convex

86

Cohen,
Collision

289

a 273

stretching

Convex

346

286-289

273-277,

approximate

80

Clustering

CNF

36
26-28

to binary

decimal

147

Decreasing DFS numbers

223

Index

468

Decreasing first

364

fit

Decryption

Domino principle

186

sequence 252, 262

Dekel,E. 410

binary search tree

heap 69-70

in a

in a

hashing 80

Double

induction

Deo,N.

for 234-235,

95, 189-197,221-225,

Dynamic

Dershowitz, N.
Deterministic

coloring 262,432
cover 262

4, 113

193,388
definition

algorithm,

DFS, see Depth-first

of

in

158

search

Diameter
of a graph

Edit

113,247

E. A.

247, 254

204-208,

algorithm

317

Directed acyclic graph

197, 199-203,226,
246, 249, 253, 255,258, 260, 338,
208,

376

113-114,131,137, 141,144, 158,

169, 172, 176, 183,264, 278-281,
286, 287, 289, 299-304,311-314,

316-319,
379-380,

383,

394-396,

50-51

382

Divide-and-crush
149

Dobkin,
Dodecahedron

D. P.

175
6

graph

relations

259

paths 23-24

155

steps

149

of parallel

Efficiency
Efficient

algorithms

Election, 1988Presidential
definition of 62

376

of 341

definition

algorithm,

Element,

D. F.

317
336

trees 263

Embedding

377

Emulation

146

Encoding

295

Encryption

end 5
Eppinger,

409-410,431

Divide-and-conquer recurrence

186

Ellipsoid algorithm

algorithms 103-104,

Divide-and-conquer

map 17

Elliott,

371,378,430-431

Distributed algorithms

of a planar

Editing a text file

Edmonds, J. 243

316

Dijkstra, E. W.

83,

Edge-disjoint

340

71

Dictionary

266

the plane

of a graph
Edge-biconnected

115,389

of a set of points

Dijkstra's

286

H.

Edelsbrunner,

Edge

Descendant (in a tree)

W.

247

192,223

192-196

tree

114,

34

decomposition

Ebert,J.

194

108-111,

155-158,407
Ear

nonrecursive implementation

410

programming

429

194

edge

66

S. 174

Dvorak,

Dwork, C.

crossedge 194

numbers

S. E.
variable

Durian, B. 174

227-233, 246-250,388,426-427,
back edge 194
forward

409, 411,

123, 386-389,

114

Dreyfus,
Dummy

378

search

Depth-first

tree) 75

an AVL

440

247,410

Depth of a circuit

DNA

18

technique

Doubling

114

E. V.

Denardo,

73-74

Dense graphs,
algorithms
244-246

Dinic,

91

Double

Double rotation (in

Deletion

Diffie,

of points

Domination

Degree of a vertex
Degree

set 348-349
289

Dominating

295

J. L.

Equations, system
Erdos, P. 175
EREW

86
of

315

378,380-381,383-389,410,

412-415

Errors
in

design

by induction

111-112

Index

in

induction

in

reductions

proofs 28-29
334-336

Franco, J. 369

Euclidean graph 254, 365

traveling

294, 297-298,316,318

Euclid's algorithm
187,247

Euler,L.

formula

17-18,30

387-389,409-411,414
187-190, 247, 251-252, 373,

graph

230, 369
372

problem

Even-cycle

3-sets

Full-history

see EREW

96-98

one-to-one

N.

247,

of polynomials
317

transform

Fourier

Fast

287

M.G.

Federal Express

Feedback-edge

257,

Girth
Goldberg,

309-317,443

numbers 34-35,46,

62

155

comparisons

Golovina,

174
First
fit 364, 370, 374
M. J. 86, 175, 317, 369,380, 410
Fischer,
Flajolet, P. 55
network
238-243,
246, 248, 327, 329,
Flow,
338

Flynn,

M.J.

410

Forest

174,248

Four-Russian

H.

Gonnet,G.

248

247,248
85,86,

175

Goodrich, M. T. 410
416

problem

409

R. L.

55,286

95-96,98-101,185-264,

algorithms

356-363,369,

(in DFS)

194

algorithm

306-109,

A. S.

R. E.

Gondran, M.

325-326, 347-349,

Fox,G. C. 409
Fraenkel,

31

Graph

178

186-187
edge

369

Graham's scan 275-277,287-288

Fortune, S. 287
Forward

Gomory,

Graham,

Johnson sorting

L. I.

Gottlieb, A.

113, 174,247

Folding principle 377

Ford and

S.

Golumbic, M. 247

Gossip

R. W.

Ford,L.R.

Goldstine, H. H. 113
Golomb, S. 369

Fingerprinting

Floyd,

of a graph 255
A. V. 248

Goldwasser,

FIFO (first-in-first-out)
File

49, 60,318

410

E.

274-275, 286

Gillis,J. 30

373, 429

FFT

Fibonacci

265-291,

247,409

Gift-wrapping algorithm

Feng,T. 411

Fich,F.

A.

Gibbons,

321

set

102-104,

331-333,434-436

Geometric series 53

inverse 314-316
Faust,

318

Geometricalgorithms
309-317,443

363, 369

356,

409

E. F.

Gehringer,

Faradzev, LA.

431

86

B. A.

Garey, M.R. 344,355,

GCD 297-298,318

86

R.

248, 366,

Galil,Z. 174,175,247,248
Galler,

294-297

Extendible hashing 86

Fagin,

51-53, 57,

relations

137

Gabow, H.

Exclusive-read exclusive-write,
Exponentiation

248

recurrence

Function,

Exact coverby

409

D. R.

336, 369

247, 248,

S.

146

R. M.

Fulkerson,

432
Even,

Freetrees 83
Fujimoto,

tour technique
Eulerian

410

247

Frequency

constant 53

Euler's

G.

Fredman, M. L.

Frederickson,

365-367

salesman

410

N.

Francez,

351-354,

370-374,

414-415,

426-434
316, 317

biconnected 217;seealso

Biconnected

component

175

469

bipartite

111-112, 117,187,372

470 Index

252, 369, 352-354,

problems

coloring

code

Gray

370,371,373

Greatest common divisor

356

226
186

directed

definition
edge biconnected,
415-416
embedding
Euclidean

of 259

365

254,

255

Hamiltonian

369

243-246,

induced 95-96,

114, 187,251

Haken, D.

55

Half

291

plane

336

Hall, M.

323, 336

P.

theorem

Hamilton's

89

234-238,248,251,260-261,

291, 338,

342, 366,431;seealso

Matching

problems

by induction,

involving

18-19,

representation
in a

strongly

subgraph

subgraph
traversal

connected

206,428

260

443
tree 104
E. 316

Height of a node
M.

Hellman,

174

443

S. T.

Hedetniemi,

undirected 186
unipathic

137-141,

Hedetniemi, S. M.

induced 187
189-199

weighted, definition

86

Heap 68-70, 87, 138-141,148,178,179,

Heapsort

of a

260

80

double

226-233, 247, 250,

86

86

extendible

186

unilateral

78-80,

coalesced

186

251,260

86

universal

263

53-54

79

functions

Hashing

simple 186
split

85

247

Harmonic series

of 83-84

residual 242

selfloops

F.

373

31

R. W.

Harary,

Hash

372

regular

356, 372,

258,

Hansen, W. J.

34, 35

248, 261, 356,

369
path

338

248,

319

243-246,

cycle

Hamming,

186

multigraph

23-24,

puzzle.

Hamiltonian

proofs

6, 243

Hamilton's quaternions

255

matching

336

324,

Hamilton, W. R.

isomorphism 248, 369

planar

287

H.

Hall's

263,433

kernel

369

Guting,R.

Hall,

253

injective

join

174

Guibas,L.J.

187-190,247,251-252,373,

of a

interval

M. 336

Grotschel,

Gusfield, D.

432
girth

290
Gries,D. 4,31,113,175
Guessing 11-14,43-44,47-49, 58,345

graph

Eulerian

55

Grid polygon

115,389

definition of 186
see Directed acyclic
acyclic,

directed,

235, 237, 363

Mesh

see

Grid,

318

398,410

Greene, D. H.

186-187

diameter

A.

Greenberg,

234-235, 244-246

210-211,

algorithms

Greedy

297-298,

318

of polynomials

of
connected, definition
217-234
decomposition
definitions

33-34

ami

complete 96, 244,263,

condensation

320

31, 33-34,

20-23,

open 22,320

348

of a

complement

dense

31

Gray,F.

(continued)

Graph

in a

Heuristic 211,254,369
of

187,201

HEX

369

471

Index

Hibbard,T.
W.

Hillis,

102-104, 282

Hinrichs, K. 287
Hoare,C. A. R.
Hoey, D. 287

Inner

174

248,

House,

design 2

to algorithm

174

Huffman's

174, 177, 180,

334-335
Hwang, K. 409

Incident

J. 174
325

matrix

Increasing subsequence problem

174,336

Iyengar,

S. S.

(and

virtually

choosing

the base of

choosing

the order

of

everywhere)

137,167, 172
17, 30, 92-94,

96-101,112-113,
128,

200,

411

John, J. 174
247

D. B.

Johnson,

Johnson, D. J.

344,

Join graphs

28-29,

reversed

369, 370,

86

202-203,

111-112

217

k-connectivity

element

*th-smallest

143-145, 174,179,

396-399

usage 14
114

generalization

356, 363,

89

Jones, D. W.

18

double

double

355,

174

S. M.

Johnson,

221,282-283

common errors
definition 10

314-316

410

H. V.

Jagadish,

of 186
definition
Indegree of a vertex,
Independent set 19,262-263,356, 371, 433
Induced subgraph 95-96, 114,187,251
3, 9-36

113

248, 369

graph

Itai,A.

134,

113-114,

167-169,

175

Induction

24,

paradox

Inverse fast Fourier transform

Isomorphism,

Incerpi,

281-285,

263,433

graph

Inventor's

389-390,401-403,415

Hypercube

Interval

Invariant, loop 26-28, 31,36,

176-177

175

J. W.

Hunt,

267-270,

problems

287,289-291

145-148,

encoding

125-127

search

Intersection

148, 174

D. A.

378,

310

Interpolation

Interpolation

B.C.

Huffman,

network, algorithms

389-403

Hu,T. C. 248
Huang,

Interconnection

369,410

analogy

331,336,

54

Integral

114, 128, 310,412

92-94,

71-72

361-363,371

261, 287,

305,316,336,410
E.

178

69-70

linear programming

Integer

85, 86, 175,

Hopcroft, J. E.
rule

174,

130

sort

410

H.J.

Horowitz,

binary search tree

in a heap

Hong,J.-W. 409

Horner's

305

product

in a

Hofri, M. 55
Hoover,

142

bound

Insertion

248

CM.

Hoffman,

10-11,25

Information-theoretic lower
Injective graph 253

127, 131,135,137,
In-placesort
Input size 38,294

175

D. S.

Hirschberg,

of

variations

86

N.

elimination
D. 410

Hidden-line

25,244-246

22-24, 93,
strengthening the hypothesis
105-109, 113, 145,162, 167-169,
281, 282, 379, 418, 423,428, 435,
437, 442

strong 10, 109,157

411

Kailath,T.

Kapur,

S. S.

Karatsuba,

286

A.

Karlin,A. R.
Karmarkar,

316

86,411

N.

336,370

M. 174,175,243,261,346,
255
Kernel of a graph
B. W. 370
Kernighan,
Karp, R.

Khachian,

L. G.

336

368,

370

472

Index

Killian,

J.

369

Linesin

Klee,V. 336,369
175

S. T.

Klein,

KMP algorithm

174, 323,

148-155,

Knuth,

55, 77, 85, 86,94,

155,

357,

160,

173, 174,175,316,369,436
Knuth-Morris-Pratt

148-155,

algorithm

323,427

R. J.

Litwin,

W.

B.

Kruskal,J.

411
86

Load balancing

380

Localreduction

356

54

rules of

subsequence

increasing

187

Loopinvariant

31, 36,

26-28,

175,247,410

411

D.

Lopresti,

31, 175,248,336
174, 178, 180, 331-334, 339

Lovdsz,L.

Lowerbound

86

142

i nformation-theoretic

75

M. A.

Langston,

Lower triangular
Lucas, E. 247

30

Landis, Y. M.

Language-recognition 342
Las Vegas

344

A.M.

248, 369
174

of a path

370

164-166,175

409
G. K.

Manacher,

174

Manber, U. 4, 55,86,

318

113,

Manna, Z. 4,

113, 173

Masek,W.J.317

247,432

problems

Matching

198

234-238,248,251,

260-261, 291, 338, 342,366,

b 338
bipartite 235-238,

Lexicographic sort 128

Liestman, A. L. 443

35,234

Lin,S. 370

string

148-155

Line

3-dimensional 356

14,265

266

Linear
Linear
Linear

congruential method 160

(in hashing) 79
probing
322, 327-331, 336,
programming

338-339

induction,

Mathematical

281-287,435

431

324,337, 357

perfect

Line-sweep
technique

398,

369

R.

Manber,

201

Level (in breadth-first search)

L. A. 368
Levin,

Line segment

175, 248,

410,440

Lenstra, J. K. 370
L.

336

problem

Malek, M.

114

Lempel,A. 174,248

Lesniak,

T.J.

Majority

Lawler, E. L. 248,366,
Least common multiple
Leiserson, C. E. 411
Length

E. M.

Lynch,

Maggs, B. M.

algorithms

33

Lattice point
Law,

Luks,

algorithms 159;seealso

Probabilistic

339

matrix

Lueker,G.S. 55

174

nondeterministic

141-143

for sorting

E. 369,380,410
I.

Lakatos,

113-114, 134,

176-177

Kung, H.T. 411

Ladner,R.

167-169,

175

317

D.

T.

174,

Lipton,

Longest

Kbnigsberg bridge problem

Kronrod, M. A. 174,317

Kurtz,

409
Lipson,J.D. 309,316

Logarithms,

Komlos,J. 410
Kbnig,

64

Lipovski, G. J.

427

8, 108-111,114-115,

370
D. E.

in the plane 13

position

general
lists

Linked

problem

Knapsack

of 328

form

standard

Linear search 173

King, K.N. 113

see Induction

Matrix

adjacency 84, 99,325,

339

Boolean

304-309,317,

multiplication

319,320,325-326
and

graphs

246, 325-326,

336, 339-340

Index
Molecular biology 149,155
J. C. 247
Molluzzo,

325

incident

336

inversion

lower triangular

Monotonically increasing
Monte Carlo algorithms
Moore, J. S. 155, 174

339

301-309,316,318,

multiplication

325-326, 333-334,336,339,
398-400,410,412,416

square of a 319, 334

Multiple-instruction multiple-data

upper triangular

339

vector multiplication,

of Booleanmatrices

404-405

by

of general

242

Maximum-consecutive subsequence
in

problem

6, 247

Median

spanning

tree

see also fan-

Megiddo, N. 411

217-218

Menger,K.

Murty,

U. S.

Myers,

E. W.

(of a

interconnection 378, 389-403

389,398-400,414
A. R.

Meyer,

317,369

Miller, R. 297
MIMD

Minimum-cost spanning
tree
247,
208-212,
249, 255,257, 338,340, 365, 428, 429
Minimum
in a set 143-144, 169-171,
element

176,380-382
247-248

Minoux,M.

Minty,

G. L.

336

A.

Misra,J.

175,409

Modeof a multiset
A. 247
A.

410

246, 248,

I. 94

Nil

pointer

86,287

66

Nishizeki, T.

247

definition

of

Node,

Nonconvex

186; see also Vertex

267-270

polygon

Nondeterministic algorithm
344-346,
language recognition 344
model

of computation

Notation

171-173

324, 327,

329,338

Nonuniform

173

Mirzaian,

238-243,

Nievergelt, J.

E. 248

Miniekas,

Network flow
Newton,

376,410

393-396

on a

sorting

of trees 390

329

241

cut

393

split

in 256

broadcast

communication 146,208,256,

parallel 394-396,413

Moitra,

paths 255, 258

vertex) 19

shortest

Network

393-395,410

Moffat,

410

cost,

Negative

104,130-131,174,281,

mergesort

Mesh

164,171

4,175,317

Neighborhood

Merge

element

175,316,336
R. 247

Munro,I.

Nassimi, D.

286

K.

Mehlhorn,

309, 316-317,

171-173

Multiset

174, 179;
element

144-145,
smallest

of an

Multiplicity

McCreight, E. M. 175

MCST,seeMinimum-cost
McVoy, L. 4, 248

339,

319

176,380-382
Mazeproblem

316, 318,

301-309,

398-400,410,412,416
of polynomials 298-300,

143-144, 169-171

a set

matrices

325-326,333-334,336,

106-107

Maximum element

317,

304-309,

319-320

theorem

min-cut

Max-flow

410

376,

Multiplication

F. 30

Maurolycus,

378

186,259,262

Multigraph

333

transpose

155,174

Multicomputer

333-334

symmetric

48

function
159

174

B. M. E.

Moret,

Morris, J. H.

319

permutation

473

algorithms

graphs 83-84,
geometry

186-187

265-266

370

143

Index

474

NP-completeness 341-374
problems

NP-complete

345-357,368-374,429,

Parter, S. V.

246,

Partition

439-440

398-401,404-407

248

133-135, 144-145,

(for quicksort)

176

293-320, 379-380,

algorithms

Numeric

I. 409

Parberry,

175, 244,

Partition

357

problem

Pascal 4

o Notation

Pascal'striangle

41

39

O Notation

41-42,
OONotation
Objectivefunction

Octal tree

174, 316,
327,361

Odd-eventransposition

246

sort

convex 266, 276

411,

390-393,

Hamiltonian

174

see

shortest,

Pattern

96-98
22-23, 320

function

Open Gray code

Order statistics
143-145,
Ore,0. 247,248

174, 179,396-399

287

Outdegree of a vertex,

definition

186

of

M. C.

Pearl,

J.

369

Perfect

matching

Perfect

shuffle

376

174-175

Pigeonhole

316

V.

Pan,

C. H.

Papadimitriou,

24

inventor

Paradox,

Parallel algorithms 214, 216,304,

addition
379-380,411
of

efficiency

398-401,404-405,

410,416
of

375-379,

routing

24,30,31,113,169
287-290

290

regular 291

simple 266,270-272,288,331-333

401-403

sequence
comparisons

407-409,

376

division

Polynomial

411, 416

sorting 390-396,410,411,413,414
tree operations

33, 266-277,

grid

385-387

of

165

public-opinion

convex 266, 273-277,286-289

selection 396-399,410,414

speedup

Poll,

Polygon

409-411

prefix 382-385,410,411,413,415
ranking

17

map

Polya,G.

merge 394-396,410,413
models

Pivot 133, 136-137,144,396

Planar
248, 338
graph
Planar

377

multiplication

174

Plotkin, S. A. 336
Plummer, M. D. 248

376

folding principle
matrix

375-416

319

problem 330, 339

34, 175, 395
principle

N. 86,
Pippenger,
L.
369
Pitt,

248, 336

182

matrix

396-397,409

Pipeline

247

Pang,C.

142,

Philanthropist

Packing, bin 363-365,370-371,374

35,234

390

Perfect speedup

Permutation

345

matching

Permutation

341

see String

matching,

Paull,

Perl,Y.

P (class of problems)
problem

problems

Shortest-paths

simple 186

l-in-3SAT371

P=NP

372-373

258,

201

length of a

Ofman,Yu. 316

T.

82, 86

compression

414

Ottmann,

239

augmenting

393-396,410,411

Odd-even mergesort

One-to-one

236

alternating
233-234,

A. M.

174,317

186

Path

390

Odd-cycleproblem

Odlyzko,

M. S.

Paterson,

410

31

O. 55

Patashnik,

evaluation
greatest
multiplication

387-389,

319

318

92-94, 114, 128,310,412

common

divisor
298-300,

318
309, 316-317,

Index

Polynomial-time

Reed, D. A.

369

hierarchy

Pool 87
155, 174

V. R.

Pratt,

146

constraint

Prefix

Prim,R.C. 247
testing 369
root of unity

Primitive
Priority

queue

62, 68,

Priority

search

207,246

Reingold,

E. M.

Repeated

squaring

problems

Pruhs,K.

175
algorithm

A. H.

Kan,

160

287

C.

Ronse,

Rooted tree, definition

295-297,316

165

poll

Purdom, P. W.

55, 369

line

Rotation

(in AVL trees)

68, 86

62,
Queue, priority
Quicksort 131-137,144,174,176

problem

Routing

in

Raghavan, P.

175

search

285,435

in linked

number

Sankoff,D. 175

lists 385-387

SAT

317

46-53, 55,56,
divide-and-conquer 50-51
with

full

history

unfolding

Reduction
see

SAT, l-in-3 371

175,183

Recurrence relations

57, 58,

51-53, 137
194

215, 239, 246,317,321-340,442;

also NP-completeness

problem

(satisfiability)

346-347,

350-354,357,370,371,440

186

sequence

317

N.

Santoro,

Reachable vertex

175

S. L.

Salzberg,

Rao,K.R. 317

Recursion,

369,410

Sahni,S.

Saks,M. 175

Rank 143

Realizable

317,409

Rytter,W.

Random access machine

344
number 160, 175,415

Rational

42

411

A.G.

Ranking

of an algorithm

Ryser,H.J. 175

Random
Range

scheme

410

time

Running

127-129

sort

Ranade,

174,287,369

401-403

encryption

317

Runge,C.
Rabin, M. O.

329-330

a hypercube

L.

Rudolph,

183

tournament

Routing

RSA public-key
316

409-410

Quinn,M.J.

75

304,317

390

319

186

270

Rotating

Round-robin
Quaternions

66, 83,

of

113

A. L.

Rosenberg,

Roth,M.

Radix

55

F. S.

Rodeh,M.175,336,410

357

encryption scheme

Quadtree

G. 370

149

RNA

368

Public-opinion

244-246

155

410

D.

Roberts,

Pseudorandom numbers

83-84

Rivest,R.L. 174,295,316

278-281

Pseudopolynomial

85, 369
297, 317

Ring 305
Rinnooy

Proximity

291

410

Richards,

158-164,175,287,

175

proof

Public-key

polygon

Revision maintenance

320,402-403

PSPACE

Regular

Representation of graphs
Residual graph 242
Reversed induction
25,

313,315
86

Probabilistic algorithms
Probabilistic

graph 372

Reif,J.H.

Prefix, parallel 382-385,410, 411,413, 415

F. P. 286, 287, 365
Preparata,
Primality

409

Regular

59

SAT,2-in-4371
368

Savitch,W.J.

Saxe, J.

B. 55

Schaback,

Scheduling
Schbnhage,

174

R.

185
A.

174

295-297,

Index

476

Schorn,P. 287

Simplex algorithm

336

A.

Schrijver,

Schwab, B. 86

Single-source
Sink 99

Search

Sizeof an

120-125,285,342,431

binary

search trees 71

in binary

depth-first

227-233, 246-250,

Smit,G.V.

388,426-427,429

125-127

interpolation

linear 173

80

Seed 160
143-145, 174, 179,

problem

in a graph

Separate chaining

186

337,

407-409

119-183,380-383; seealso

Shiloach,
Shmoys,

295,316
365

Minimum-cost

spanning

Spencer,J. 175,247

Y. 410
D. B. 370

263

Square

of a matrix

Square

root

429

Siegel,H.J. 411

319, 334

272

304

Stable algorithm

255,
428,

247

P. M.

Split graph

185, 201-208,
Shortest-paths problems
212-214, 247, 249, 253,254,
257, 258, 322, 325, 338,427,
Shrira, L. 410

Stack 194
of linear

Standard

form

Standish,

T. A.

Stanton, D. 173

409

Steele,G.L. 410

Simple graph

Steiglitz,

path

Simple polygon

programming 328

85

SIMD 376,410
Simple

186

186

K.

248,336

Stirling's approximation
266,

see also
tree

Speedup 376

378-389

Spira,

D. P.

187

tree 35,186,371,429;

Spanning

270-272,

430

173

E. H.

Spanning forest

174

Siewiorek,

130

selection

Spafford,

297

Shared-memory algorithms
Shell, D. L. 174
Shellsort

131-137, 144, 174, 176

topological 199-201,249,427,
Space complexity 42

algorithms

M. I. 286-287,

390-393, 411,414

127-129

radix

cover 355

Shamos,

transposition
390-396,413-414

quicksort

155-158,175,

181-182,317,324-325,
Sequence

141-143

393-396,410

parallel

407-409

337,

Set algorithms

128, 178
for

bound

network

106-107, 119-183,317,

Sequence comparisons

Shamir, A.

178

130

odd-even

algorithms

J.

137-141, 174

mergesort 130-131

79

324-325,

Shallit,

heapsort

lower

Self-adjusting trees 86

Sequence

Johnson

lexicographic

130

sort

Selfloops

142, 164

127-129,

Ford and

insertion

396-399
Selection

31

in-place 127,131,174

Segall,Z.409
Selection

369
I. S.

bucket

R. 174

Sedgewick,

B. 113

Sorting

186

Security

86

Smith-Thomas,
M. 410
Snir,

Sominskii,

collision (in hashing)

Secondary

102-104,115

174

Solovay,R.

207,246

priority

paths

328

D. D.

Sleator,

189-197,221-225,

376

201-208

38,294

input

Slackvariable

249,426, 427
95,

shortest

Skyline problem

189, 198-199,238,246,

95,

breadth-first

336

multiple-data

Single-instruction

288, 331-333

Stockmeyer,

L.

369

54, 142

477

Index

Q. F.

410

Strassen, V.

369

Stout,

Strassen's

301-305, 316, 318,319,

algorithm

the induction

hypothesis

10,
Strong
Strongly connected
component
250,251,260

binary 34, 263-264

decision 141-143,174,177,332-333,

247,

226-233,

complete

124

problem

subsequence

Subsequence, longest
increasing
Summation problems 11,15-16
Summation

Supporting line

System

378, 404-409,

323-324
411,416

T. G.

node

Tarry,

86

369,

352-354,370,371,373

3-Dimensional

complexity

42

Timetable

Towers

of Hanoi

Tractable problem

249,427,

430

116

248, 317,322,

matrix 339

Tremaux,

247

W. T.

of a matrix

333,414

114
346

assignment

Tsitsiklis, J. N. 409
TSP, see Traveling
317
Tuckey,J.W,

Turing machine
Turner, J. S.
W. T.

344,

369
247

2-in-4SAT 371
Uhr,

L.

396

see also
tree

365

inequality

339

Transpose

spanning

Triangular

Tutte,

341

closure 214-216,

Transitive

Triangle

Truth

357,370, 371,439

374
Topological sort 199-201,
Toussaint, G. 286

Minimum-cost

Trotter,

356

Matching

3SAT 350-354,
Time

247, 248,

174,

247

G.

35,186,371,429;

spanning

55, 85,86,
410,431

247,

365,428,

208

shortest-path

R. E.

3-Coloring

208-212,

66,83, 186

rooted

30

67-68

390

quad

175

Takaoka,T. 247,426
Tarjan,

67

104

spanning

self-adjusting

Talmud

in

249, 255,257, 338,340,

octal 390

Szemeredi, E. 410
Szymanski,

representation of a

minimum-cost

175

A.

263

for

implicit representation of a
insertion
into a 71-72
machine 396-398

Szego,G. 31, 169

Szekeres,

a 73-74

from

of a

height

376

algorithms

Systolic

192-196

193

free 83

of equations 315
of distinct representatives

System

depth-first

explicit

matrix 333-334

Symmetric

in

embedding

274

Synchronization

descendant
deletion

54-55

43-46,

technique

167-169

438-439

426,

335,424,

114, 187

95-96,

198-199

breadth-first

Subgraph 186-187
induced

71-77, 86

binary search
109, 157

induction

Stuttering

115

34,

binary

86

H. R.

221

biconnected

180-181,323,427
Strong,

75

balancing

177,

174-175,

148-155,

189-199

ancestor in a 193
AVL 75-77,104, 114-115

442

435,437,

428,
matching

365-367

Euclidean

Traversal (of a graph)

Tree 18,34,66-77,186

113,145, 162,
22-24, 93, 105-109,
167-169, 281, 282, 379,418,423,

String

390-393,411,414

Traveling salesman 356, 365-367,374

326

Strengthening

sort

Transposition

salesman

346-347

problem

429

Index

478

Voronoi diagram

175

UkkonenE.

85, 86,175,305,

J. D.

Ullman,

316,

336, 409,

287

164

Voting

411
definition

graph,

Undirected

Union of

of 83,

186

260

graph

Unilateral

290

polygons

problem 80-83, 86, 256,429

graph 260
Unipathic
Universal hash functions 86
Union-find

Upfal,E. 86,411
matrix

triangular

Upper

Wagner, R. A.

175

Waldinger,R.

173

248

Warren, H. S.

248

S.

Warshall,

Wegman, M. N.

86

Weighted

187

graph

Weiner,P. 175

339

173

D.

White,

Whitney, H. 217

Vacca,G. 30

A.

Wigderson,

Vaidya,

P. M.

366

Wildcard

Valiant,

L. G.

369,402,410

Wilf,

Nat, M. 174
J. 86, 247,
Leeuwen,

van
Vertex

83,

336

186

Wirth, N. 85

cover 262-263,348,

355-356,

371,373,432-433

degreeof 186
outdegree of

of a planar

17

Vishkin,

U.

Neumann,

38-39

X3Cproblem

372

I. M.

Yaglom,
410

55,86,174
Ziv,J.

J.

31

Yao,A. 175,247

VLSI 263
von

173

186

Virus 173
Vitter,J.S.

Worm

Wrap-around 398

266
of

reachability

Wood, D. 287

186

map

of a polygon

358, 363,

Worst-case

186

of

369

algorithm

Winograd's

basis 260,431

indegree

H. S.

Williams, J. W. J. 174
Winograd, S. 316

der

van

175,369,411

174

113, 174

174

Zorat, A.

410

301,316,319

\342\200\242

Creative

Ap

i roach

i
of Arizona

University

This book

emphasizes the creative aspects of algorithm design. It teaches the thinking behind the
of algorithms. Algorithms are describedby following
the steps leading to their development
(sometimes
including wrong steps). The heart of this creative process lies in an analogy between
theorems
combinatorial
The book contains
provingmathematical
algorithms.
by induction and designing

creation

hundreds
understanding

of problems and examples. It is designed

to enhance
of the principles behind algorithm
design.

the

reader's

abilities and

problem-solving

HIGHLIGHTS

coverage of algorithms.
known
many
techniques for designing
over 500 mostly nonroutine
exercises,

\342\200\242

Current,

up-to-date

\342\200\242
Covers
\342\200\242
Contains
\342\200\242
Discusses
\342\200\242
Covers

implementation
parallel

\342\200\242
Includes

The

of efficient

introduction

ABOUT

details separately from

algorithm

and unifies

many

of them.

solutions to about

a quarter

design

of them.

ideas.

examples

for most algorithms.

is becoming an important tool in many fields

algorithms
is self-contained,
and
can serve as
biology,
engineering. This book,which
computation.
algorithms and non-numeric
non-numerical

statistics,

mathematics,
including
an

and gives

algorithms.

pseudocode

design

algorithms,

to

THE AUTHOR

has been an associate professor of computer

science
at the University of Arizona
since
received his Ph.D. in computer science from the University
of Washington in 1982. He was
an assistant professor and later an associate professor at the University
of Wisconsin, Madison. His
researchareas include design of algorithms, distributed
and parallel
computing, and computer
He received
networks.
the prestigious Presidential
Award in 1985, and the Best Paper
\\bung
Investigator
Award
of the seventh International Conferenceon Distributed
1987.
Computing
Systems,
Udi

Manber

1987. He

90000

9\"780201\"120370

ISBN

Theory
Addison-Wesley

Publishing

Company

0-201-12037-E