You are on page 1of 162

Data Structures and Programming Lecture 1

Why Data Structures? In my opinion, there are only three important ideas which must be mastered to write interesting programs.

Iteration - Do, While, Repeat, If Data Representation - variables and pointers Subprograms and Recursion - modular design and abstraction

At this point, I expect that you have mastered about 1.5 of these 3. It is the purpose of Computer Science II to finish the job. Data types vs. Data Structures A data type is a well-defined collection of data with a well-defined set of operations on it. A data structure is an actual implementation of a particular abstract data type. Example: The abstract data type Set has the operations EmptySet(S), Insert(x,S), Delete(x,S), Intersection(S1,S2), Union(S1,S2), MemberQ(x,S), EqualQ(S1,S2), SubsetQ(S1,S2). This semester, we will learn to implement such abstract data types by building data structures from arrays, linked lists, etc. Modula-3 Programming Control Structures: IF-THEN-ELSE, CASE-OF Iteration Constructs: REPEAT-UNTIL (at least once), WHILE-DO (at least 0), FOR-DO (exactly n times). Elementary Data Types: INTEGER, REAL, BOOLEAN, CHAR Enumerated Types: COINSIDE = {HEADS, TAIL, SIDE} Operations: +, -, <, >, #

Elementary Data Structures ArraysThese let you access lots of data fast. (good) You can have arrays of any other data type. (good) However, you cannot make arrays bigger if your program decides it needs more space. (bad) RecordsThese let you organize non-homogeneous data into logical packages to keep everything together. (good) These packages do not include operations, just data fields (bad, which is why we need objects) Records do not help you process distinct items in loops (bad, which is why arrays of records are used) SetsThese let you represent subsets of a set with such operations as intersection, union, and equivalence. (good) Built-in sets are limited to a certain small size. (bad, but we can build our own set data type out of arrays to solve this problem if necessary) Subroutines Subprograms allow us to break programs into units of reasonable size and complexity, allowing us to organize and manage even very long programs. This semester, you will first encounter programs big enough that modularization will be necessary for survival. Functions are subroutines which return values, instead of communicating by parameters. Abstract data types have each operation defined by a subroutine. Subroutines which call themselves are recursive. Recursion provides a very powerful way to solve problems which takes some getting used to. Such standard data structures as linked lists and trees are inherently recursive data structures.

Parameter Passing There are two mechanisms for passing data to a subprogram, depending upon whether the subprogram has the power to alter the data it is given. In pass by value, a copy of the data is passed to the subroutine, so that no matter what happens to the copy the original is unaffected. In pass by reference, the variable argument is renamed, not copied. Thus any changes within the subroutine effect the original data. These are the VAR parameters. Example: suppose the subroutine is declared Push(VAR s:stack, e:integer) and called with Push(t,x). Any changes with Push to e have no effect on x, but changes to s effect t. Generic Modula-3 Program I
MODULE Prim EXPORTS Main; (* Prime number testing with Repeat *) IMPORT SIO; VAR candidate, i: INTEGER; BEGIN SIO.PutText("Prime number test\n"); REPEAT SIO.PutText("Please enter a positive number; enter 0 to quit. "); candidate:= SIO.GetInt(); IF candidate > 2 THEN i:= 1; REPEAT i:= i + 1 UNTIL ((candidate MOD i) = 0) OR (i * i > candidate); IF (candidate MOD i) = 0 THEN SIO.PutText("Not a prime number\n") ELSE SIO.PutText("Prime number\n") END; (*IF (candidate MOD i) = 0 ...*) ELSIF candidate > 0 THEN SIO.PutText("Prime number\n") (*1 and 2 are prime*) END; (*IF candidate > 2*) UNTIL candidate <= 0; END Prim.

Generic Modula-3 Program II


MODULE Euclid2 EXPORTS Main; (*17.05.94. LB*)

(* The Euclidean algorithm (with controlled input): Compute the greatest common divisor (GCD) *) IMPORT SIO; VAR a, b: INTEGER; x, y: CARDINAL; <*FATAL SIO.Error*> BEGIN (*statement part*) SIO.PutText("Euclidean algorithm\nEnter 2 positive numbers: a:= SIO.GetInt(); WHILE a <= 0 DO SIO.PutText("Please enter a positive number: a:= SIO.GetInt(); END; (*WHILE a < 0*) b:= SIO.GetInt(); WHILE b <= 0 DO SIO.PutText("Please enter a positive number: b:= SIO.GetInt(); END; (*WHILE b < 0*) "); (* input values *) (* working variables *)

");

");

x:= a; y:= b; (*x and y can be changed by the algorithm*) WHILE x # y DO IF x > y THEN x:= x - y ELSE y:= y - x END; END; (*WHILE x # y*) SIO.PutText("Greatest common divisor = "); SIO.PutInt(x); SIO.Nl(); END Euclid2.

Programming Proverbs KISS - ``Keep it simple, stupid.'' - Don't use fancy features when simple ones suffice. RTFM - ``Read the fascinating manual.'' - Most complaints from the compiler can be solved by reading the book. Logical errors are something else. Make your documentation short but sweet. - Always document your variable declarations, and tell what each subprogram does. Every subprogram should do something and hide something - If you cannot concisely explain what your subprogram does, it shouldn't exist. This is why I write the header comments before I write the subroutine.

Program defensively - Add the debugging statements and routines at the beging, because you know you are going to need them later. A good program is a pretty program. - Remember that you will spend more time reading your programs than we will. Perfect Shuffles
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 1 27 2 28 3 29 4 30 5 31 6 32 7 33 8 34 9 35 10 36 11 37 12 38 13 39 14 40 15 41 16 42 17 43 18 44 19 45 20 46 21 47 22 48 23 49 1 14 27 40 2 15 28 41 3 16 29 42 4 17 30 43 5 18 31 44 6 19 32 45 7 20 33 46 8 21 34 47 9 22 35 48 10 23 36 49 11 24 37 50 12 25 1 33 14 46 27 8 40 21 2 34 15 47 28 9 41 22 3 35 16 48 29 10 42 23 4 36 17 49 30 11 43 24 5 37 18 50 31 12 44 25 6 38 19 51 32 13 1 17 33 49 14 30 46 11 27 43 8 24 40 5 21 37 2 18 34 50 15 31 47 12 28 44 9 25 41 6 22 38 3 19 35 51 16 32 48 13 29 45 10 26 42 7 1 9 17 25 33 41 49 6 14 22 30 38 46 3 11 19 27 35 43 51 8 16 24 32 40 48 5 13 21 29 37 45 2 10 18 26 34 42 50 7 15 23 31 39 47 4 1 5 9 13 17 21 25 29 33 37 41 45 49 2 6 10 14 18 22 26 30 34 38 42 46 50 3 7 11 15 19 23 27 31 35 39 43 47 51 4 8 12 16 20 24 28 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

47 48 49 50 51 52

24 50 25 51 26 52

38 51 13 26 39 52

45 26 7 39 20 52

23 39 4 20 36 52

12 20 28 36 44 52

32 36 40 44 48 52

42 44 46 48 50 52

47 48 49 50 51 52

Software Engineering and Top-Down Design Lecture 2


Software Engineering and Saddam Hussain Think about the Patriot missiles which tried to shoot down SCUD missiles in the Persian Gulf war and think about how difficult it is to produce working software! 1. How do you test a missile defense system? 2. How do you satisfy such tight constraints as program speed, computer size/weight, flexibility to recognize different types of missiles? 3. How do you get hundreds of people to work on the same program without getting chaos? Even today, there is great controversy about how well the missiles actually did in the war. Testing and Verification How do you know that your program works? Not by testing it! ``Testing reveals the presence, but not the absence of bugs.'' - Dijkstra Still, it is important to design test cases which exercise the boundary conditions of the program. Example: Linked list insertion. The boundary cases include:

insertion before the first element. insertion after the last element.

insertion into the empty list. insertion between element (the general case).

Test Case Generation In the Microsoft Excel group, there is one tester for each programmer! Types of test cases include: Boundary cases - Make sure that each line of code and branch of IF is executed at least once. Random data - Automatically generated test data can be useful to test user patterns you might otherwise not consider, but you must verify that the results are correct! Other users - People who were not involved in writing the program will have vastly different ideas of how to use it. Adversaries - People who want to attack your program and have access to the source for often find bugs by reading it. Verification But how can we know that our program works? The ideal way is to mathematically prove it. For each subprogram, there is a precise set of preconditions which we assume is satisfied by the input parameters, and a precise set of post-conditions which are satisfied by the output parameters. If we can show that any input satisfying the preconditions is always transformed to output satisfying the post conditions, we haveproven the subprogram correct. Top-Down Refinement To correctly build a complicated system requires first setting broad goals and refining them over time. Advantages include: A hierarchy hides information - This permits you to focus attention on only a manageable amount of detail.

With the interfaces defined by the hierarchy, changes can be made without effecting the rest of the structure - Thus systems can be maintained without being ground to a halt. Progress can be made in parallel by having different people work on different subsections - Thus you can organize to build large systems. Stepwise Refinement in Programming The best way to build complicated programs is to construct the hierarchy one level at a time, finally writing the actual functions when the task is small enough to be easily done.

Build a prototype to throw away, because you will, anyway. Anything difficult put off for last. If necessary, decompose it into another level of detail.

Most of software engineering is just common sense, but it is very easy to ignore common sense. Building a Military Threat Module Build-Military: The first decision is now to organize it, not what type of tank to buy. Several different organizations are possible, and in planning we should investigate each one:

Offense, Defense Army, Navy, Air Force, Marines, Coast Guard ...

Procedure Army: Tanks, Troops, Guns ... Procedure Troops: Training, Recruitment, Supplies Top-Down Design Example ``Teaching Software Engineering is like telling children to brush their teeth.'' anonymous professor. To make this more concrete, lets outline how a non-trivial program should be structured.

Suppose that you wanted to write a program to enable a person to play the game Battleship against a computer. Tell me what to do! What is Battleship? Each side places 5 ships on a grid, and then takes turns guessing grid points until one side has covered all the ships: For each query, the answer ``hit'', ``miss'', or ``you sunk my battleship'' must be given. There are two distinct views of the world, one reflecting the truth about the board, the other reflecting what your opponent knows. Program: Battleship Interesting subproblems are: display board, generate query, respond to query, generate initial configuration, move-loop (main routine). What data structure should we use? Two-dimensional arrays. How do I enforce separation between my view and your view?

Data Structures and Programming Lecture 3


Steven S. Skiena Stacks and Queues The first data structures we will study this semester will be lists which have the property that the order in which the items areused is determined by the order they arrive.

Stacks are data structures which maintain the order of last-in, first-out Queues are data structures which maintain the order of first-in, first-out

Queues might seem fairer, which is why lines at stores are organized as queues instead of stacks, but both have important applications in programs as a data structure. Operations on Stacks The terminology associated with stacks comes from the spring loaded plate containers common in dining halls. When a new plate is washed it is pushed on the stack. When someone is hungry, a clean plate is popped off the stack. A stack is an appropriate data structure for this task since the plates don't care about when they are used! Maintaining Procedure Calls Stacks are used to maintain the return points when Modula-3 procedures call other procedures which call other procedures ... Jacob and Esau In the biblical story, Jacob and Esau were twin brothers where Esau was born first and thus inherited Issac's birthright. However, Jacob got Esau to give it away for a bowl of soup, and so Jacob went to become a patriarch of Israel. But why was Jacob justified in so tricking his brother??? Rashi, a famous 11th century Jewish commentator, explained the problem by saying Jacob was conceived first, then Esau second, and Jacob could not get around the narrow tube to assume his rightful place first in line!

Therefore Rebecca was modeled by a stack. ``Push'' Issac, Push ``Jacob'', Push ``Esau'', Pop ``Esau'', Pop ``Jacob''

Abstract Operations on a Stack


Push(x,s) and Pop(x,s) - Stack s, item x. Note that there is no search operation. Initialize(s), Full(s), Empty(s), - The latter two are Boolean queries.

Defining these abstract operations lets us build a stack module to use and reuse without knowing the details of the implementation. The easiest implementation uses an array with an index variable to represent the top of the stack. An alternative implementation, using linked lists is sometimes better, for it can't ever overflow. Note that we can change the implementations without the rest of the program knowing! Declarations for a stack
INTERFACE Stack; (* Stack of integer elements *) TYPE ET = INTEGER; PROCEDURE PROCEDURE PROCEDURE PROCEDURE END Stack. Push(elem : ET); Pop(): ET; Empty(): BOOLEAN; Full(): BOOLEAN; (*14.07.94 RM, LB*) (*element type*) (*adds element to top of stack*) (*removes and returns top element*) (*returns true if stack is empty*) (*returns true if stack is full*)

Stack Implementation
MODULE Stack; (*14.07.94 RM, LB*) (* Implementation of an integer stack *) CONST Max = 8; (*maximum number of elements on stack*)

TYPE S = RECORD info: ARRAY [1 .. Max] OF ET; top: CARDINAL := 0; (*initialize stack to empty*) END; (*S*) VAR stack: S; (*instance of stack*)

PROCEDURE Push(elem:ET) = (*adds element to top of stack*) BEGIN INC(stack.top); stack.info[stack.top]:= elem END Push; PROCEDURE Pop(): ET = (*removes and returns top element*) BEGIN DEC(stack.top); RETURN stack.info[stack.top + 1] END Pop;

PROCEDURE Empty(): BOOLEAN = (*returns true if stack is empty*) BEGIN RETURN stack.top = 0 END Empty; PROCEDURE Full(): BOOLEAN = (*returns true if stack is full*) BEGIN RETURN stack.top = Max END Full; BEGIN END Stack.

Using the Stack Type


MODULE StackUser EXPORTS Main; (*14.02.95. LB*) (* Example client of the integer stack *) FROM Stack IMPORT Push, Pop, Empty, Full; FROM SIO IMPORT Error, GetInt, PutInt, PutText, Nl; <* FATAL Error *> (*suppress warning*) BEGIN PutText("Stack User. Please enter numbers:\n"); WHILE NOT Full() DO Push(GetInt()) (*add entered number to stack*) END; WHILE NOT Empty() DO PutInt(Pop()) (*remove number from stack and return it*) END; Nl(); END StackUser.

FIFO Queues Queues are more difficult to implement than stacks, because action happens at both ends. The easiest implementation uses an array, adds elements at one end, and moves all elements when something is taken off the queue. It is very wasteful moving all the elements on each DEQUEUE. Can we do better? More Efficient Queues Suppose that we maintaining pointers to the first (head) and last (tail) elements in the array/queue?

Note that there is no reason to explicitly clear previously unused cells. Now both ENQUEUE and DEQUEUE are fast, but they are wasteful of space. We need a array bigger than the total number of ENQUEUEs, instead of the maximum number of items stored at a particular time. Circular Queues Circular queues let us reuse empty space! Note that the pointer to the front of the list is now behind the back pointer! When the queue is full, the two pointers point to neighboring elements. There are lots of possible ways to adjust the pointers for circular queues. All are tricky! How do you distinguish full from empty queues, since their pointer positions might be identical? The easiest way to distinguish full from empty is with a counter of how many elements are in the queue. FIFO Queue Interface
INTERFACE Fifo; (* A queue of text elements *) TYPE ET = TEXT; PROCEDURE PROCEDURE PROCEDURE PROCEDURE END Fifo. Enqueue(elem:ET); Dequeue(): ET; Empty(): BOOLEAN; Full(): BOOLEAN; (*14.07.94 RM, LB*) (*element type*) (*adds element to end*) (*removes and returns first element*) (*returns true if queue is empty*) (*returns true if queue is full*)

Priority Queue Implementation


MODULE Fifo; (*14.07.94 RM, LB*) (* Implementation of a fifo queue of text elements *) CONST Max = 8; queue*) (*Maximum number of elements in FIFO

TYPE Fifo = RECORD info: ARRAY [0 .. Max - 1] OF ET; in, out, n: CARDINAL := 0;

END; (*Fifo*) VAR w: Fifo; PROCEDURE Enqueue(elem:ET) = (*adds element to end*) BEGIN w.info[w.in]:= elem; w.in:= (w.in + 1) MOD Max; INC(w.n); elements*) END Enqueue; (*contains a FIFO queue*)

(*stores new element*) (*increments in-pointer in ring*) (*increments number of stored

PROCEDURE Dequeue(): ET = (*removes and returns first element*) VAR e: ET; BEGIN e:= w.info[w.out]; (*removes oldest element*) w.out:= (w.out + 1) MOD Max; (*increments out-pointer in ring*) DEC(w.n); (*decrements number of stored elements*) RETURN e; (*returns the read element*) END Dequeue;

Utility Routines
PROCEDURE Empty(): BOOLEAN = (*returns true if queue is empty*) BEGIN RETURN w.n = 0; END Empty; PROCEDURE Full(): BOOLEAN = (*returns true if queue is full*) BEGIN RETURN w.n = Max END Full; BEGIN END Fifo.

User Module
MODULE FifoUser EXPORTS Main; (*14.07.94. LB*) (* Example client of the text queue. *) FROM Fifo IMPORT Enqueue, Dequeue, Empty, Full; (* operations of the queue *) FROM SIO IMPORT Error, GetText, PutText, Nl; <* FATAL Error *> (*supress warning*) BEGIN PutText("FIFO User. Please enter texts:\n"); WHILE NOT Full() DO

Enqueue(GetText()) END; WHILE NOT Empty() DO PutText(Dequeue() & " END; Nl(); END FifoUser.

")

Other Queues Double-ended queues - These are data structures which support both push and pop and enqueue/dequeue operations. Priority Queues(heaps) - Supports insertions and ``remove minimum'' operations which useful in simulations to maintain a queue of time events. We will discuss simulations in a future class.

Pointers and Dynamic Memory Allocation Lecture 4


Pointers and Dynamic Memory Allocation Although arrays are good things, we cannot adjust the size of them in the middle of the program. If our array is too small - our program will fail for large data. If our array is too big - we waste a lot of space, again restricting what we can do. The right solution is to build the data structure from small pieces, and add a new piece whenever we need to make it larger. Pointers are the connections which hold these pieces together! Pointers in Real Life In many ways, telephone numbers serve as pointers in today's society.

To contact someone, you do not have to carry them with you at all times. All you need is their number. Many different people can all have your number simultaneously. All you need do is copy the pointer. More complicated structures can be built by combining pointers. For example, phone trees or directory information.

Addresses are a more physically correct analogy for pointers, since they really are memory addresses. Linked Data Structures All the dynamic data structures we will build have certain shared properties.

We need a pointer to the entire object so we can find it. Note that this is a pointer, not a cell. Each cell contains one or more data fields, which is what we want to store. Each cell contains a pointer field to at least one ``next'' cell. Thus much of the space used in linked data structures is not data! We must be able to detect the end of the data structure. This is why we need the NIL pointer.

Pointers in Modula-3 A node in a linked list can be declared:


type pointer = REF node; node = record info : item; next : pointer; end; p,q,r : pointer; x,y,z : node; (* pointers *) (* records *)

var

Note circular definition. Modula-3 lets you get away with this because it is a reference type. Pointers are the same sizeregardless of what they point to! We want dynamic data structures, where we make nodes as we need them. Thus declaring nodes as variables are not the way to go! Dynamic Allocation

To get dynamic allocation, use new:


p := New(ptype);

New(ptype) allocates enough space to store exactly one object of the type ptype. Further, it returns a pointer to this empty cell. Before a new or otherwise explicit initialization, a pointer variable has an arbitrary value which points to trouble! Warning - initialize all pointers before use. Since you cannot initialize them to explicit constants, your only choices are

NIL - meaning explicitly nothing. New(ptype) - a fresh chunk of memory. assignment to some previously initialized pointer of the same type.

Pointer Examples Example: p := new(node); q := new(node); p.x grants access to the field x of the record pointed to by p.
p^.info := "music"; q^.next := nil;

The pointer value itself may be copied, which does not change any of the other fields. Note this difference between assigning pointers and what they point to.
p := q;

We get a real mess. We have completely lost access to music and can't get it back! Pointers are unidirectional. Alternatively, we could copy the object being pointed to instead of the pointer itself.
p^ := q^;

What happens in each case if we now did:


p^.info := "data structures";

Where Does the Space Come From? Can we really get as much memory as we want without limit just by using New? No, because there are the physical limits imposed by the size of the memory of the computer we are using. Usually Modula-3 systems let the dynamic memory come from the ``other side'' of the ``activation record stack'' used to maintain procedure calls: Just as the stack reuses memory when a procedure exits, dynamic storage must be recycled when we don't need it anymore. Garbage Collection The Modula-3 system is constantly keeping watch on the dynamic memory which it has allocated, making sure that somethingis still pointing to it. If not, there is no way for you to get access to it, so the space might as well be recycled. The garbage collector automatically frees up the memory which has nothing pointing to it. It frees you from having to worry about explicitly freeing memory, at the cost of leaving certain structures which it can't figure out are really garbage, such as a circular list. Explicit Deallocation Although certain languages like Modula-3 and Java support garbage collection, others like C++ require you to explicitly deallocate memory when you don't need it. Dispose(p) is the opposite of New - it takes the object which is pointed to by p and makes it available for reuse. Note that each dispose takes care of only one cell in a list. To dispose of an entire linked structure we must do it one cell as a time. Note we can get into trouble with dispose:

Of course, it is too late to dispose of music, so it will endure forever without garbage collection. Suppose we dispose(p), and later allocation more dynamic memory with new. The cell we disposed of might be reused. Now what does q point to? Answer - the same location, but it means something else! So called dangling references are a horrible error, and are the main reason why Modula-3 supports garbage collection. A dangling reference is like a friend left with your old phone number after you move. Reach out and touch someone - eliminate dangling references! Security in Java It is possible to explicitly dispose of memory in Modula-3 when it is really necessary, but it is strongly discouraged. Java does not allow one to do such operations on pointers at all. The reason is security. Pointers allow you access to raw memory locations. In the hands of skilled but evil people, unchecked access to pointers permits you to modify the operating system's or other people's memory contents. Java is a language whose programs are supposed to be transferred across the Internet to run on your computer. Would you allow a stranger's program to run on your machine if they could ruin your files?

Linked Stacks and Queues Lecture 5


Pointers about Pointers
var p, q : ^node;

p = new(node) creates a new node and sets p to point to it. p describes the node which is pointed to by p.

p .item describes the item field of the node pointed to by p. dispose(p) returns to the system the memory used by the node pointed to by p. This is not used because of Modula-3 garbage collection. NIL is the only value a pointer can have which is not an address. Linked Stacks The problem with array-based stacks are that the size must be determined at compile time. Instead, let's use a linked list, with the stack pointer pointing to the top element. To push a new element on the stack, we must do:
p^.next = top; top = p;

Note this works even for the first push if top is initialized to NIL! Popping from a Linked Stack To pop an item from a linked stack, we just have to reverse the operation.
p = top; top = top^.next; p^.next = NIL;

(*avoid dangling reference*)

Note again that this works in the boundary case of one item on the stack. Note that to check we don't pop from an empty stack, we must test whether top = NIL before using top as a pointer. Otherwise things crash or segmentation fault. Linked Stack in Modula-3
MODULE Stacks; (*14.07.94 RM, LB*) (* Implementation of the abstract, generic stack. *) REVEAL T = BRANDED REF RECORD info: ET; next: T; END; (*T*) PROCEDURE Create(): T = (*creates and intializes a new stack*) BEGIN RETURN NIL; (* a new, empty stack is simply NIL *) END Create;

PROCEDURE Push(VAR stack: T; elem:ET) = (*adds element to stack*) VAR new: T := NEW(T, info:= elem, next:= stack); (*create element*) BEGIN stack:= new (*add element at top*) END Push; PROCEDURE Pop(VAR stack: T): ET = (*removes and returns top element, or NIL for empty stack*) VAR first: ET := NIL; (* Pop returns NIL for empty stack*) BEGIN IF stack # NIL THEN first:= stack.info; (*copy info from first element*) stack:= stack.next; (*remove first element*) END; (*IF stack # NIL*) RETURN first; END Pop; PROCEDURE Empty(stack: T): BOOLEAN = (*returns TRUE for empty stack*) BEGIN RETURN stack = NIL END Empty; BEGIN END Stacks.

Generic Stack Interface


INTERFACE Stacks; (*14.07.94 RM, LB*) (* Abstract generic stack. *) TYPE T <: REFANY; ET = REFANY; (*type of stack*) (*type of elements*) (*creates and intializes a

PROCEDURE Create(): T; new stack*)

PROCEDURE Push(VAR stack: T; elem: ET); (*adds element to stack*) PROCEDURE Pop(VAR stack: T): ET; (*removes and returns top element, or NIL for empty stack*) PROCEDURE Empty(stack: T): BOOLEAN; (*returns TRUE for empty stack*) END Stacks.

Generic Stacks Client


MODULE StacksClient EXPORTS Main; (* LB *) (* Example client of both the generic stack and the type FractionType. This program builds up a stack of fraction numbers as well as of complex numbers.

*) IMPORT Stacks; IMPORT FractionType; FROM Stacks IMPORT Push, Pop, Empty; FROM SIO IMPORT PutInt, PutText, Nl, PutReal, PutChar; TYPE Complex = REF RECORD r, i: REAL END; VAR stackFraction: Stacks.T:= Stacks.Create(); stackComplex : Stacks.T:= Stacks.Create(); c: Complex; f: FractionType.T; BEGIN (*StacksClient*) PutText("Stacks Client\n"); FOR i:= 1 TO 4 DO Push(stackFraction, FractionType.Create(1, i)); 1/i*) END; (*stores numbers

FOR i:= 1 TO 4 DO Push(stackComplex, NEW(Complex, r:= FLOAT(i), i:= 1.5 * FLOAT(i))); END; WHILE NOT Empty(stackFraction) DO f:= Pop(stackFraction); PutInt(FractionType.Numerator(f)); PutText("/"); PutInt(FractionType.Denominator(f), 1); END; Nl(); WHILE NOT Empty(stackComplex) DO c:= Pop(stackComplex); PutReal(c.r); PutChar(':'); PutReal(c.i); PutText(" "); END; Nl(); END StacksClient.

Linked Queues Queues in arrays were ugly because we need wrap around for circular queues. Linked lists make it easier.

We need two pointers to represent our queue - one to the rear for enqueue operations, and one to the front for dequeueoperations. Note that because both operations move forward through the list, no back pointers are necessary! Enqueue and Dequeue To enqueue an item :

p^.next := NIL; if (back = NIL) then begin (* empty queue *) front := p; back := p; end else begin (* non-empty queue *) back^.next := p; back := p; end;

To dequeue an item:
p := front; front := front^.next; p^.next := NIL; if (front = NIL) then back := NIL;

(* now-empty queue *)

Building the Calculator Lecture 6


Reverse Polish Notation HP Calculators use reverse Polish notation or postfix notation. Instead of the conventional a + b, we write A B +. Our calculator will do the same. Why? Because it is the easiest notation to implement!

The rule for conversion is to read the expression from left to right. When we see a number, push it on the operation stack. When we see an operation, pop the last two numbers on stack, do the operation, and push the result on the stack.

Look Ma, no parentheses! Algorithms for the calculator To implement addition, we add digits from right to left, with the carry one place if the sum is greater than 10. Note that the last carry can go beyond one or both numbers, so you must handle this special case. To implement subtraction, we work on digits from right to left, and borrow 10 if necessary from the digit to the left. A borrow from the leftmost digit is complicated, since that gives a negative number. This is why I suggest completing addition first before worrying about subtraction. I recommend to test which number has a larger absolute value, subtract from that, and then adjust the sign accordingly.

Parsing the Input There are several possible ways to handle the problem of reading in the input line and parsing it, i.e. breaking it into its elementary components of numbers and operators. The way that seems best to me is to read the entire line as one character string in a variable of type TEXT. As detailed in your book, you can use the function Text.Length(S) to get the length of this string, and the functionText.GetChar(S,i) to retreive any given character. Useful functions on characters include the function ORD(c), which returns the integer character code of c. Thus ORD(c) -ORD('0') returns the numerical value of a digit character. You can test characters for equality to identify special symbols.

Standard I/O The easiest way to read and write from the files is to use I/O redirection from UNIX. Suppose calc is your binary program, and it expects input from the keyboard and output to the screen. By running calc < fileinat the command prompt, it will take its input from the file filein instead of the keyboard. Thus by writing your program to read from regular I/O, you can debug it interactively and also run my test files. Programming Hints 1. Write the comments first, for your sake. 2. Make sure your main routine is abstract enough that you can easily see what the program does. 3. Isolate the details of your data structures to a few abstract operations. 4. Build good debug print routines first.

List Insertion and Deletion Lecture 7


Search, Insert, Delete There are three fundamental operations we need for any database:

Insert: add a new record at a given point Delete: remove an old record Search: find a record with a given key

We will see a wide variety of different implementation of these operations over the course of the semester. How would you implement these using an array? With linked lists, we can creating arbitrarily large structures, and never have to move any items.

Most of these operations should be pretty simple now that you understand pointers! Searching in a Linked List
Procedure Search(head:pointer, key:item):pointer; Var p:pointer; found:boolean; Begin found:=false; p:=head; While (p # NIL) AND (not found) Do Begin If (p^.info = key) then found = true; Else p = p^.next; End; return p; END;

Search performs better when the item is near the front of the list than the back. What happens when the item isn't found? Insertion into a Linked List The easiest way to insert a new node p into a linked list is to insert it at the front of the list:
p^.next = front; front = p;

To maintain lists in sorted order, however, we will want to insert a node between the two appropriate nodes. This means that as we traverse the list we must keep pointers to both the current node and the previous node.
MODULE Intlist; (*16.07.94. RM, LB*) (* Implementation of sorted integer lists. *) REVEAL T = BRANDED REF RECORD key: INTEGER; next: T := NIL; END; (*T*) PROCEDURE Create(): T = (* returns a new, empty list *) BEGIN (*reveal inner structure of T*) (*key value*) (*pointer to next element*)

RETURN NIL; END Create;

(*creation is trivial; empty list is NIL*)

PROCEDURE Insert(VAR list: T; value:INTEGER) = (* inserts new element in list and maintains order *) VAR current, previous: T; new: T := NEW(T, key:= value); (*create new element*) BEGIN IF list = NIL THEN list:= new (*first element*) ELSIF value < list.key THEN (*insert at beginning*) new.next:= list; list:= new; ELSE (*find position for insertion*) current:= list; previous:= current; WHILE (current # NIL) AND (current.key <= value) DO previous:= current; (*previous hobbles after*) current:= current.next; END; (*after the loop previous points to the insertion point*) new.next:= current; (*current = NIL if insertion point is the end*) previous.next:= new; (*insert new element*) END; (*IF list = NIL*) END Insert;

Make sure you understand where these cases come from and can verify why all of them work correct. Deletion of a Node To delete a node from a singly linked-list, we must have pointers to both the node-to-die and the previous node, so we can reconnect the rest of the list.
PROCEDURE Remove(VAR list: T; value:INTEGER; VAR found: BOOLEAN) = (* deletes (first) element with value from sorted list, or returns false in found if the element was not found *) VAR current, previous: T; BEGIN IF list = NIL THEN found:= FALSE ELSE (*start search*) current:= list; previous:= current; WHILE (current # NIL) AND (current.key # value) DO previous:= current; (*previous hobbles after*) current:= current.next; END; (*holds: current = NIL or current.key = value, but not both*) IF current = NIL THEN found:= FALSE (*value not found*) ELSE found:= TRUE; (*value found*) IF current = list THEN list:= current.next (*element found at beginning*)

ELSE previous.next:= current.next END; END; (*IF current = NIL*) END; (*IF list = NIL*) END Remove;

Passing Procedures as Arguments Note the passing of a procedure as a parameter - it is legal, and useful to make more general functions, for example a sort routine for both increasing and decreasing order, or any order.
PROCEDURE Iterate(list: T; action: Action) = (* applies action to all elements (with key value as parameter) *) BEGIN WHILE list # NIL DO action(list.key); list:= list.next; END; END Iterate; BEGIN (* Intlist *) END Intlist.

Pointers and Parameter Passing Pointers provide, for better or (usually) worse, and alternate way to modify parameters. Let us look at two different ways to swap the ``values'' of two pointers.
Procedure Swap1(var p,q:pointer); Var r:pointer; begin r:=q; q:=p; p:=r; end;

This is perhaps the simplest and best way - we just exchange the values of the pointers... Alternatively, we could swap the values of what is pointed to, and leave the pointers unchanged.
Procedure Swap2(p,q : pointer); var tmp : node; begin tmp := q^; (*1*)

q^ := p^; p^ := tmp; end;

(*2*) (*3*)

After step (*1*): After step (*2*): After step (*3*): Side Effects of Pointers If swap2, since we do not change the values of p and q, they do not need to be var parameters! However, copying the values did not do the same thing as copying the pointers, because in the first case the physical locationof the data changed, while in the second the data stayed put. If data which is pointed to moves, the value of what is pointed to can change! Moral: you must be careful about the side effects of pointer operations!!! C language does not have var parameters. All side effects are done by passing pointers. Additional pointer operations in C language help make this practical.

Doing the Shuffle Lecture 8


Programming Style Although programming style (like writing style) is a somewhat subjective thing, there is a big difference between good and bad. The good programmer doesn't just strive for something that works, but something that works elegantly and efficiently; something that can be maintained and understood by others.

Just like a good writer rereads and rewrites their prose, a good programmer rewrites their program to make it cleaner and better. To get a better sense of programming style, let's critique some representative solutions to the card-shuffling assignment to see what's good and what can be done better. Ugly looking main
MODULE card EXPORTS Main; IMPORT SIO; TYPE index=[1..200]; Start=ARRAY index OF INTEGER; Left=ARRAY index OF INTEGER; Right=ARRAY index OF INTEGER; Final=ARRAY index OF INTEGER; VAR i,j,times,mid,k,x: INTEGER; start: Start; left: Left; right: Right; final: Final; BEGIN SIO.PutText("deck size shuffles\n"); SIO.PutText("----------------\n"); SIO.PutText(" 200 "); SIO.PutInt( times ); REPEAT i:=1; WHILE i<=200 DO start[i]:=i; i:=i+1; END; (*Repeat the following until perfect shuffle*) (*original deck*)

j:=1; (*splits into two decks*) mid:=100; WHILE (j<=100) DO left[j]:=start[j]; right[j]:=start[j+mid]; j:=j+1; END; x:=1; k:=1; (*shuffle them into one deck*) WHILE k<=200 DO final[k]:=left[x]; final[k+1]:=right[x]; k:=k+2; END; UNTIL start[2]=final[2]; (*check if complete shuffle*)

times:=times+1; END card.

There are no variable or block comments. This program would be hard to understand. This is an ugly looking program - the structure of the program is not reflected by the white space. Indentation and blank lines appear to be added randomly. There are no subroutines used, so everything is one big mess. See how the dependence on the number of cards is used several times within the body of the program, instead of just in one CONST. What Does shufs Do?
PROCEDURE shufs( nn : INTEGER )= VAR i : INTEGER; count : INTEGER; BEGIN FOR i := 1 TO 200 DO shuffled[i] := i; END; count := 0; REPEAT count := count + 1; FOR i := 1 TO 200 DO tempshuf *) END; FOR i := 1 TO nn DO (* shuffle 1st half *) shuffled[2*i-1] := tempshuf[i]; END; FOR i := nn+1 TO 2*nn DO (* shuffle 2nd half *) shuffled[2*(i-nn)] := tempshuf[i]; END; UNTIL shuffled = unshuffled ; original? *) (* did it return to (* copy shuffled -> (* reset this array *) (* shuffling procedure *) (* index variable *) (* COUNT variable *)

(* start counter from 0 *)

tempshuf[i] := shuffled[i];

(* print out the data *) Wr.PutText(Stdio.stdout , "2*n= " & Fmt.Int(2*nn) & " \t" );

Wr.PutText(Stdio.stdout , Fmt.Int(count) & "\n" ); END shufs;

Every subroutine should ``do'' something that is easily described. What does shufs do? The solution to such problems is to write the block comments for the subroutine does before writing the subroutine. If you can't easily explain what it does, you don't understand it. How many comments are enough?
MODULE Shuffles EXPORTS Main; IMPORT SIO; TYPE Array= ARRAY [1..200] OF INTEGER; VAR original, temp1, temp2: Array; counter: INTEGER; (*Create an integer array from *) (*1 to 200 and called Array *) (*Declare original,temp1 and *) (*temp2 to be Array *) (*Declare counter to be integer*)

(********************************************************************) (* This is a procedure called shuffle used to return a number of *) (* perfect shuffle. It input a number from the main and run the *) (* program with it and then return the final number of perfect shuffle *) (********************************************************************) PROCEDURE shuffle(total: INTEGER) :INTEGER = VAR half, j, p: INTEGER; (*Declare half, j, p to be integer *) BEGIN FOR j:= 1 TO total DO original[j] := j; temp1[j] := j; END; (*for*) half := total DIV 2; REPEAT j := 0; p := 1; REPEAT j := j + 1; temp2[p] := temp1[j]; (* Save the number from the first half *) (* of the original array into temp2 *) p := p + 1;

half*) *)

temp2[p] := temp1[half+j]; (* Save the number from the last (* of the original array into temp2

p := p + 1; UNTIL p = total + 1; (*REPEAT_UNTIL used to make a new array of temp1*) INC (counter); (* increament counter when they shuffle once *) FOR i := 1 TO total DO temp1[i] := temp2[i]; END; (* FOR loop used to save all the elements from temp2 to temp1 *) UNTIL temp1 = original; (* REPEAT_UNTIL, when two array match exactly *) (* same then quick *) RETURN counter; (* return the counter *) END shuffle; (* end procedure shuffle *) (********************************************************************) (* This is the Main for shuffle program that prints out the numbers *) (* of perfect shuffles necessary for a deck of 2n cards *) (********************************************************************) BEGIN ... END Shuffles.

(* end the main program called Shuffles *)

This program has many comments which should be obvious to anyone who can read Modula-3. More useful would be enhanced block comments telling you what the program is done and how it works. The ``is it completely reshuffled yet?'' test is done cleanly, although all of the 200 cards are tested regardless of deck size. The shuffle algorithm is too complicated. Algorithms must be pretty, too
MODULE prj1 EXPORTS Main; IMPORT SIO; CONST n : INTEGER = 100;

(*size of split deck*) (*n sized deck type*) (*2n sized deck type*) (*merged deck*)

TYPE nArray = ARRAY[1..n] OF INTEGER; twonArray = ARRAY[1..2*n] OF INTEGER; VAR merged : twonArray; count : INTEGER;

PROCEDURE shuffle(size:INTEGER; VAR merged:twonArray)= VAR topdeck, botdeck : nArray; (*arrayed split decks*) BEGIN FOR i := 1 TO size DO topdeck[i] := merged[i]; (*split entire deck*) botdeck[i] := merged[i+size]; (*into top, bottom decks*) END; FOR j := 1 TO size DO merged[2*j-1] := topdeck[j]; (*If odd then 2*n-1 position.*) merged[2*j] := botdeck[j]; (*If even then 2*n position*) END; END shuffle; PROCEDURE printout(count:INTEGER; size:INTEGER)= BEGIN SIO.PutInt(size); SIO.PutText(" "); SIO.PutInt(count); SIO.PutText(" \n"); END printout; PROCEDURE checkperfect(merged:twonArray; i:INTEGER) : BOOLEAN= VAR size : INTEGER; check : BOOLEAN; BEGIN check := FALSE; size := 0; REPEAT INC(size, 1); (*check to see if*) IF merged[size+1] - merged[size] = 1 THEN (*deck is perfectly*) check := TRUE; (*shuffled, if so *) END; (*card progresses by 1*) UNTIL (check = FALSE OR size - 1 = i); RETURN check; END checkperfect;

Checkperfect is much more complicated than it need be; just check whether merged[i] = i. You can return without the BOOLEAN variable. A good thing is that the deck size is all a function of a CONST. The shuffle is slightly wasteful of space - two extra full arrays instead of two extra half arrays. Why does this work correctly?
BEGIN

SIO.PutLine("Welcome to Paul's card shuffling program!"); SIO.PutLine(" DECK SIZE NUMBER OF SHUFFLES SIO.PutLine(" _________________________________ num_cards := 2; REPEAT counter := 0; FOR i := 1 TO (num_cards) DO deck[i] :=i; END; (*initializes deck*) REPEAT deck := Shuffle(deck,num_cards); INC(counter); UNTIL deck[2] = 2; SIO.PutInt(num_cards,16); SIO.PutInt(counter,19); SIO.PutText("\n"); INC(num_cards,2); (*increments the number of cards in deck by 2.*) UNTIL ( num_cards = ((2*n)+2)); END ShuffleCards.

"); ");

Why we know that this stopping condition suffices to get us all the cards in the right position. This should be proven prior to use. Why use a Repeat loop when For will do? Program Defensively I am starting to see the wreckage of several programs because students are not building their programs to be debugged.

Add useful debug print statements! Have your program describe what it is doing! Document what you think your program does! Otherwise, how do you know whine it is doing it! Build your program in stages! Thus you localize your bugs, and make sure you understand simple things before going on to complicated things. Use spacing to show the structure of your program. A good program is a pretty program!

Recursive and Doubly Linked Lists Lecture 9


Recursive List Implementation

The basic insertion and deletion routines for linked lists are more elegantly written using recursion.
PROCEDURE Insert(VAR list: T; value:INTEGER) = (* inserts new element in list and maintains order *) VAR new: T; (*new node*) BEGIN IF list = NIL THEN list:= NEW(T, key := value) (*list is empty*) ELSIF value < list.key THEN (*proper place found: insert*) new := NEW(T, key := value); new.next := list; list := new; ELSE (*seek position for insertion*) Insert(list.next, value); END; (*IF list = NIL*) END Insert; PROCEDURE Remove(VAR list:T; value:INTEGER; VAR found:BOOLEAN) = (* deletes (first) element with value from sorted list, or returns false in found if the element was not found *) BEGIN IF list = NIL THEN (*empty list*) found := FALSE ELSIF value = list.key THEN (*elemnt found*) found := TRUE; list := list.next ELSE (*seek for the element to delete*) Remove(list.next, value, found); END; END Remove;

Doubly Linked Lists Often it is necessary to move both forward and backwards along a linked list. Thus we need another pointer from each node, to make it doubly linked. List types are analogous to dance structures:

Conga line - singly linked list. Chorus line - doubly linked list. Hora circle - double linked circular list.

Extra pointers allow the flexibility to have both forward and backwards linked lists:
type pointer = REF node; node = record info : item; front : pointer;

end;

back : pointer;

Insertion How do we insert p between nodes q and r in a doubly linked list?


p^.front = r; p^.back = q; r^.back = p; q^.front = p;

It is not absolutely necessary to have pointer r, since r = q .front, but it makes it cleaner. The boundary conditions are inserting before the first and after the last element. How do we insert before the first element in a doubly linked list (head)?
p^.back = NIL; p^.front = head; head^.back = p; head = p; (* must point to entire structure *)

Inserting at the end is similar, except head doesn't change, and a back pointer is set to NIL. Linked Lists: Pro or Con? The advantages of linked lists include:

Overflow can never occur unless the memory is actually full. Insertions and deletions are easier than for contiguous (array) lists. With large records, moving pointers is easier and faster than moving the items themselves.

The disadvantages of linked lists include:


The pointers require extra space. Linked lists do not allow random access. Time must be spent traversing and changing the pointers. Programming is typically trickier with pointers.

Recursion and Backtracking Lecture 10


Recursion Recursion is a wonderful, powerful way to solve problems. Elegant recursive procedures seem to work by magic, but the magic is same reason mathematical induction works! Example: Prove For n=1, . , so its true. Assume it is true up to n-1.

Example: All horses are the same color! (be careful of your basis cases!) The Tower of Hanoi
MODULE Hanoi EXPORTS Main; (*18.07.94*) (* Implementation of the game Towers of Hanoi. *) PROCEDURE Transfer(from, to: Post) = (*moves a disk from post "from" to post "to"*) BEGIN WITH f = posts[from], t = posts[to] DO INC(t.top); t.disks[t.top]:= f.disks[f.top]; f.disks[f.top]:= 0; DEC(f.top); END; (*WITH f, t*) END Transfer; PROCEDURE Tower(height:[0..Height] ; from, to, between: Post) = (*Does the job through recursive calls on itself*) BEGIN IF height > 0 THEN

Tower(height - 1, from, between, to); Transfer(from, to); Display(); Tower(height - 1, between, to, from); END; END Tower; BEGIN (*main program Hanoi*) posts[Post.Start].top:= Height; FOR h:= 1 TO Height DO posts[Post.Start].disks[h]:= Height - (h - 1) END; Tower(Height, Post.Start, Post.Finish, Post.Temp); END Hanoi.

To count the number of moves made,

Recursion not only made a complicated problem understandable, it made it easy to understand. Combinatorial Objects Many mathematical objects have simple recursive definitions which can be exploited algorithmically. Example: How can we build all subsets of n items? Build all subsets of n-1 items, copy the subsets, and add item n to each of the subsets in one copy but not the other. Once you start thinking recursively, many things have simpler formulations, such as traversing a linked list or binary search. Gray codes We saw how to generate subsets recursively. Now let us generate them in an interesting order. All subsets of can be represented as binary strings of length n, where bit i tells whether i is in the subset or not. Obviously, all subsets must differ in at least one element, or else they would be identical. An order where they differ by exactly one from each other is called a Gray code.

For n=1, {},{1}. For n=2, {},{1},{1,2},{2}. For n=3, {},{1},{1,2},{2},{2,3},{1,2,3},{1,3},{3} Recursive construction algorithm: Build a Gray Code of , make a reverse copy of it, append n to each subset in the reverse copy, and stick the two together! Formulating Recursive Programs Think about the base cases, the small cases where the problem is simple enough to solve. Think about the general case, which you can solve if you can solve the smaller cases. Unfortunately, many of the simple examples of recursion are equally well done by iteration, making students suspicious. Further, many of these classic problems have hidden costs which make recursion seem expensive, but don't be fooled! Factorials
PROCEDURE Factorial (n: CARDINAL): CARDINAL = BEGIN IF n = 0 THEN RETURN 1 (* trivial case *) ELSE RETURN n * Factorial(n-1) (* recursive branch *) END (* IF*) END Factorial;

Be sure you understand how the parameter passing mechanism works. Would this program work if n was a VAR parameter? Fibonacci Numbers The Fibonacci numbers are given by the recurrence relation .

PROCEDURE Fibonacci(n : CARDINAL) : CARDINAL = BEGIN (* Fibonacci *) IF n <= 1 THEN RETURN 1 ELSE RETURN Fibonacci(n-1) + Fibonacci(n-2) END (* IF *) END Fibonacci;

(*n > 1*)

How much time does this elementary Fibonacci function take? Implementing Recursion Part of the mystery of recursion is the question of how the machine keeps everything straight. How come local variables don't get trashed? The answer is that whenever a procedure or function is called, the local variables are pushed on a stack, so the new recursive call is free to use them. When a procedure ends, the variables are popped off the stack to restore them to where they were before the call. Thus the space used is equal to the depth of the recursion, since stack space is reused. Tail Recursion Tail recursion costs space, but not time. It can be removed mechanically and is by some compilers. Moral: Do not be afraid to use recursion if the algorithm is efficient. The overhead of recursion vs. maintaining your own stack is too small to worry about. By being clever, you can sometimes save stack space. Consider the following variation of Quicksort:

If (p-1 < h-p) then

Qsort(1,p)

Qsort(p,h)

else

Qsort(p,h)

Qsort(1,p)

By doing the smaller half first, the maximum stack depth is worst case. Applications of Recursion

in the

You may say, ``I just want to get a job and make lots of money. What can recursion do for me? We will look at three applications

Backtracking Game Tree Search Recursion Descent Compilation

The N-Queens Problem Backtracking is a way to solve hard search problems. For example, how can we put n queens on an queens attack each other? Tree Pruning board so that no two

Backtracking really pays off when we can prove a node early in the search tree. Thus we need never look at its children, or grandchildren, or great.... We apply backtracking to big problems, so the more clever we are, the more time we save.

There are total sets of eight squares but no two queens can be in the same row. There are ways to place eight queens in different rows. However, since no two queens can be in the same column, there are only 8! permutations of columns, or only 40,320 possibilities. We must also be clever to test as quickly as possible the new queen does not violate a diagonal constraint

Applications of Recursion Lecture 11


Game Trees Chess playing programs work by constructing a tree of all possible moves from a given position, so as to select the best possible path. The player alternates at each level of the tree, but at each node the player whose move it is picks the path that is best for them. A player has a forced loss if lead down a path where the other guy wins if they play correctly. This is a recursive problem since we can always maximize, by just changing perspective. In a game like chess, we will never reach the bottom of the tree, so we must stop at a particular depth. Alpha-beta Pruning Sometimes we don't have to look at the entire game tree to get the right answer:

No matter what the red score is, it cannot help max and thus need not be looked at. An advanced strategy called alpha-beta running reduces search accordingly. Recursive Descent Compilation Compilers do two useful things

They identify whether a program is legal in the language. They translate it into assembly language.

To do either, we need a precise description of the language, a BNF grammar which gives the syntax. A grammar for Modula-3 is given throughout your text. The language definition can be recursive!! Our compiler will follow the grammar to break the program into smaller and smaller pieces. When the pieces get small enough, we can spit out the appropriate chunk of assembly code. To avoid getting into infinite loops, we place our trust in the fellow who wrote the grammar. Proper design can ensure that there are no such troubles.

Abstraction and Modules Lecture 12


Abstract Data Types It is important to structure programs according to abstract data types: collections of data with well-defined operations on it Example: Stack or Queue. Data: A sequence of items Operations: Initialize, Empty?, Full?, Push, Pop, Enqueue, Dequeue

Example: Infinite Precision Integers. Data: Linked list of digits with sign bit. Operations: Print number, Read Number, Add, Subtract, Multiply, Divide, Exponent, Module, Compare. Abstract data types add clarity by separating the definitions from the implementations. What Do We Want From Modules? Separate Compilation - We should be able to break the program into smaller files. Further, we shouldn't need the source for each Module to link it together, just the compiled object code files. Communicate Desired Information Between Modules - We should be able to define a type or procedure in one module and use it in another. Information Hiding - We should be able to define a type or procedure in one module and forbid using it in another! Thus we can clearly separate the definition of an abstract data type from its implementation! Modula-3 supports all of these goals by separating interfaces (.i3 files) from implementations (.m3 files). Example: The Piggy Bank Below is an interface file to:
INTERFACE PiggyBank; (*RM*) (* Interface to a piggy bank: You can insert money with "Deposit". The only other permissible operation is smashing the piggy bank to get the ``money back'' The procedure "Smash" returns the sum of all deposited amounts and makes the piggy bank unusable.

*)

PROCEDURE Deposit(cash: CARDINAL); PROCEDURE Smash(): CARDINAL; END PiggyBank.

Note that this interface does not reveal where or how the total value is stored, nor how to initialize it. These are issues to be dealt with within the implementation of the module.

Piggy Bank Implementation


MODULE PiggyBank; (*RM/CW*) (* Implementation of the PiggyBank interface *) VAR contents: INTEGER; PROCEDURE Deposit(cash: CARDINAL) = (* changes the state of the piggy bank *) BEGIN <*ASSERT contents >= 0*> contents := contents + cash END Deposit; (* state of the piggy bank *)

(* piggy bank still okay? *)

PROCEDURE Smash(): CARDINAL = VAR oldContents: CARDINAL := contents; (* contents before smashing *) BEGIN contents := -1; (* smash piggy bank *) RETURN oldContents END Smash; BEGIN contents := 0 END PiggyBank. (* initialization of state variables in body *)

A Client Program for the Bank


MODULE Saving EXPORTS Main; (*RM*) (* Client of the piggy bank: In a loop the user is prompted for the amount of deposit. Entering a negative amount smashes the piggy bank.

*)

FROM PiggyBank IMPORT Deposit, Smash; FROM SIO IMPORT GetInt, PutInt, PutText, Nl, Error; <*FATAL Error*> VAR cash: INTEGER; BEGIN (* Saving *) PutText("Amount of deposit (negative smashes the piggy bank): \n"); REPEAT cash := GetInt(); IF cash >= 0 THEN Deposit(cash) ELSE PutText("The smashed piggy bank contained $"); PutInt(Smash()); Nl() END; UNTIL cash < 0 END Saving.

Interface File Conventions Imports describe what procedures a given module makes available. Exports describes what we are willing to make public, ultimately including the ``MAIN'' program. By naming files with the same .m3 and .i3 names, the ``ezbuild'' make command can start from the file with the main program, and final all other relevant files. Ideally, the interface file should hide as much detail about the internal implementation of a module from its users as possible. This is not easy without sophisticated language features. Hiding the Details
INTERFACE Fraction; (*RM*) (* defines the data type for rational numbers *) TYPE T = RECORD num : INTEGER; den : INTEGER; END; PROCEDURE Init (VAR fraction: T; num: INTEGER; den: INTEGER := 1); (* Initialize "fraction" to be "num/den" *) PROCEDURE PROCEDURE PROCEDURE PROCEDURE *) Plus Minus Times Divide (x, (x, (x, (x, y y y y : : : : T) T) T) T) : : : : T; T; T; T; (* (* (* (* x x x x + * / y y y y *) *) *) *)

PROCEDURE Numerator

(x : T): INTEGER; (* returns the numerator of x

PROCEDURE Denominator (x : T): INTEGER; (* returns the denominator of x *) END Fraction.

Note that there is a dilemma here. We must make type T public so these procedures can use it, but would like to prevent users from accessing (or even knowing about) the fields num and dem directly. Subtypes and REFANY

Modula-3 permits one to declare subtypes of types, A <: B, which means that anything of type A is of type B, but everything of type type B is not necessarily of type A. This proves important in implementing advanced object-oriented features like inheritance. REFANY is a pointer type which is a supertype of any other pointer. Thus a variable of type REFANY can store a copy of any other pointer. This enables us to define public interface files without actually revealing the guts of the fraction type implementation. Fraction type with REFANY
INTERFACE FractionType; (*19.12.94. RM, LB*) (* defines the data type of rational numbers, compare with Example 10.10! *) TYPE T <: REFANY; hidden*) PROCEDURE PROCEDURE PROCEDURE PROCEDURE PROCEDURE PROCEDURE PROCEDURE (*T is a subtype of Refany; its structure is

Create (numerator: INTEGER; denominator: INTEGER := 1): T; Plus (x, y : T) : T; (* x + y *) Minus (x, y : T) : T; (* x - y *) Mult (x, y : T) : T; (* x * y *) Divide (x, y : T) : T; (* x : y *) Numerator (x : T): INTEGER; Denominator (x : T): INTEGER;

END FractionType.

Somewhere within a module we must reveal the implementation of type T. This is done with a REVEAL statement:
MODULE FractionType; (*19.12.94. RM, LB*) (* Implementation of the type FractionType. Compare with Example 10.12. In this version the structure of elements of the type is hidded in the interface. The structure is revealed here. *) REVEAL T = BRANDED REF RECORD (*opaque structure of T*) num, den: INTEGER END; (*T*)

...

The Key Idea about REFANY

With generic pointers, it becomes necessary for type checking to be done a run-time, instead of at compile-time as done to date. This gives more flexibility, but much more room for you to hang yourself. For example:
TYPE Student = REF RECORD lastname,firstname:TEXT END; Address = REF RECORD street:TEXT; number:CARDINAL END; VAR r1 : Student; r2 := NEW(Student, firstname:="Julie", lastname:="Tall"); adr := NEW(Address, street:="Washington", number:="21"); any := REFANY; (* always a safe assignment *) (* legal because any is of type student *) (* produces a run-time error, not compile-time

BEGIN any := r2; r1 := any; adr := any; *)

You should worry about the ideas behind generic implementations (why does Modula-3 do it this way?) more than the syntactic details (how does Modula-3 let you do this?). It is very easy to get overwhelmed by the detail. Generic Types When we think about the abstract data type ``Stack'' or ``Queue'', the implementation of th the data structure is pretty much the same whether we have a stack of integers or reals. Without generic types, we are forced to declare the type of everything at compile time. Thus we need two distinct sets of functions, like PushInteger and PushReal for each operation, which is waste. Object-Oriented programming languages provide features which enable us to create abstract data types which are more truly generic, making it cleaner and easier to reuse code.

Object-Oriented Programming Lecture 13


Why Objects are Good Things

Modules provide a logical grouping of procedures on a related topic. Objects provide a logical grouping of data and associated operations. The emphasis of modules is on procedures; the emphasis of objects is on data. Modules are verbs followed by nouns:Push(S,x), while objects are nouns followed by verbs: S.Push(x). This provides only an alternate notation for dealing with things, but different notations can sometimes make it easier to understand things - the history of Calculus is an example. Objects do a great job of encapsulating the data items within, because the only access to them is through the methods, or associated procedures. Stack Object
MODULE StackObj EXPORTS Main; (* Stack implemented as object type. *) IMPORT SIO; TYPE ET = INTEGER; Stack = OBJECT top: Node := NIL; METHODS push(elem:ET):= Push; pop() :ET:= Pop; empty(): BOOLEAN:= Empty; END; (*Stack*) Node = REF RECORD info: ET; next: Node the stack *) END; (*Node*) (*Type of elements*) (*points to stack*) (*Push implements push*) (*Pop implements pop*) (*Empty implements empty*) (*Stands for any information*) (*Points to the next node in (*24.01.95. LB*)

PROCEDURE Push(stack: Stack; elem:ET) = (*stack: receiver object (self)*) VAR new: Node := NEW(Node, info:= elem); (*Element instantiate*) BEGIN new.next:= stack.top; stack.top:= new; (*new element added to top*) END Push; PROCEDURE Pop(stack: Stack): ET = (*stack: receiver object (self)*) VAR first: ET; BEGIN

first:= stack.top.info; element*) stack.top:= stack.top.next; RETURN first END Pop; PROCEDURE Empty(stack: Stack): BOOLEAN = (*stack: receiver object (self)*) BEGIN RETURN stack.top = NIL END Empty; VAR stack1, stack2: Stack := NEW(Stack); i1, i2: INTEGER; BEGIN stack1.push(2); stack2.push(6); i1:= stack1.pop(); i2:= stack2.pop(); SIO.PutInt(i1); SIO.PutInt(i2); SIO.Nl(); END StackObj.

(*Info copied from first (*first element removed*)

(*2 stack objects created*) (*2 pushed onto stack1*) (*6 pushed onto stack2*) (*pop element from stack1*) (*pop element from stack2*)

Object-Oriented Programming Object-oriented programming is a popular, recent way of thinking about program organization. OOP is typically characterized by three major ideas:

Encapsulation - objects incorporate both data and procedures. Inheritance - classes (object types) are arranged in a hierarchy, and each class inherits but specializes methods and data from its ancestors. Polymorphism - a particular object can take on different types at different times. We saw this with REFANY variables whose types depend upon what is assigned it it (dynamic binding).

Inheritance When we define an object type (class), we can specify that it be derived from (subtype to) another class. For example, we can specialize the Stack object into a GarbageCan:
TYPE GarbageCan = Stack OBJECT OVERRIDES pop():= Yech; (* Remove something from can?? *) dump():= RemoveAll; (* Discard everything from can *)

END; (*GarbageCan*)

The GarbageCan type is a form of stack (you can still push in it the same way), but we have modified the pop and dump methods. This subtype-supertype relation defines a hierarchy (rooted tree) of classes. The appropriate method for a given object is determined at run time (dynamic binding) according to the first class at or above the current class to define the method. OOP and the Calculator Program How might object-oriented programming ideas have helped in writing the calculator program? Many of you noticed that the linked stack type was similar to the long integer type, and wanted to reuse the code from one in another. The following type hierarchy shows one way we could have exploited this, by creating special stack methods push and pop, and overwriting the add and subtract methods for general long-integers. Philosophical issue: should Long-Integer be a subtype of Positive-LongInteger or visa versa? Why didn't I urge you to do it this way? In my opinion, the complexity of mastering and using the OOP features of Modula-3 would very much overwhelm the code savings from such a small program. Object-oriented features differ significantly from language to language, but the basic principles outlined here are fairly common. However, you should see why inheritance can be a big win in organizing larger programs.

Simulations Lecture 14
Simulations Often, a system we are interested in may be too complicated to readily understand, and too expensive or big to experiment with.

What direction will an oil spill move in the Persian Gulf, given certain weather conditions? How much will increases in the price of oil change the American unemployment rate? Now much traffic can an airport accept before long delays become common?

We can often get good insights into hard problems by performing mathematical simulations. Scoring in Jai-alai Jai-alai is a Basque variation of handball, which is important because you can bet on it in Connecticut. What is the best way to bet? The scoring system in use in Connecticut is very interesting. Eight players or teams appear in each match, numbered 1 to 8. The players are arranged in a queue, and the top two players in the queue play each other. The winner gets a point and keeps playing, the loser goes to the end of the queue. Winner is the first one to get to 7 points. This scoring obviously favors the low numbered players. For fairness, after the first trip through the queue, each point counts two. But now is this scoring system fair? Simulating Jai-Alai
1 PLAYS 2 1 WINS THE POINT, GIVING HIM 1 PLAYS 3 3 WINS THE POINT, GIVING HIM 4 PLAYS 3 3 WINS THE POINT, GIVING HIM 5 PLAYS 3 3 WINS THE POINT, GIVING HIM 6 PLAYS 3 3 WINS THE POINT, GIVING HIM 7 PLAYS 3 3 WINS THE POINT, GIVING HIM 8 PLAYS 3 1 1 2 3 4 5

8 WINS THE POINT, GIVING HIM 8 PLAYS 2 2 WINS THE POINT, GIVING HIM 1 PLAYS 2 2 WINS THE POINT, GIVING HIM 4 PLAYS 2 2 WINS THE POINT, GIVING HIM 5 PLAYS 2 5 WINS THE POINT, GIVING HIM 5 PLAYS 6 5 WINS THE POINT, GIVING HIM 5 PLAYS 7 7 WINS THE POINT, GIVING HIM 3 PLAYS 7 7 WINS THE POINT, GIVING HIM 8 PLAYS 7 7 WINS THE POINT, GIVING HIM 1 PLAYS 7 7 WINS THE POINT, GIVING HIM WIN-PLACE-SHOW IS BETTER THAN AVERAGE TRIFECTAS: WIN PLACE SHOW 7 OCCURRENCES 2 3 7

1 2 4 6 2 4 2 4 6 8 2 1 TRIALS 3

Is the Scoring Fair? How can we test if the scoring system is fair? We can simulate a lot of games and see how often each player wins the game! But when player A plays a point against player B, how do we decide who wins? If the players are all equally matched, we can flip a coin to decide. We can use a random number generator to flip the coin for us! What data structures do we need?

A queue to maintain the order of who is next to play. An array to keep track of each player's score during the game. A array to keep track of how often a player has won so far.

Simulation Results
Jai-alai Simulation Results Pos 1 2 3 4 5 6 7 8 win 16549 16207 13584 12349 10103 10352 9027 11829 %wins 16.55 16.21 13.58 12.35 10.10 10.35 9.03 11.83 place 17989 17804 16735 13314 10997 7755 8143 7263 %places 17.99 17.80 16.73 13.31 11.00 7.75 8.14 7.26 show 15123 15002 14551 13786 13059 11286 9007 8186 %shows 15.12 15.00 14.55 13.79 13.06 11.29 9.01 8.19

total games = 100000

Compare these to the actual win results from Berenson's Jai-alai 1983-1986: 1 14.1%, 2 14.6%, 3 12.8%, 4 11.5%, 5 12.0%, 6 12.4%, 7 11.1%, 8 11.3% Were these results good? Yes, but not good enough to bet with! The matchmakers but the best players in the middle, so as to even the results. A more complicated model will be necessary for better results. Limitations of Simulations Although simulations are good things, there are several reasons to be skeptical of any results we get. Is the underlying model for the simulation accurate? Are the implicit assumptions reasonable, or are there biases? How do we know the program is an accurate implementation of the given model? After all, we wrote the simulation because we do not know the answers! How do you debug a simulation of two galaxies colliding or the effect of oil price increases on the economy? So much rides on the accuracy of simulations it is critical to build in selfverification tests, and prove the correctness of implementation. Random Number Generator

We have shown that random numbers are useful for simulations, but how do we get them? First we must realize that there is a philosophical problem with generating random numbers on a deterministic machine. ``Anyone who considers arithmetical methods of producing random digits is , of course, in a state of sin.'' - John Von Neumann What we really want is a good way to generate pseudo-random numbers, a sequence which has the same properties as a truly random source. This is quite difficult - people are lousy at picking random numbers. Note that the following sequence produces 0's + 1's with equal frequency but does not look like a fair coin:

Even recognizing random sequences is hard. Are the digits of random?

pseudo-

Should all those palindromes (535, 979, 46264, 383) be there? The Middle Square Method Von Neumann suggested generating random numbers by taking a big integer, squaring it, and using the middle digits as the seed/random number.

It looks random to me... But what happens when the middle digits just happen to be 0000000000? From then on, all digits will be zeros! Linear Congruential Generators The most popular random number generators, because of simplicity, quality, and small state requirements are linear congruential generators. If is the last random number we generated, then

The quality of the numbers generated depends upon careful selection of the seed and the constants a, c, and m.

Why does it work? Clearly, the numbers are between 0 and m-1. Taking the remainder mod m is like seeing where a roulette ball drops in a wheel with m slots.

SUNY at Stony BrookMidterm 1 CSE 214 - Data Structures October 10, 1997 Midterm Exam Name: Signature: ID #: Section #: INSTRUCTIONS:

You may use either pen or pencil. Check to see that you have 4 exam pages plus this cover (5 total). Look over all problems before starting work. Your signature above signs the CSE 214 Honor Pledge: ``On my honor as a student I have neither given nor received aid on this exam.'' Think before you write. Good luck!!

1) (25 points) Assume that you have the linked structure on the left, where each node contains a .next field consisting of a pointer, and the pointer p points to the structure as shown. Describe the sequence of Modula-3 pointer

manipulations necessary to convert it to the linked structure on the right. You may not change any of the .info fields, but you may use temporary pointerstmp1, tmp2, and tmp3 if you wish. Many different solutions were possible, including:
tmp1 := p; p := p.next; p^.next^.next := tmp1;

2) (30 points) Write a procedure which ``compresses'' a linked list by deleting consecutive copies of the same character from it. For example, the list (A,B,B,C,A,A,A,C,A) should be compressed to (A,B,C,A,C,A). Thus the same character can appear more than once in the compressed list, only not successively. Your procedure must have one argument as defined below, a VAR parameter head pointing to the front of the linked list. Each node in the list has .info and .next fields.
PROCEDURE compress(VAR head : pointer);

Many different solutions are possible, but recursive solutions are particularly clean and elegant.
PROCEDURE compress(VAR head : pointer); VAR second : pointer; (* pointer to next element *)

BEGIN IF (head # NIL) THEN second := head^.next; IF (second # NIL) IF (head^.info = second^.info) THEN head^.next = second^.next; compress(head); ELSE compress(head^.next); END; END; END; END;

3) (20 points) Provide the output of the following program:


MODULE strange; EXPORTS main; IMPORT SIO; TYPE ptr_to_integer = REF INTEGER; VAR a, b : ptr_to_integer; PROCEDURE modify(x : ptr_to_integer; VAR y : ptr_to_integer);

begin x^ := 3; SIO.PutInt(a^); SIO.PutInt(x^); SIO.Nl(); y^ := 4; SIO.PutInt(b^); SIO.PutInt(y^); SIO.Nl(); end; begin a := NEW(INTEGER); b := NEW(INTEGER); a^ := 1; b^ := 2; SIO.PutInt(a^); SIO.PutInt(b^); SIO.Nl(); modify(a,b); SIO.PutInt(a^); SIO.PutInt(b^); SIO.Nl(); end.

Answers:
1 3 4 3 2 3 4 4

4) (25 points) Write brief essays answering the following questions. Your answer must fit completely in the space allowed (a) Explain the difference between objects and modules? ANSWER: Several answers possible, but the basic differences are (1) the notation to use them, and (2) that objects encapsulate both procedures and data where modules are procedure oriented. (b) What is garbage collection? ANSWER: The automatic reuse of dynamic memory which, because of pointer dereferencing, is no longer accessible. (c) What might be an advantage of a doubly-linked list over a singly-linked list for certain applications? ANSWER: Additional flexibility in moving both forward and in reverse on a linked list. Specific advantages include being able to delete a node from a list given just a pointer to the node, and efficiently implementing double-ended queues (supporing push, pop, enqueue, and dequeue).

Asymptotics Lecture 15
Analyzing Algorithms There are often several different algorithms which correctly solve the same problem. How can we choose among them? There can be several different criteria:

Ease of implementation Ease of understanding Efficiency in time and space

The first two are somewhat subjective. However, efficiency is something we can study with mathematical analysis, and gain insight as to which is the fastest algorithm for a given problem. Time Complexity of Programs What would we like as the result of the analysis of an algorithm? We might hope for a formula describing exactly how long a program implementing it will run. Example: Binary search will take of n elements. milliseconds on an array

This would be great, for we could predict exactly how long our program will take. But it is not realistic for several reasons:
1. Dependence on machine type - Obviously, binary search will run faster

on a CRAY than a PC. Maybe binary search will now take ms? 2. Dependence on language/compiler - Should our time analysis change when someone uses an optimizing compiler? 3. Dependence of the programmer - Two different people implementing the same algorithm will result in two different programs, each taking slightly differed amounts of time.

4. Should your time analysis be average or worst case? - Many algorithms

return answers faster in some cases than others. How did you factor this in? Exactly what do you mean by average case? 5. How big is your problem? - Sometimes small cases must be treated different from big cases, so the same formula won't work. Time Complexity of Algorithms For all of these reasons, we cannot hope to analyze the performance of programs precisely. We can analyze the underlyingalgorithm, but at a less precise level. Example: Binary search will use about iterations, where each iteration takes time independent of n, to search an array of n elements in the worst case. Note that this description is true for all binary search programs regardless of language, machine, and programmer. By describing the worst case instead of the average case, we saved ourselves some nasty analysis. What is the average case? Algorithms for Multiplications Everyone knows two different algorithms for multiplication: repeated addition and digit-by-digit multiplication. Which is better? Let's analyze the complexity of multiplying an n-digit number by an m-digit number, where . . Thus requires ``about'' n+m steps, one

In repeated addition, we explicity use that adding an n-digit + m-digit number, for each digit.

How many additions can we do in the worst case? The biggest n-digit number is all nines, and .

The total time complexity is the cost per addition times the number of additions, so the total complexity .

Digit-by-Digit Multiplication Since multiplying one digit by one other digit can be done by looking up in a multiplication table (2D array), each step requires a constant amount of work. Thus to multiply an n-digit number by one digit requires ``about'' n steps. With m ``extra'' zeros (in the worst case), ``about'' n+ m steps certainly suffice. We must do m such multiplications and add them up - each add costs as much as the multiplication. The total complexity is the cost-per-multiplication * number-of-multiplications + cost-per-addition * number-of- multiplication Which is faster? .

Clearly the repeated addition method is much slower by our analysis, and the difference is going to increase rapidly with n... Further, it explains the decline and fall of Roman empire - you cannot do digitby-digit multiplication with Roman numbers! Growth Rates of Functions To compare the efficiency of algorithms then, we need a notation to classify numerical functions according to their approximate rate of growth. We need a way of exactly comparing approximately defined functions. This is the big Oh Notation: If f(n) and g(n) are functions defined for positive integers, then f(n)= O(g(n)) means that there exists a constant csuch that all sufficiently large positive integers. for

The idea is that if f(n)=O(g(n)), then f(n) grows no faster (and possibly slower) than g(n). Note this definition says nothing about algorithms - it is just a way to compare numerical functions! Examples Example: clearly Example: pick, is not is . Why? For all n > 100, , so it satisfies the definition for c=100. . Why? No matter what value of c you is not true for n>c!

In the big Oh Notation, multiplicative constants and lower order terms are unimportant. Exponents are important.

Ranking functions by the Big Oh The following functions are different according to the big Oh notation, and are ranked in increasing order: O(1) Constant growth

Logarithmic growth (note:independent of base!)

Polynomial growth: ordered by exponent

O(n) Linear Growth

Quadratic growth

Exponential growth Why is the big Oh a Big Deal? Suppose I find two algorithms, one of which does twice as many operations in solving the same problem. I could get the same job done as fast with the slower algorithm if I buy a machine which is twice as fast. But if my algorithm is faster by a big Oh factor - No matter how much faster you make the machine running the slow algorithmthe fast-algorithm, slow machine combination will eventually beat the slow algorithm, fast machine combination. I can search faster than a supercomputer for a large enough dictionary, If I use binary search and it uses sequential search!

An Application: The Complexity of Songs Suppose we want to sing a song which lasts for n units of time. Since n can be large, we want to memorize songs which require only a small amount of brain space, i.e. memory. Let S(n) be the space complexity of a song which lasts for n units of time. The amount of space we need to store a song can be measured in either the words or characters needed to memorize it. Note that the number of characters is since every word in a song is at most 34 letters long Supercalifragilisticexpialidocious! What bounds can we establish on S(n)? S(n) = O(n), since in the worst case we must explicitly memorize every word we sing - ``The Star-Spangled Banner'' The Refrain Most popular songs have a refrain, which is a block of text which gets repeated after each stanza in the song: Bye, bye Miss American pie Drove my chevy to the levy but the levy was dry Them good old boys were drinking whiskey and rye Singing this will be the day that I die. Refrains made a song easier to remember, since you memorize it once yet sing it O(n) times. But do they reduce the space complexity? Not according to the big oh. If

Then the space complexity is still O(n) since it is only halved (if the verse-size = refrain-size):

The k Days of Christmas To reduce S(n), we must structure the song differently.

Consider ``The k Days of Christmas''. All one must memorize is: On the kth Day of Christmas, my true love gave to me, On the First Day of Christmas, my true love gave to me, a partridge in a pear tree But the time it takes to sing it is

If Beer

, then

, so

. 100 Bottles of

What do kids sing on really long car trips? n bottles of beer on the wall, n bottles of beer. You take one down and pass it around n-1 bottles of beer on the ball. All you must remember in this song is this template of size current value of n. The storage size for n depends on its value, but suffice. This for this song, Uh-huh, uh-huh Is there a song which eliminates even the need to count? That's the way, uh-huh, uh-huh I like it, uh-huh, huh Reference: D. Knuth, `The Complexity of Songs', Comm. ACM, April 1984, pp.18-24 . , and the bits

Introduction to Sorting Lecture 16


Sorting Sorting is, without doubt, the most fundamental algorithmic problem 1. Supposedly, 25% of all CPU cycles are spent sorting 2. Sorting is fundamental to most other algorithmic problems, for example binary search. 3. Many different approaches lead to useful sorting algorithms, and these ideas can be used to solve many other problems. What is sorting? It is the problem of taking an arbitrary permutation of n items and rearranging them into the total order,

Knuth, Volume 3 of ``The Art of Computer Programming is the definitive reference of sorting. Issues in Sorting Increasing or Decreasing Order? - The same algorithm can be used by both all we need do is change to in the comparison function as we desire.

What about equal keys? - Does the order matter or not? Maybe we need to sort on secondary keys, or leave in the same order as the original permutations. What about non-numerical data? - Alphabetizing is sorting text strings, and libraries have very complicated rules concerning punctuation, etc. Is BrownWilliams before or after Brown America before or after Brown, John? We can ignore all three of these issues by assuming a comparison function which depends on the application. Compare (a,b)should return ``<'', ``>'', or ''=''. Applications of Sorting

One reason why sorting is so important is that once a set of items is sorted, many other problems become easy. SearchingBinary search lets you test whether an item is in a dictionary in time.

Speeding up searching is perhaps the most important application of sorting. Closest pairGiven n numbers, find the pair which are closest to each other. Once the numbers are sorted, the closest pair will be next to each other in sorted order, so an O(n) linear scan completes the job. Element uniquenessGiven a set of n items, are they all unique or are there any duplicates? Sort them and do a linear scan to check all adjacent pairs. This is a special case of closest pair above. Frequency distribution - ModeGiven a set of n items, which element occurs the largest number of times? Sort them and do a linear scan to measure the length of all adjacent runs. Median and SelectionWhat is the kth largest item in the set? Once the keys are placed in sorted order in an array, the kth largest can be found in constant time by simply looking in the kth position of the array. How do you sort? There are several different ideas which lead to sorting algorithms:

Insertion - putting an element in the appropriate place in a sorted list yields a larger sorted list. Exchange - rearrange pairs of elements which are out of order, until no such pairs remain. Selection - extract the largest element form the list, remove it, and repeat. Distribution - separate into piles based on the first letter, then sort each pile.

Merging - Two sorted lists can be easily combined to form a sorted list.

Selection Sort In my opinion, the most natural and easiest sorting algorithm is selection sort, where we repeatedly find the smallest element, move it to the front, then repeat...
* 5 7 3 2 2 * 7 3 5 2 3 * 7 5 2 3 5 * 7 2 3 5 7 * 8 8 8 8 8

If elements are in an array, swap the first with the smallest element- thus only one array is necessary. If elements are in a linked list, we must keep two lists, one sorted and one unsorted, and always add the new element to the back of the sorted list. Selection Sort Implementation
MODULE SimpleSort EXPORTS Main; (*1.12.94. LB*) (* Sorting and text-array by selecting the smallest element *) TYPE Array = ARRAY [1..N] OF TEXT; VAR a: Array; (*the array in which to search*) x: TEXT; (*auxiliary variable*) last, (*last valid index *) min: INTEGER; (* current minimum*) BEGIN ... FOR i:= FIRST(a) TO last - 1 DO min:= i; (*index of smallest element*) FOR j:= i + 1 TO last DO IF Text.Compare(a[j], a[min]) = -1 THEN (*IF a[i] < a[min]*) min:= j END; END; (*FOR j*) x:= a[min]; (* swap a[i] and a[min] *) a[min]:= a[i]; a[i]:= x; END; (*FOR i*) ...

END SimpleSort.

The Complexity of Selection Sort One interesting observation is that selection sort always takes the same time no matter what the data we give it is! Thus the best case, worst case, and average cases are all the same! Intuitively, we make n iterations, each of which ``on average'' compares n/2, so we should make about comparisons to sort n items.

To do this more precisely, we can count the number of comparisons we make. To find the largest takes (n-1) steps, to find the second largest takes (n-2) steps, to find the third largest takes (n-3) steps, ... to find the last largest takes 0 steps.

An advantage of the big Oh notation is that fact that the worst case time is obvious - we have n loops of at most nsteps each. If instead of time we count the number of data movements, there are n-1, since there is exactly one swap per iteration. Insertion Sort In insertion sort, we repeatedly add elements to a sorted subset of our data, inserting the next element in order:
* 5 7 3 2 5 * 7 3 2 3 5 * 7 2 2 3 5 * 7 2 3 5 7 * 8 8 8 8 8

InsertionSort(A)

for i = 1 to n-1 do

j=i

while (A[j] > A[j-1]) do swap(A[j],A[j1])

In inserting the element in the sorted section, we might have to move many elements to make room for it. If the elements are in an array, we scan from bottom to top until we find the j such that one to make room. , then move from j+1 to the end down

If the elements are in a linked list, we do the sequential search until we find where the element goes, then insert the element there.No other elements need move! Complexity of Insertion Sort Since we do not necessarily have to scan the entire sorted section of the array, the best, worst, and average cases for insertion sort all differ! Best case: the element always gets inserted at the end, so we don't have to move anything, and only compare against the last sorted element. We have (n-1) insertions, each with exactly one comparison and no data moves per insertion! What is this best case permutation? It is when the array or list is already sorted! Thus insertion sort is a great algorithm when the data has previously been ordered, but slightly messed up. Worst Case Complexity

Worst case: the element always gets inserted at the front, so all the sorted elements must be moved at each insertion. The ith insertion requires (i-1) comparisons and moves so:

What is the worst case permutation? When the array is sorted in reverse order. This is the same number of comparisons as with selection sort, but uses more movements. The number of movements might get important if we were sorting large records. Average Case Complexity Average Case: If we were given a random permutation, the chances of the ith insertion requiring comparisons are equal, and hence 1/i.

The expected number of comparisons is for the ith insertion is:

Summing up over all n keys,

So we do half as many comparisons/moves on average! Can we use binary search to help us get below time?

About this document ...

Next: About this document Up: My Home Page

Steve Skiena Thu Oct 16 20:14:07 EDT 1997

Mergesort and Quicksort Lecture 17


Faster than O( ) Sorting?

Can we find a sorting algorithm which does significantly better than comparing each pair of elements? If not, we are doomed to quadratic time complexity....

Since sorting large numbers of items is such an important problem an Logarithms It is important to understand deep in your bones what logarithms are and where they come from. A logarithm is simply an inverse exponential function. Saying equivalent to saying that . is algorithm is the way to go!

Exponential functions, like the amount owed on a n year mortgage at an interest rate of per year, are functions which grow distressingly fast. Thus inverse exponential functions, ie. logarithms, grow refreshingly slowly.

Binary search is an example of an algorithm. After each comparison, we can throw away half the possible number of keys. Thus twenty comparisons suffice to find any name in the million-name Manhattan phone book! If you have an algorithm which runs in blindingly fast even on very large instances. Properties of Logarithms Recall the definition, . time, take it, because this is

Asymptotically, the base of the log does not matter:

Thus, that is just a constant.

, and note

Asymptotically, any polynomial function of n does not matter:Note that

since and .

Any exponential dominates every polynomial. This is why we will seek to avoid exponential time algorithms. Federal Sentencing Guidelines 2F1.1. Fraud and Deceit; Forgery; Offenses Involving Altered or Counterfeit Instruments other than Counterfeit Bearer Obligations of the United States. (a) Base offense Level: 6 (b) Specific offense Characteristics

(1) If the loss exceeded $2,000, increase the offense level as follows:

The federal sentencing guidelines are designed to help judges be consistent in assigning punishment. The time-to-serve is a roughly linear function of the total level. However, notice that the increase in level as a function of the amount of money you steal grows logarithmically in the amount of money stolen. This very slow growth means it pays to commit one crime stealing a lot of money, rather than many small crimes adding up to the same amount of money, because the time to serve if you get caught is much less. The Moral: ``if you are gonna do the crime, make it worth the time!'' Mergesort

Given two sorted lists with a total of n elements, at most n-1 comparisons are required to merge them into one sorted list. Repeatedly compare the top elements on each list. Example: and .

No more comparisons are needed once the list is empty. Fine, but how do we get the smaller sorted lists to start with? We do merges of even smaller lists! Working backwards, we eventually get to lists of one element, which are by definition sorted! Mergesort Example

Note that on each iteration, the size of the sorted lists doubles, form 1 to 2 to 4 to 8 to 16 ...to n. How many doublings (or iterations) does it take before the entire array of size n is sorted? Answer: .

How much work do we do per iteration?

In merging the lists of 1 element, we have comparison, for a total of comparisons.

merges, each requiring 1

In merging the lists of 2 elements, we have most 3 comparisons, for a total of ... In merging the lists of 2 elements, we have at most comparisons, for a total of

merges, each requiring at comparisons.

merges, each requiring .

This is always less than n per stage!!! If we make at most n comparisons in each of stages, we make at most comparisons in total! - it is the

Make sure you understand why mergesort is conceptually simplest Space Requirements for Mergesort algorithm we will see.

How much extra space (over the space used to represent the input elements) do we need to do mergesort? It is easy to merge two sorted linked lists without using any extra space. However, to merge two sorted arrays (or portions of an array), we must use a third array to store the result of the merge. This avoids steping on elements we have not needed yet: Example: Merge ((4,5,6), (1,2,3)). QuickSort Although Mergesort is , it is somewhat inconvienient to implementate using arrays, since we need space to merge.

In practice, the fastest sorting algorithm is Quicksort, which uses partitioning as its main idea. Example: Pivot about 10. 17 12 6 19 23 8 5 10 - before 6 8 5 10 23 19 12 17 - after Partitioning places all the elements less than the pivot in the left part of the array, and all elements greater than the pivot in theright part of the array. The pivot fits in the slot between them. Note that the pivot element ends up in the correct place in the total order! Partitioning the elements Once we have selected a pivot element, we can partition the array in one linear scan, by maintaining three sections of the array: < pivot, > pivot, and unexplored. Example: pivot about 10
| 17 12 6 19 23 8 5 | 10 | 5 12 6 19 23 8 | 17 5 | 12 6 19 23 8 | 17 5 | 8 6 19 23 | 12 17 5 8 | 6 19 23 | 12 17 5 8 6 | 19 23 | 12 17 5 8 6 | 23 | 19 12 17 5 8 6 ||23 19 12 17 5 8 6 10 19 12 17 23

As we scan from left to right, we move the left bound to the right when the element is less than the pivot, otherwise we swap it with the rightmost unexplored element and move the right bound one step closer to the left. Since the partitioning step consists of at most n swaps, takes time linear in the number of keys. But what does it buy us? 1. The pivot element ends up in the position it retains in the final sorted order. 2. After a partitioning, no element flops to the other side of the pivot in the final sorted order.

Thus we can sort the elements to the left of the pivot and the right of the pivot independently! This gives us a recursive sorting algorithm, since we can use the partitioning approach to sort each subproblem. Quicksort Implementation
MODULE Quicksort EXPORTS Main; (*18.07.94. LB*) (* Read in an array of integers, sort it using the Quicksort algorithm, and output the array. See Chapter 14 for the explanation of the file handling and Chapter 15 for exception handling, which is used in this example. IMPORT SIO, SF; VAR out: SIO.Writer; TYPE ElemType = INTEGER; VAR array: ARRAY [1 .. 10] OF ElemType; PROCEDURE InArray(VAR a: ARRAY OF ElemType) RAISES {SIO.Error} = (*Reads a sequence of numbers. Passes SIO.Error for bad file format.*) VAR in:= SF.OpenRead("vector"); (*open input file*) BEGIN FOR i:= FIRST(a) TO LAST(a) DO a[i]:= SIO.GetInt(in) END; END InArray; PROCEDURE OutArray(READONLY a: ARRAY OF ElemType) = (*Outputs an array of numbers*) BEGIN FOR i:= FIRST(a) TO LAST(a) DO SIO.PutInt(a[i], 4, out) END; SIO.Nl(out); END OutArray; PROCEDURE Quicksort(VAR a: ARRAY OF ElemType; left, right: CARDINAL) = VAR i, j: INTEGER; x, w: ElemType; BEGIN (*Partitioning:*) i:= left; left*) j:= right; x:= a[(left + right) DIV 2]; REPEAT (*i iterates upwards from (*j iterates down from right*) (*x is the middle element*)

*)

part*) part*)

WHILE a[i] < x DO INC(i) END; WHILE a[j] > x DO DEC(j) END;

(*skip elements < x in left (*skip elements > x in right (*swap a[i] and a[j]*)

IF i <= j THEN w:= a[i]; a[i]:= a[j]; a[j]:= w; INC(i); DEC(j); END; (*IF i <= j*) UNTIL i > j;

(*recursive application of partitioning to subarrays:*) IF left < j THEN Quicksort(a, left, j) END; IF i < right THEN Quicksort(a, i, right) END; END Quicksort; BEGIN TRY (*grasps bad file format*) InArray(array); (*read an array in*) out:= SF.OpenWrite(); (*create output file*) OutArray(array); (*output the array*) Quicksort(array, 0, NUMBER(array) - 1); (*sort the array*) OutArray(array); (*display the array*) SF.CloseWrite(out); (*close output file to make it permanent*) EXCEPT SIO.Error => SIO.PutLine("bad file format"); END; (*TRY*) END Quicksort.

Best Case for Quicksort Since each element ultimately ends up in the correct position, the algorithm correctly sorts. But how long does it take? The best case for divide-and-conquer algorithms comes when we split the input as evenly as possible. Thus in the best case, each subproblem is of size n/2. The partition step on each subproblem is linear in its size. Thus the total effort in partitioning the problems of size isO(n).

The recursion tree for the best case looks like this: The total partitioning on each level is O(n), and it take levels of perfect partitions to get to single element subproblems. When we are down to single

elements, the problems are sorted. Thus the total time in the best case is .

Worst Case for Quicksort Suppose instead our pivot element splits the array as unequally as possible. Thus instead of n/2 elements in the smaller half, we get zero, meaning that the pivot element is the biggest or smallest element in the array. Now we have n-1 levels, instead of since the first n/2 levels each have , for a worst case time of elements to partition. ,

Thus the worst case time for Quicksort is worse than Heapsort or Mergesort. To justify its name, Quicksort had better be good in the average case. Showing this requires some fairly intricate analysis. The divide and conquer principle applies to real life. If you will break a job into pieces, it is best to make the pieces of equal size! Intuition: The Average Case for Quicksort The book contains a rigorous proof that quicksort is in the average case. I will instead give an intuitive, less formal explanation of why this is so. Suppose we pick the pivot element at random in an array of n keys. Half the time, the pivot element will be from the center half of the sorted array. Whenever the pivot element is from positions n/4 to 3n/4, the larger remaining subarray contains at most 3n/4 elements. If we assume that the pivot element is always in this range, what is the maximum number of partitions we need to get from nelements down to 1 element?

good partitions suffice. At most levels of decent partitions suffices to sort an array of n elements. But how often when we pick an arbitrary element as pivot will it generate a decent partition? Since any number ranked between n/4 and 3n/4 would make a decent pivot, we get one half the time on average. If we need levels of decent partitions to finish the job, and half of random partitions are decent, then on average the recursion tree to quicksort the array has levels.

Since O(n) work is done partitioning on each level, the average time is .

More careful analysis shows that the expected number of comparisons is .

What is the Worst Case? The worst case for Quicksort depends upon how we select our partition or pivot element. If we always select either the first or last element of the subarray, the worst-case occurs when the input is already sorted!
A B D F B D F D F F H H H H H J J J J J J K K K K K K K

Having the worst case occur when they are sorted or almost sorted is very bad, since that is likely to be the case in certain applications. To eliminate this problem, pick a better pivot:

1. Use the middle element of the subarray as pivot. 2. Use a random element of the array as the pivot. 3. Perhaps best of all, take the median of three elements (first, last, middle) as the pivot. Why should we use median instead of the mean? Whichever of these three rules we use, the worst case remains . However, because the worst case is no longer a natural order it is much more difficult to occur. Is Quicksort really faster than Mergesort? Since Mergesort is and selection sort is debate about which will be better for decent-sized files. , there is no

But how can we compare two algorithms to see which is faster? Using the RAM model and the big Oh notation, we can't! When Quicksort is implemented well, it is typically 2-3 times faster than mergesort or heapsort. The primary reason is that the operations in the innermost loop are simpler. The best way to see this is to implement both and experiment with different inputs. Since the difference between the two programs will be limited to a multiplicative constant factor, the details of how you program each algorithm will make a big difference. If you don't want to believe me when I say Quicksort is faster, I won't argue with you. It is a question whose solution lies outside the tools we are using. The best way to tell is to implement them and experiment. Combining Quicksort and Insertion Sort When we compare the expected number of comparisons for Quicksort + Insertion sort, a funny thing happens for small n:

Why not take advantage of this, and switch over to insertion sort when the size of the subarray falls below a certain threshhold? Why not indeed? But how do we find the right switch point to optimize performance? Experiments are more useful than analysis here. Randomization Suppose you are writing a sorting program, to run on data given to you by your worst enemy. Quicksort is good on average, but bad on certain worst-case instances. If you used Quicksort, what kind of data would your enemy give you to run it on? Exactly the worst-case instance, to make you look bad. But instead of picking the median of three or the first element as pivot, suppose you picked the pivot element at random. Now your enemy cannot design a worst-case instance to give to you, because no matter which data they give you, you would have the same probability of picking a good pivot! Randomization is a very important and useful idea. By either picking a random pivot or scrambling the permutation before sorting it, we can say: ``With high probability, randomized quicksort runs in Where before, all we could say is: ``If you give me random input data, quicksort runs in expected time.'' time.''

Since the time bound how does not depend upon your input distribution, this means that unless we are extremely unlucky (as opposed to ill prepared or unpopular) we will certainly get good performance. Randomization is a general tool to improve algorithms with bad worst-case but good average-case complexity. The worst-case is still there, but we almost certainly won't see it.

Priority Queues and Heapsort Lecture 18


Who's Number 2? In most sports playoffs, a single elimination tournament is used to decide the championship. The Marlins were clearly the best team in the 1997 World Series, since they were the only one without a loss. But who is number 2? The Giants, Braves, and Indians all have equal claims, since only the champion beat them! Each game can be thought of as a comparison. Given n keys, we would like to determine the k largest values. Can we do better than just sorting all of them? In the tournament example, each team represents an leaf of the tree and each game is an internal node of the tree. Thus there are n-1 games/comparisons for n teams/leaves. Note that the champion is identified even though no team plays more than games!

Lewis Carroll, author of ``Alice in Wonderland'', studied this problem in the 19th century in order to design better tennis tournaments! We will seek a data structure which will enable us to repeatedly identify the largest key, and then delete it to retrieve the largest remaining key. This data structure is called a heap, as in ``top of the heap''.

Binary Heaps A binary heap is defined to be a binary tree with a key in each node such that:
1. All leaves are on, at most, two adjacent levels. 2. All leaves on the lowest level occur to the left, and all levels except the

lowest are completely filled.


3. The key in the root is

all its children, and the left and right subtrees are again binary heaps. (This is a recursive definition)

Conditions 1 and 2 specify the shape of the tree, while condition 3 describes the labeling of the nodes tree. Unlike the tournament example, each label only appears on one node. Note that heaps are not binary search trees, but they are binary trees. Heap Test Where is the largest element in a heap? Answer - the root. Where is the second largest element? Answer - as the root's left or right child. Where is the smallest element? Answer - it is one of the leaves. Can we do a binary search to find a particular key in a heap? Answer - No! A heap is not a binary search tree, and cannot be effectively used for searching. Why Do Heaps Lean Left? As a consequence of the structural definition of a heap, each of the n items can be assigned a number from 1 to n with the property that the left child of node number k has a number 2k and the right child number 2k+1.

Thus we can store the heap in an n element array without pointers! If we did not enforce the left constraint, we might have holes, and need room for elements to store n things. This implicit representation of trees saves memory but is less flexible than using pointers. For this reason, we will not be able to use them when we discuss binary search trees. Constructing Heaps Heaps can be constructed incrementally, by inserting new elements into the left-most open spot in the array. If the new element is greater than its parent, swap their positions and recur. Since at each step, we replace the root of a subtree by a larger one, we preserve the heap order. Since all but the last level is always filled, the height h of an n element heap is bounded because:

so

. , since each insertion takes at

Doing n such insertions takes most time.

Deleting the Root The smallest (or largest) element in the heap sits at the root. Deleting the root can be done by replacing the root by the nth key (which must be a leaf) and letting it percolate down to its proper position! The smallest element of (1) the root, (2) its left child, and (3) its right child is moved to the root. This leaves at most one of the two subtrees which is not in heap order, so we continue one level down.

After completed in

steps of O(1) time each, we reach a leaf, so the deletion is time.

This percolate-down operation is called often Heapify, for it merges two heaps with a new root. Heapsort An initial heap can be constructed out on n elements by incremental insertion in time:

Build-heap(A)

for i = 2 to n do

HeapInsert(A[i], A)

Exchanging the maximum element with the last element and calling heapify repeatedly gives an sorting algorithm, named Heapsort.

Heapsort(A)

Build-heap(A)

for i = n to 1 do

swap(A[1],A[i])

n = n - 1

Heapify(A,1)

Advantages of heapsort include:


No extra space (Quicksort needs a stack) No worst case trouble. Simpler to get fast and correct than Quicksort.

The Lesson of Heapsort Always ask yourself, ``Can we use a different data structure?'' Selection sort scans throught the entire array, repeatedly finding the smallest remaining element.

For i = 1 to n

A:

Find the smallest of the first n-i+1 items.

B:

Pull it out of the array and put it first.

Using arrays or unsorted linked lists as the data structure, operation A takes O(n) time and operation B takes O(1). Using heaps, both of these operations can be done within balancing the work and achieving a better tradeoff. time,

Priority Queues A priority queue is a data structure on sets of keys supporting the operations: Insert(S, x) - insert x into set S, Maximum(S) - return the largest key in S, and ExtractMax(S) - return and remove the largest key in S These operations can be easily supported using a heap.

Insert - use the trickle up insertion in . Maximum - read the first element in the array in O(1). Extract-Max - delete first element, replace it with the last, decrement the element counter, then heapify in .

Application: Heaps as stacks or queues


In a stack, push inserts a new item and pop removes the most recently pushed item. In a queue, enqueue inserts a new item and dequeue removes the least recently enqueued item.

Both stacks and queues can be simulated by using a heap, when we add a new time field to each item and order the heap according it this time field.

To simulate the stack, increment the time with each insertion and put the maximum on top of the heap. To simulate the queue, decrement the time with each insertion and put the maximum on top of the heap (or increment times and keep the minimum on top)

This simulation is not as efficient as a normal stack/queue implementation, but it is a cute demonstration of the flexibility of a priority queue. Discrete Event Simulations In simulations of airports, parking lots, and jai-alai - priority queues can be used to maintain who goes next. In a simulation, we often need to schedule events according to a clock. When someone is born, we may then immediately decide when they will die, and we will have to be reminded when to bury them!

The stack and queue orders are just special cases of orderings. In real life, certain people cut in line. Sweepline Algorithms in Computational Geometry In the priority queue, we will store the points we have not yet encountered, ordered by x coordinate. and push the line forward one stop at a time. Greedy Algorithms In greedy algorithms, we always pick the next thing which locally maximizes our score. By placing all the things in a priority queue and pulling them off in order, we can improve performance over linear search or sorting, particularly if the weights change. Example: Sequential strips in triangulations.

Sequential and Binary Search Lecture 19


Sequential Search The simplest algorithm to search a dictionary for a given key is to test successively against each element. This works correctly regardless of the order of the elements in the list. However, in the worst case all elements might have to be tested.
Procedure Search(head:pointer, key:item):pointer; Var p:pointer; found:boolean; Begin found:=false; p:=head; While (p # NIL) AND (not found) Do Begin If (p^.info = key) then

End; return p; END;

found = true; Else p = p^.next;

With and Without Sentinels A sentinel is a value placed at the end of an array to insure that the normal case of searching returns something even if the item is not found. It is a way to simplify coding by eliminating the special case.
MODULE LinearSearch EXPORTS Main; (*1.12.94. LB*) (* Linear search without a sentinel *) ... i:= FIRST(a); WHILE (i <= last) AND NOT Text.Equal(a[i], x) DO INC(i) END; IF i > last THEN SIO.PutText("NOT found"); ELSE SIO.PutText("Found at position: "); SIO.PutInt(i) END; (*IF i > last*) SIO.Nl(); END LinearSearch.

The sentinel insures that the search will eventually succeed:


MODULE SentinelSearch EXPORTS Main; (* Linear search with sentinel. *) ... (* Do search *) a[LAST(a)]:= x; (*sentinel at position N+1*) i:= FIRST(a); WHILE x # a[i] DO INC(i) END; (* Output result *) IF i = LAST(a) THEN SIO.PutText("NOT found"); ELSE SIO.PutText("Found at position: "); SIO.PutInt(i) END; SIO.Nl(); END SentinelSearch. (*27.10.93. LB*)

Weighted Sequential Search

Sometimes sequential search is not a bad algorithm, especially when the list isn't long. After all, sequential search is easier to implement than binary search, and does not require the list to be sorted. However, if we are going to do a sequential search, what order do we want the elements? Sorted order in a linked list doesn't really help, except maybe to help us stop early if the item isn't in the list. Suppose you were organizing your personal phone book for sequential search. You would want your most frequently called friends to be at the front: In sequential search, you want the keys ordered by frequency of use! Why? If is the probability of searching for the ith key, which is a distance from the front, the expected search time is

which is minimized by placing the list in decreasing probability of access order. For the list (Cheryl,0.4), (Lisa,0.25), (Lori,0.2), (Lauren,0.15), the expected search time is:

If access probability had been uniform, the expected search time would have been

So I win using this order, and win even more if the access probabilities are furthered skewed. But how do I find the probabilities? Self-Organizing Lists Since it is often impractical to compute usage frequencies, and because usage frequencies often change in the middle of a program (locality), we would like our data structure to automatically adjust to the distribution. Such data structures are called self-organizing.

The idea is to use a heuristic to move an element forward in the list whenever it is accessed. There are two possibilities:

Move forward one is the ``conservative'' approach. (1,2,3,4,5) becomes (1,2,4,3,5) after a Find(4). Move to front is the ``liberal'' approach. (1,2,3,4,5) becomes (4,1,2,3,5) after a Find(4).

Which Heuristic is Better? Move-forward-one can get caught in traps which won't fool move-to-front: For list (1,2,3,4,5,6,7,8), the queries Find(8), Find(7), Find(8), Find(7), ... will search the entire list every time. With move-to-front, it averages only two comparisons per query! In fact, it can be shown that the total search time with move-to-front is never more than twice the time if you knew the actual probabilities in advance!! We will see self-organization again later in the semester when we talk about splay trees. Let's Play 20 Questions!
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.

Binary Search Binary Search is an incredibly powerful technique for searching an ordered list. It is familiar to everyone who uses a telephone book! The basic algorithm is to find the middle element of the list, compare it against the key, decide which half of the list must contain the key, and repeat with that half.

Two requirements to support binary search:


Random access of the list elements, so we need arrays instead of linked lists. The array must contain elements in sorted order by the search key.

Why Do Twenty Questions Suffice? With one question, I can distinguish between two words: A and B; ``Is the key ?''

With two questions, I can distinguish between four words: A,B,C,D; ``Is the ?''

Each question I ask em doubles the number of words I can search in my dictionary. , which is much larger than any portable dictionary! Thus I could waste my first two questions because Exponents and Logs Recall the definitions of exponent and logarithm from high school: .

Thus exponentiation and logarithms are inverse functions, since Note that the logarithm of a big number is a much smaller number.

Thus the number of questions we must ask is the base two logarithm of the size of the dictionary.

Implementing Binary Search

Although the algorithm is simple to describe informally, it is tricky to produce a working binary search function. The first published binary search algorithm appeared in 1946, but the first correct published program appeared in 1962! The difficulty is maintaining the following two invariants with each iteration:

The key must always remain between the low and high indices. The low or high indice must advance each iteration.

The boundary cases are very tricky: zero elements left, one elements left, two elements left, and an even or odd number of elements! Versions of Binary Search There are at least two different versions of binary search, depending upon whether we want to test for equality at each query or only at the end. For the later, suppose we want to search for ``k'':
iteration bottom top mid --------------------------------------1 2 14 (1+14)/2=7 2 1 7 (1+7)/2=4 3 5 7 (5+7)/2=6 4 6 7 (7+7)/2=7

Since , 7 is the right spot. However, we must now test if entry[7]='k'. If not, the item isn't in the array. Alternately, we can test for equality at each comparison. Suppose we search for ``c'':
iteration bottom top mid -----------------------------------1 1 14 (1+14)/2 = 7 2 1 6 (1+6)/2 = 3 3 1 2 (1+2)/2 = 1 4 2 2 (2+2)/2 = 2

Now it will be found! Recursive Binary Search Implementation


PROCEDURE Search( READONLY array: ARRAY [0 .. MaxInd - 1] OF INTEGER; left, right: [0 .. MaxInd - 1]; argument: INTEGER): [0..MaxInd] =

(*Implements binary search in an array*) VAR middle := left + (right - left) DIV 2; BEGIN (* binary search *) IF argument = array[middle] THEN (*found*) RETURN middle ELSIF argument < array[middle] THEN (*search in left half*) IF left < middle THEN RETURN Search(array, left, middle - 1, argument) ELSE (*left boundary reaches middle: not found*) RETURN MaxInd END (*IF left < middle*) ELSE (*search in right half*) IF middle < right THEN RETURN Search(array, middle + 1, right, argument) ELSE (*middle reaches right boundary: not found*) RETURN MaxInd END (*IF middle < right*) END (*IF argument = array[middle]*) END Search;

Arrays and Access Formulas Lecture 20


One-dimensional Arrays The easiest way to view a one - dimensional array is as a contiguous block of memory locations of length (# of array elements) (size of each element) Because the size (in bytes) of each element is the same, the compiler can translated A[500] into the address of the record If A points to the first location of of k-byte records, then

This is the access formula for a one-dimensional array. Two-Dimensional Arrays

How does the compiler know where to store element A[i,j] of a twodimensional array? By chopping the matrix into rows, it can be stored like a one- dimensional array: If A points to the first location of A[l1..h1,l2..h2] of k-byte records, then:

Is this access formula for row-major or column-major order, assuming the first index gives the row? For three dimensions, cut the matrix into two dimensional slabs, and use the previous formula. For k-dimensional arrays, we can find a similar formula by induction. Thus we can access any element in a k-dimensional array in O(k) time, which is constant for any reasonably dimension. Fortran stores its arrays in column-major order, while most other languages use row-major order. But why might we really need to know what is going on under the hood? In C language, pointers are usually used to cruise through arrays. Cruising through a 2D array meaningfully requires knowing the order of the elements. Also, in a computer with virtual memory or a cache, it is often faster to access elements if they are close to the last one we have read. Knowing the access function lets us choose the right way to order nested loops.

(*row-major*) (*column-major*)

Do i=1 to n Do j=1 to n

Do j=1 to n

Do i=1 to n

A[i,j] = 0 A[i,j] = 0

Triangular Tables By playing with our own access functions we can build efficient arrays of whatever shape we want, including triangular and banded arrays. Triangular tables prove useful for representing any symmetric function, such as the distance from A to B, D[a,b] = D[b,a]. Thus we can save almost half the memory of a rectangular array by storing it as a triangle The access formula is:

since the identity Faster than Binary Search?

can be proven by induction.

Binary search takes time to find a particular key in a sorted array. It can be shown that, in the worst case, no faster algorithm exists. So how might we do faster? This is not a contradiction. Suppose we wanted to search on a field containing an ID number between 1 and the number of records. Rather than doing a binary search on this field, why not use it as an index in an array! Accessing such an array element is O(1) instead of Interpolation Search Binary search is only optimal when you know nothing about your data except that it is sorted! !

When you look up AAA in the telephone book, you don't start in the middle. We use our understanding of how things are named in the real world to choose where to prove next. Such an algorithm is called an interpolation search, since we are interpolating(guessing) where the key should be. Interpolation search is only as good as our guesses. If we do not understand the data as well as you think, interpolation search can be very slow - recall the Shifflett's of Charlottesville! With interpolation search, the cost of making a good guess might overwhelm the reduction in the number of guesses, so watch out! The Key Ideas on Access Formulas A pointer tells us exactly where in memory an item is. An array reference A[i] lets us quickly calculate exactly where the ith element of A is in memory, knowing only i, the starting location of A, and the size of each array item. Any time we can compute the exact position for an item in memory by a simple access formula, we can find it as quickly as we can compute the formula! Must Array Indices be Integers? We have seen that binary search is slower than table lookup. Why can't the entire world be one big array? One reason is that many of the fields we wish to search on are not integers, for example, names in a telephone book. What address in the machine is defined by ``Skiena''? To compute the appropriate address we need a function to map arbitrary keys to addresses. Such hash functions form the basis of an important search technique, hashing!

Hashing Lecture 21
Hashing

One way to convert form names to integers is to use the letters to form a base ``alphabet-size'' number system: To convert ``STEVE'' to a number, observe that e is the 5th letter of the alphabet, s is the 19th letter, t is the 20th letter, and v is the 22nd letter. Thus ``Steve''

Thus one way we could represent a table of names would be to set aside an array big enough to contain one element for each possible string of letters, then store data in the elements corresponding to real people. By computing this function, it tells us where the person's phone number is immediately!! What's the Problem? Because we must leave room for every possible string, this method will use an incredible amount of memory. We need a data structure to represent a sparse table, one where almost all entries will be empty. We can reduce the number of boxes we need if we are willing to put more than one thing in the same box! Example: suppose we use the base alphabet number system, then take the remainder Now the table is much smaller, but we need a way to deal with the fact that more than one, (but hopefully every few) keys can get mapped to the same array element. The Basics of Hashing The basics of hashing is to apply a function to the search key so we can determine where the item is without looking at the other items. To make the table of reasonable size, we must allow for collisions, two distinct keys mapped to the same location.

We a special hash function to map keys (hopefully uniformly) to integers in a certain range.

We set up an array as big as this range, and use the valve of the function as the index to store the appropriate key.Special care must be taken to handle collisions when they occur.

There are several clever techniques we will see to develop good hash functions and deal with the problems of duplicates. Hash Functions The verb ``hash'' means ``to mix up'', and so we seek a function to mix up keys as well as possible. The best possible hash function would hash m keys into n ``buckets'' with no more than function keys per bucket. Such a function is called a perfect hash

How can we build a hash function? Let us consider hashing character strings to integers. The ORD function returns the character code associated with a given character. By using the ``base character size'' number system, we can map each string to an integer. The First Three SSN digits Hash The first three digits of the Social Security Number The last three digits of the Social Security Number What is the big picture?
1. A hash function which maps an arbitrary key to an integer turns

searching into array access, hence O(1). 2. To use a finite sized array means two different keys will be mapped to the same place. Thus we must have some way to handle collisions. 3. A good hash function must spread the keys uniformly, or else we have a linear search. Ideas for Hash Functions

Truncation - When grades are posted, the last four digits of your SSN are used, because they distribute students more uniformly than the first four digits.

Folding - We should get a better spread by factoring in the entire key. Maybe subtract the last four digits from the first five digits of the SSN, and take the absolute value? Modular Arithmetic - When constructing pseudorandom numbers, a good trick for uniform distribution was to take a big number mod the size of our range. Because of our roulette wheel analogy, the numbers tend to get spread well if the tablesize is selected carefully.

Prime Numbers are Good Things Suppose we wanted to hash check totals by the dollar value in pennies mod 1000. What happens? , , and

Prices tend to be clumped by similar last digits, so we get clustering. If we instead use a prime numbered Modulus like 1007, these clusters will get broken: , , and . In general, it is a good idea to use prime modulus for hash table size, since it is less likely the data will be multiples of large primes as opposed to small primes - all multiples of 4 get mapped to even numbers in an even sized hash table! The Birthday Paradox No matter how good our hash function is, we had better be prepared for collisions, because of the birthday paradox. Assuming 365 days a year, what is the probability that exactly two people share a birthday? Once the first person has fixed their birthday, the second person has 365 possible days to be born to avoid a collision, or a 365/365 chance. With three people, the probability that no two share is . In general, the probability of there beingno collisions after n insertions into an m-element table is

When m = 366, this probability sinks below 1/2 when N = 23 and to almost 0 when .

The moral is that collisions are common, even with good hash functions. What about Collisions? No matter how good our hash functions are, we must deal with collisions. What do we do when the spot in the table we need is occupied?

Put it somewhere else! - In open addressing, we have a rule to decide where to put it if the space is already occupied. Keep a list at each bin! - At each spot in the hash table, keep a linked list of keys sharing this hash value, and do a sequential search to find the one we need. This method is called chaining.

Collision Resolution by Chaining The easiest approach is to let each element in the hash table be a pointer to a list of keys. Insertion, deletion, and query reduce to the problem in linked lists. If the n keys are distributed uniformly in a table of size m/n, each operation takes O(m/n) time. Chaining is easy, but devotes a considerable amount of memory to pointers, which could be used to make the table larger. Still, it is my preferred method. Open Addressing We can dispense with all these pointers by using an implicit reference derived from a simple function: If the space we want to use is filled, we can examine the remaining locations:
1. Sequentially 2. Quadratically 3. Linearly

The reason for using a more complicated scheme is to avoid long runs from similarly hashed keys. Deletion in an open addressing scheme is ugly, since removing one element can break a chain of insertions, making some elements inaccessible. Performance on Set Operations With either chaining or open addressing:

Search - O(1) expected, O(n) worst case. Insert - O(1) expected, O(n) worst case. Delete - O(1) expected, O(n) worst case.

Pragmatically, a hash table is often the best data structure to maintain a dictionary. However, the worst-case running time is unpredictable. The best worst-case bounds on a dictionary come from balanced binary trees, such as red-black trees.

Tree Structures Lecture 21


Trees ``I think that I shall never see a poem as lovely as a tree. Poems are wrote by fools like me, but only G-d can make a tree.'' - Joyce Kilmer We have seen many data structures which allow fast search, but not fast, flexible update. Sorted Tables search, O(n) insertion, O(n) deletion.

Hash Tables - The number of insertions are essentially bounded by the table size, which must be specified in advance. Worst case O(n) search. Binary trees will enable us to search, insert, and delete fast, without predefining the size of our data structure!

How can we get this flexibility? The only data structure we have seen which allows fast insertion/ deletion is the linked list, with updates in O(1) time but search in O(n) time. To get search time, we used binary search, meaning we always had a choice of two next elements to look at. To combine these ideas, we want a ``linked list'' with two pointers per node! This is the basic idea behind search trees! Rooted Trees We can use a recursive definition to specify what we mean by a ``rooted tree''. A rooted tree is either (1) empty, or (2) consists of a node called the root, together with two rooted trees called the left subtree and right subtree of the root. A binary tree is a rooted tree where each node has at most two descendants, the left child and the right child. A binary tree can be implemented where each node has left and right pointer fields, an (optional) parent pointer, and a data field. Rooted trees in Real Life Rooted trees can be used to model corporate heirarchies and family trees. Note the inherently recursive structure of rooted trees. Deleting the root gives rise to a certain number of smaller subtrees. In a rooted tree, the order among ``brother'' nodes matters. Thus left is different from right. The five distinct binary trees with five nodes: Binary Search Trees A binary search tree is a binary tree where each node contains a key such that:

All keys in the left subtree precede the key in the root. All keys in the right subtree succeed the key in the root. The left and right subtrees of the root are again binary search trees.

Left: A binary search tree. Right: A heap but not a binary search tree. For any binary tree on n nodes, and any set of n keys, there is exactly one labeling to make it a binary search tree!! Binary Tree Search Searching a binary tree is almost like binary search! The difference is that instead of searching an array and defining the middle element ourselves, we just follow the appropriate pointer! The type declaration is simply a linked list node with another pointer. Left and right pointers are identical types.
TYPE T = BRANDED REF RECORD key: ElemT; left, right: T := NIL; END; (*T*)

Dictionary search operations are easy in binary trees. The algorithm works because both the left and right subtrees of a binary search tree are binary search trees - recursive structure, recursive algorithm. Search Implementation
PROCEDURE Search(tree: T; e: ElemT): BOOLEAN = (*Searches for an element e in tree. Returns TRUE if present, else FALSE*) BEGIN IF tree = NIL THEN RETURN FALSE (*not found*) ELSIF tree.key = e THEN RETURN TRUE (*found*) ELSIF e < tree.key THEN RETURN Search(tree.left, e) (*search in left tree*) ELSE RETURN Search(tree.right, e) (*search in right tree*) END; (*IF tree...*) END Search;

This takes time proportional to the height of the tree, O(h). Good, balanced trees have height Building Binary Trees , while bad, unbalanced trees have height O(n).

To insert a new node into an existing tree, we search for where it should be, then replace that NIL pointer with a pointer to the new node. Each NIL pointer defines a gap in the space of keys! The pointer in the parent node must be modified to remember where we put the new node. Insertion Routine
PROCEDURE Insert(VAR tree: T; e: ElemT) = BEGIN IF tree = NIL THEN tree:= NEW(T, key:= e); (*insert at proper place*) ELSIF e < tree.key THEN Insert(tree.left, e) (*search place in left tree*) ELSE Insert(tree.right, e) (*search place in right tree*) END; (*IF tree...*) END Insert;

Tree Shapes and Sizes Suppose we have a binary tree with n nodes. How many levels can it have? At least and at most n.

How many pointers are in the tree? There are n nodes in tree, each of which has 2 pointers, for a total of 2n pointers regardless of shape. How many pointers are NIL, i.e ``wasted''? Except for the root, each node in the tree is pointed to by one tree pointer Thus the number of NILs is Traversal of Binary Trees How can we print out all the names in a family tree? An essential component of many algorithms is to completely traverse a tree data structure. The key is to make sure we visit each node exactly once. The order in which we explore each node and its children matters for many applications. , for .

There are six permutations of {left, right, node} which define traversals. The most interesting traversals are inorder {left, node, right}, preorder {node, left, right}, postorder {left, right, node}, Why do we care about different traversals? Depending on what the tree represents, different traversals have different interpretations. An in-order traversals of a binary serach tree sorts the keys! Inorder traversal: 748251396, Preorder traversal: 124785369, Postorder traversal: 784529631 Reverse Polish notation is simply a post order traversal of an expression tree, like the one below for expression 2+3*4+(3*4)/5.
PROCEDURE Traverse(tree: T; action: Action; order := Order.In; direction := Direction.Right) = PROCEDURE PreL(x: T; depth: INTEGER) = BEGIN IF x # NIL THEN action(x.key, depth); PreL(x.left, depth + 1); PreL(x.right, depth + 1); END; (*IF x # NIL*) END PreL; PROCEDURE PreR(x: T; depth: INTEGER) = BEGIN IF x # NIL THEN action(x.key, depth); PreR(x.right, depth + 1); PreR(x.left, depth + 1); END; (*IF x # NIL*) END PreR; PROCEDURE InL(x: T; depth: INTEGER) = BEGIN IF x # NIL THEN InL(x.left, depth + 1); action(x.key, depth); InL(x.right, depth + 1); END; (*IF x # NIL*) END InL; PROCEDURE InR(x: T; depth: INTEGER) = BEGIN IF x # NIL THEN InR(x.right, depth + 1); action(x.key, depth); InR(x.left, depth + 1); END; (*IF x # NIL*)

END InR; PROCEDURE PostL(x: T; depth: INTEGER) = BEGIN IF x # NIL THEN PostL(x.left, depth + 1); PostL(x.right, depth + 1); action(x.key, depth); END; (*IF x # NIL*) END PostL; PROCEDURE PostR(x: T; depth: INTEGER) = BEGIN IF x # NIL THEN PostR(x.right, depth + 1); PostR(x.left, depth + 1); action(x.key, depth); END; (*IF x # NIL*) END PostR; BEGIN (*Traverse*) IF direction = Direction.Left THEN CASE order OF | Order.Pre => PreL(tree, 0); | Order.In => InL(tree, 0); | Order.Post => PostL(tree, 0); END (*CASE order*) ELSE (* direction = Direction.Right*) CASE order OF | Order.Pre => PreR(tree, 0); | Order.In => InR(tree, 0); | Order.Post => PostR(tree, 0); END (*CASE order*) END (*IF direction*) END Traverse;

Deletion from Binary Search Trees Insertion was easy because the new node goes in as a leaf and only its parent is affected. Deletion of a leaf is just as easy - set the parent pointer to NIL. But what if the node to be deleted is an interior node? We have two pointers to connect to only one parent!! Deletion is somewhat more tricky than insertion, because the node to die may not be a leaf, and thus effect other nodes. Case (a), where the node is a leaf, is simple - just NIL out the parents child pointer.

Case (b), where a node has one chld, the doomed node can just be cut out. Case (c), relabel the node as its predecessor (which has at most one child when z has two children!) and delete the predecessor!
PROCEDURE Delete(VAR tree: T; e: ElemT): BOOLEAN = (*Deletes an element e in tree. Returns TRUE if present, else FALSE*) PROCEDURE LeftLargest(VAR x: T) = VAR y: T; BEGIN IF x.right = NIL THEN (*x points to largest element left*) y:= tree; (*y now points to target node*) tree:= x; (*tree assumes the largest node to the left*) x:= x.left; (*Largest node left replaced by its left subtree*) tree.left:= y.left; (*tree assumes subtrees ...*) tree.right:= y.right; (*... of deleted node*) ELSE (*Largest element left not found*) LeftLargest(x.right) (*Continue search to the right*) END; END LeftLargest; BEGIN IF tree = NIL THEN RETURN FALSE ELSIF e < tree.key THEN RETURN Delete(tree.left, e) ELSIF e > tree.key THEN RETURN Delete(tree.right, e) ELSE (*found*) IF tree.left = NIL THEN tree:= tree.right; ELSIF tree.right = NIL THEN tree:= tree.left; ELSE (*Target node has two nonempty subtrees*) LeftLargest(tree.left) (*Search in left subtree*) END; (*IF tree.left...*) RETURN TRUE END; (*IF tree...*) END Delete;

SUNY at Stony BrookMidterm 2 CSE 214 - Data Structures November 21, 1997 Midterm Exam Name: Signature: ID #: Section #: INSTRUCTIONS:

You may use either pen or pencil. Check to see that you have 5 exam pages plus this cover (6 total). Look over all problems before starting work. Your signature above signs the CSE 214 Honor Pledge: ``On my honor as a student I have neither given nor received aid on this exam.'' Think before you write. Good luck!!

1) (20 points) Show the state of the array after each pass by the following sorting routines. You do not have to show the array after every move or comparison, but only after each execution of the main sorting loop or recursive call. Sort in increasing order.
------------------------------------------------------------| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 | -------------------------------------------------------------

(a) Insertion Sort 10 points


------------------------------------------------------------| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 | | 34 \ 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 | | 34 | 125 \ 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 | | 5 | 34 | 125 \ 19 | 87 | 243 | 19 | -3 | 117 | 36 | | 5 | 19 | 34 | 125 \ 87 | 243 | 19 | -3 | 117 | 36 | | 5 | 19 | 34 | 87 | 125\ 243 | 19 | -3 | 117 | 36 | | 5 | 19 | 34 | 87 | 125| 243 \ 19 | -3 | 117 | 36 | | 5 | 19 | 19 | 34 | 87 | 125| 243 \ -3 | 117 | 36 | | -3 | 5 | 19 | 19 | 34 | 87 | 125| 243 \ 117 | 36 | | -3 | 5 | 19 | 19 | 34 | 87 | 117| 125| 243 \ 36 | | -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117| 125| 243 | -------------------------------------------------------------

(b) Quicksort (pivot on rightmost element) 10 points - there are many variants
| -----------------------------------------------------------34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |

| 34 | 5 | 19 | 19 | -3 | 36 | 125 | 87 | 243 | 117 | | -3 | 34 | 5 | 19 | 19 | 36 | 87 | 117 | 125 | 243 | | -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117 | 125 | 243 | | -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117| 125| 243 | -------------------------------------------------------------

2) (25 points) In class, we discussed two different heuristics for self-organizing sequential search. With the move-to-front heuristic, on a query we perform a sequential search for the appropriate element, which when found is moved from its current position to the front of the list. With the move-forward-one heuristic, on a query we perform a sequential search for the appropriate element, which when found is moved one element closer to the front of the list. For both algorithms, if the search key is already at the front of the list, no change occurs. (a) Write a function to implement sequential search in a linked list, with the move-to-front heuristic. You may assume that there are at least two elements in the list and that the item is always found.
PROCEDURE ListSearch (VAR p : pointer) : pointer = var q, head:pointer; head := p; q := p; if (q^.info = key) then return(q); else p := p^.next; while (p^.info # key) do p := p^.next; q := q^.next; end q^.next := p^.next; p^.next := head; head := p; return (p); end

points for searching, 10 points for move to front. (b) Which of these two heuristics is better suited for implementation with arrays? Why? 5 points move-forward-one is better for arrays since it can be done via one swap.

3) (15 points) Assume you have an array with 11 elements that is to be used to store data as an hash table. The hash function computes the number mod 11. Given the following list of insertions to the table:
2 4 13 18 22 31 33 34 42 43 49

Show the resulting table after the insertions for each of the following hashing collision handling methods. a) Show the resulting table after the insertions for chaining. (array of linked lists) 10 points 0 - 22, 33 1 - 34 2 - 2, 13 3 - 4 - 4 5 - 49 6 - 7 - 18 8 - 9 - 31, 42 10 - 43 b) List an advantage and a disadvantage of chaining compared to open addressing 5 points. Advantages - deletion is easier and hash table cannot be filled. Disadvantages - the links use up memory which can go to a bigger hash table. 4) (20 points) Write brief essays answering the following questions. Your answer must fit completely in the space allowed (a) Is f(n) = O(g(n)) if points and . ? Show why or why not.

No! There is no constant such that

(b) Consider the following variant of insertion sort. Instead of using sequential search to find the position of the next element we insert into the sorted array, we use a binary search. We then move the appropriate elements over to create room for the new insertion. What is the worst case number of element comparisons performed using this version of insertion sort on n items (big Oh)? points

(c) What is the worst case number of element movements performed using the above version of insertion sort on n items (big Oh)?

6 points

I took off more points for inconsistancies between the answers... 5) (20 points) The integer square root of integer n (SQRT(n)) is the largest integer x such that . For example, SQRT(8) = 2, while SQRT(9) = 3. Write a Modula-3 function to compute SQRT(n). For full credit, your algorithm should run in time. Partial credit will be given for an algorithm. (Hint: think about the ideas behind binary search)
PROCEDURE sqrt(n : INTEGER):INTEGER = var low, high, mid : integer;

low := 1; high := n; while (high - low) > 1 do mid := (high+low) div 2; if (mid * mid) > n then low := mid+1; else high := mid; end; return (mid); end;

Random Search Trees Lecture 23


How good are Random Trees? Who have seen that binary trees can have heights ranging from How tall are they on average? to n.

By using an intuitive argument, like I did with quicksort. I will convince you a random tree is usually quite close to balanced. The text contains a more rigorous proof, which you should look at. Consider the first insertion into an empty tree. This node becomes the root and never changes. Since in a binary search tree all keys less than the root go in the left subtree, the root acts as a partition or pivot element! Let's say a key is a 'good' pivot element if it is in the center half of the sorted space of keys. Half of the time, our root will be a 'good' pivot element. The next insertion will form the root of a subtree, and will be drawn at random from the items either > root or < root. Again, half the time each insertion will be a 'good' partition of the appropriate subset of keys. The bigger half of a good partition contains at most 3n/4 items. Thus the maximum depth of good splits k is:

so

. on

Doubling the depth to account for bad splits still makes in average!

On average, random search trees are very good - more careful analysis shows the average height after n insertions is . Since this is only 39% more than a perfectly balanced tree. ,

Of course, if we get unlucky and insert keys in sorted order, we are doomed to the worst case performance.

insert(a)

insert(b)

insert(c)

insert(d)

What we want is an insertion/deletion procedure which adjusts the tree a little after each insertion, keeping it close enough to balanced so the maximum height is logarithmic, but flexible enough so we can still update fast! Perfectly Balanced Trees Perfectly balanced trees require a lot of work to maintain: If we insert the key 1, we must move every single node in the tree to rebalance it, taking time.

Therefore, when we talk about "balanced" trees, we mean trees whose height is , so all dictionary operations (insert, delete, search, min/max, time.

successor/predecessor) take

Red-Black trees are binary search trees where each node is assigned a color, where the coloring scheme helps us maintain the height as .

AVL Trees Lecture 24


Steven S. Skiena AVL Trees

An AVL tree is a binary search tree in which the heights of the left and right subtrees of the root differ by at most 1, and the left and right subtrees are again AVL trees. Therefore, we can label each node of an AVL tree with a balance factor as well as a key:

``='' - both subtrees of the node are of equal height ``/'' - the left subtree is one taller than the right subtree `` '' - the right subtree is one taller than the left subtree.

AVL trees are named after their inventors, the Russians G.M. Adel'son-Velshi, and E.M. Laudis in 1962. These are the most unbalanced possible AVL trees with a skew always to the right. By maintaining the balance of each node (i.e. the subtree below it) when we insert a new node, we can easily see whether or not to take action! The balance is more useful than maintaining the height of each node because it is a relative, not absolute measure. Thus we can move subtrees around without affecting their balance, even if they end up at different heights. How good are AVL trees? To find out how bad they can be, we want to find what the minimum number of modes a tree of height h can have. If is a minimum node AVL tree, its left and right subtrees must themselves be minimum node AVL trees of smaller size. Further, they should differ in height by 1 to take advantage of AVL freedom. Counting the root node,

Such trees are called Fibonacci trees and

Thus the worse case AVL tree is almost as good as a random tree - on average it is very close to an optional tree.

Why are Fibonacci trees of logarithmic height? Recall that the Fibonacci numbers are defined , , .

Since we are adding the last two numbers together, we are more than doubling the next-to-last and somewhat less that doubling the last number. In fact, height , so a tree with nodes has

AVL Trees Interface


INTERFACE AVLTree; (*08.07.94. CW, LB*) (* Balanced binary search tree, subtype of "BinaryTree.T" *) IMPORT BinaryTree; TYPE T <: BinaryTree.T; END AVLTree. (*T is a subtype of BinaryTree.T *)

AVL Trees Implementation


MODULE AVLTree EXPORTS AVLTree, AVLTreeRep; (*08.07.94. CW*) (* Implementation of the balanced binary search tree as subtype of "BinaryTree.T". The methods "insert" and "delete" are overwritten to keep the tree balanced when elements are inserted or deleted. The other methods are inhereted from the supertype. *) IMPORT BinaryTree, BinTreeRep; REVEAL T = BinaryTree.T BRANDED OBJECT OVERRIDES delete:= Delete; insert:= Insert; END; PROCEDURE Insert(tree: T; e: REFANY) = PROCEDURE RR (VAR root: BinTreeRep.NodeT) = (*simple rotation right*)

VAR left:= root.left; BEGIN root.left:= left.right; left.right:= root; NARROW(root, NodeT).balance:= 0; root:= left; END RR; PROCEDURE RL (VAR root: BinTreeRep.NodeT) = (*simple rotation left*) VAR right:= root.right; BEGIN root.right:= right.left; right.left:= root; NARROW(root, NodeT).balance:= 0; root:= right; END RL; PROCEDURE RrR (VAR root: BinTreeRep.NodeT) = (*double rotation right*) VAR right:= root.left.right; BEGIN root.left.right:= right.left; right.left:= root.left; IF NARROW(right, NodeT).balance = -1 THEN NARROW(root, NodeT).balance:= +1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(right, NodeT).balance = +1 THEN NARROW(root.left, NodeT).balance:= -1 ELSE NARROW(root.left, NodeT).balance:= 0 END; root.left:= right.right; right.right:= root; root:= right; END RrR; PROCEDURE RrL (VAR root: BinTreeRep.NodeT) = (*double rotation left*) VAR left:= root.right.left; BEGIN root.right.left:= left.right; left.right:= root.right; IF NARROW(left, NodeT).balance = +1 THEN NARROW(root, NodeT).balance:= -1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(left, NodeT).balance = -1 THEN NARROW(root.right, NodeT).balance:= +1 ELSE NARROW(root.right, NodeT).balance:= 0 END; root.right:= left.left; left.left:= root; root:= left; END RrL; PROCEDURE InsertBal(VAR root: BinTreeRep.NodeT; new: REFANY;

VAR bal: BOOLEAN) = BEGIN IF root = NIL THEN root:= NEW(NodeT, info:= new, balance:= 0); ELSIF tree.compare(new, root.info)<0 THEN InsertBal(root.left, new, bal); IF NOT bal THEN (* bal stops recursion*) WITH done=NARROW(root, NodeT).balance DO CASE done OF |+1=> done:= 0; bal:= TRUE; (*insertion ok*) | 0=> done:= -1; (*still balanced, but IF NARROW(root.left, NodeT).balance = -1 THEN RR(root) ELSE RrR(root) END; NARROW(root, NodeT).balance:= 0; bal:= TRUE; (*after rotation tree ok*) END; (*CASE*) END (*WITH*) END (*IF*) ELSE InsertBal(root.right, new, bal); IF NOT bal THEN (* bal is set to stop the recurs. adjustm. of balance *) WITH done=NARROW(root, NodeT).balance DO CASE done OF |-1=> done:= 0; bal:= TRUE; (*insertion ok *) | 0=> done:= +1; (*still balanced, but continue*) |+1=> IF NARROW(root.right, NodeT).balance = +1 THEN RL(root) ELSE RrL(root) END; NARROW(root, NodeT).balance:= 0; bal:= TRUE; (*after rotation tree ok*) END; (*CASE*) END (*WITH*) END (*IF*) END; END InsertBal; VAR balanced:= FALSE; BEGIN (*Insert*) InsertBal(tree.root, e, balanced) END Insert; PROCEDURE Delete(tree: T; e: REFANY): REFANY = PROCEDURE RR (VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = (*simple rotation right*) |-1=>

continue*)

VAR left:= root.left; BEGIN root.left:= left.right; left.right:= root; IF NARROW(left, NodeT).balance = 0 THEN NARROW(root, NodeT).balance:= -1; NARROW(left, NodeT).balance:= +1; bal:= TRUE; ELSE NARROW(root, NodeT).balance:= 0; NARROW(left, NodeT).balance:= 0; (*depth changed: continue*) END; root:= left; END RR; PROCEDURE RL (VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = (*simple rotation left*) VAR right:= root.right; BEGIN root.right:= right.left; right.left:= root; IF NARROW(right, NodeT).balance = 0 THEN NARROW(root, NodeT).balance:= +1; NARROW(right, NodeT).balance:= -1; bal:= TRUE; ELSE NARROW(root, NodeT).balance:= 0; NARROW(right, NodeT).balance:= 0; (*depth changed: continue*) END; root:= right; END RL; PROCEDURE RrR (VAR root: BinTreeRep.NodeT) = (*double rotation right*) VAR right:= root.left.right; BEGIN root.left.right:= right.left; right.left:= root.left; IF NARROW(right, NodeT).balance = -1 THEN NARROW(root, NodeT).balance:= +1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(right, NodeT).balance = +1 THEN NARROW(root.left, NodeT).balance:= -1 ELSE NARROW(root.left, NodeT).balance:= 0 END; root.left:= right.right; right.right:= root; root:= right; NARROW(right, NodeT).balance:= 0; END RrR; PROCEDURE RrL (VAR root: BinTreeRep.NodeT) =

(*double rotation left*) VAR left:= root.right.left; BEGIN root.right.left:= left.right; left.right:= root.right; IF NARROW(left, NodeT).balance = +1 THEN NARROW(root, NodeT).balance:= -1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(left, NodeT).balance = -1 THEN NARROW(root.right, NodeT).balance:= +1 ELSE NARROW(root.right, NodeT).balance:= 0 END; root.right:= left.left; left.left:= root; root:= left; NARROW(left, NodeT).balance:= 0; END RrL; PROCEDURE BalanceLeft(VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = BEGIN WITH done = NARROW(root, NodeT).balance DO CASE done OF |-1=> done:= 0; (*new depth: continue*) | 0=> done:= 1; bal:= TRUE; (*balanced ->ok*) |+1=> (*balancing needed*) IF NARROW(root.right, NodeT).balance >= 0 THEN RL(root, bal) ELSE RrL(root) END END (*CASE*) END (*WITH*) END BalanceLeft; PROCEDURE BalanceRight(VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = BEGIN WITH done = NARROW(root, NodeT).balance DO CASE done OF |+1=> done:= 0; (*new depth: continue*) | 0=> done:= -1; bal:= TRUE; (*balanced ->ok*) |-1=> (*balancing needed*) IF NARROW(root.left, NodeT).balance <= 0 THEN RR(root, bal) ELSE RrR(root) END END (*CASE*) END (*WITH*) END BalanceRight; PROCEDURE DeleteSmallest(VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN): REFANY = VAR deleted: REFANY; BEGIN IF root.left = NIL THEN

deleted:= root.info; root:= root.right; RETURN deleted; ELSE deleted:= DeleteSmallest(root.left, bal); IF NOT bal THEN BalanceLeft(root, bal) END; RETURN deleted; END; END DeleteSmallest; PROCEDURE Delete(VAR root: BinTreeRep.NodeT; elm: REFANY; VAR bal: BOOLEAN): REFANY = VAR deleted: REFANY; BEGIN IF root = NIL THEN RETURN NIL ELSIF tree.compare(root.info, elm)>0 THEN deleted:= Delete(root.left, elm, bal); IF deleted # NIL THEN IF NOT bal THEN BalanceLeft(root, bal) END; RETURN deleted; ELSE RETURN NIL; END ELSIF tree.compare(root.info, elm)<0 THEN deleted:= Delete(root.right, elm, bal); IF deleted # NIL THEN IF NOT bal THEN BalanceRight(root, bal) END; RETURN deleted; ELSE RETURN NIL; END ELSE deleted:= root.info; IF root.left = NIL THEN root:= root.right; ELSIF root.right = NIL THEN root:= root.left; ELSE root.info:= DeleteSmallest(root.right, bal); IF NOT bal THEN BalanceRight(root, bal) END; END; RETURN deleted; END; END Delete; VAR balanced:= FALSE; BEGIN (*Delete*) RETURN Delete(tree.root, e, balanced) END Delete; BEGIN END AVLTree.

Deletion from AVL Trees We have seen that AVL trees are But what about deletion? Don't ask! Actually, you can rebalance an AVL tree in more complicated than insertion. but it is for insertion and query.

We will later study B-trees, where deletion is simpler, so don't worry about the details of deletions form AVL trees.

AVL Trees Lecture 24


AVL Trees An AVL tree is a binary search tree in which the heights of the left and right subtrees of the root differ by at most 1, and the left and right subtrees are again AVL trees. Therefore, we can label each node of an AVL tree with a balance factor as well as a key:

``='' - both subtrees of the node are of equal height ``/'' - the left subtree is one taller than the right subtree `` '' - the right subtree is one taller than the left subtree.

AVL trees are named after their inventors, the Russians G.M. Adel'son-Velshi, and E.M. Laudis in 1962. These are the most unbalanced possible AVL trees with a skew always to the right.

By maintaining the balance of each node (i.e. the subtree below it) when we insert a new node, we can easily see whether or not to take action! The balance is more useful than maintaining the height of each node because it is a relative, not absolute measure. Thus we can move subtrees around without affecting their balance, even if they end up at different heights. How good are AVL trees? To find out how bad they can be, we want to find what the minimum number of modes a tree of height h can have. If is a minimum node AVL tree, its left and right subtrees must themselves be minimum node AVL trees of smaller size. Further, they should differ in height by 1 to take advantage of AVL freedom. Counting the root node,

Such trees are called Fibonacci trees and

Thus the worse case AVL tree is almost as good as a random tree - on average it is very close to an optional tree. Why are Fibonacci trees of logarithmic height? Recall that the Fibonacci numbers are defined , , .

Since we are adding the last two numbers together, we are more than doubling the next-to-last and somewhat less that doubling the last number. In fact, height , so a tree with nodes has

AVL Trees Interface

INTERFACE AVLTree; (*08.07.94. CW, LB*) (* Balanced binary search tree, subtype of "BinaryTree.T" *) IMPORT BinaryTree; TYPE T <: BinaryTree.T; END AVLTree. (*T is a subtype of BinaryTree.T *)

AVL Trees Implementation


MODULE AVLTree EXPORTS AVLTree, AVLTreeRep; (*08.07.94. CW*) (* Implementation of the balanced binary search tree as subtype of "BinaryTree.T". The methods "insert" and "delete" are overwritten to keep the tree balanced when elements are inserted or deleted. The other methods are inhereted from the supertype. *) IMPORT BinaryTree, BinTreeRep; REVEAL T = BinaryTree.T BRANDED OBJECT OVERRIDES delete:= Delete; insert:= Insert; END; PROCEDURE Insert(tree: T; e: REFANY) = PROCEDURE RR (VAR root: BinTreeRep.NodeT) = (*simple rotation right*) VAR left:= root.left; BEGIN root.left:= left.right; left.right:= root; NARROW(root, NodeT).balance:= 0; root:= left; END RR; PROCEDURE RL (VAR root: BinTreeRep.NodeT) = (*simple rotation left*) VAR right:= root.right; BEGIN root.right:= right.left; right.left:= root; NARROW(root, NodeT).balance:= 0; root:= right; END RL; PROCEDURE RrR (VAR root: BinTreeRep.NodeT) = (*double rotation right*) VAR right:= root.left.right; BEGIN root.left.right:= right.left; right.left:= root.left; IF NARROW(right, NodeT).balance = -1

THEN NARROW(root, NodeT).balance:= +1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(right, NodeT).balance = +1 THEN NARROW(root.left, NodeT).balance:= -1 ELSE NARROW(root.left, NodeT).balance:= 0 END; root.left:= right.right; right.right:= root; root:= right; END RrR; PROCEDURE RrL (VAR root: BinTreeRep.NodeT) = (*double rotation left*) VAR left:= root.right.left; BEGIN root.right.left:= left.right; left.right:= root.right; IF NARROW(left, NodeT).balance = +1 THEN NARROW(root, NodeT).balance:= -1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(left, NodeT).balance = -1 THEN NARROW(root.right, NodeT).balance:= +1 ELSE NARROW(root.right, NodeT).balance:= 0 END; root.right:= left.left; left.left:= root; root:= left; END RrL; PROCEDURE InsertBal(VAR root: BinTreeRep.NodeT; new: REFANY; VAR bal: BOOLEAN) = BEGIN IF root = NIL THEN root:= NEW(NodeT, info:= new, balance:= 0); ELSIF tree.compare(new, root.info)<0 THEN InsertBal(root.left, new, bal); IF NOT bal THEN (* bal stops recursion*) WITH done=NARROW(root, NodeT).balance DO CASE done OF |+1=> done:= 0; bal:= TRUE; (*insertion ok*) | 0=> done:= -1; (*still balanced, but IF NARROW(root.left, NodeT).balance = -1 THEN RR(root) ELSE RrR(root) END; NARROW(root, NodeT).balance:= 0; bal:= TRUE; (*after rotation tree ok*) END; (*CASE*) END (*WITH*) END (*IF*) |-1=>

continue*)

ELSE InsertBal(root.right, new, bal); IF NOT bal THEN (* bal is set to stop the recurs. adjustm. of balance *) WITH done=NARROW(root, NodeT).balance DO CASE done OF |-1=> done:= 0; bal:= TRUE; (*insertion ok *) | 0=> done:= +1; (*still balanced, but continue*) |+1=> IF NARROW(root.right, NodeT).balance = +1 THEN RL(root) ELSE RrL(root) END; NARROW(root, NodeT).balance:= 0; bal:= TRUE; (*after rotation tree ok*) END; (*CASE*) END (*WITH*) END (*IF*) END; END InsertBal; VAR balanced:= FALSE; BEGIN (*Insert*) InsertBal(tree.root, e, balanced) END Insert; PROCEDURE Delete(tree: T; e: REFANY): REFANY = PROCEDURE RR (VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = (*simple rotation right*) VAR left:= root.left; BEGIN root.left:= left.right; left.right:= root; IF NARROW(left, NodeT).balance = 0 THEN NARROW(root, NodeT).balance:= -1; NARROW(left, NodeT).balance:= +1; bal:= TRUE; ELSE NARROW(root, NodeT).balance:= 0; NARROW(left, NodeT).balance:= 0; (*depth changed: continue*) END; root:= left; END RR; PROCEDURE RL (VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = (*simple rotation left*) VAR right:= root.right; BEGIN root.right:= right.left; right.left:= root; IF NARROW(right, NodeT).balance = 0 THEN

NARROW(root, NodeT).balance:= +1; NARROW(right, NodeT).balance:= -1; bal:= TRUE; ELSE NARROW(root, NodeT).balance:= 0; NARROW(right, NodeT).balance:= 0; (*depth changed: continue*) END; root:= right; END RL; PROCEDURE RrR (VAR root: BinTreeRep.NodeT) = (*double rotation right*) VAR right:= root.left.right; BEGIN root.left.right:= right.left; right.left:= root.left; IF NARROW(right, NodeT).balance = -1 THEN NARROW(root, NodeT).balance:= +1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(right, NodeT).balance = +1 THEN NARROW(root.left, NodeT).balance:= -1 ELSE NARROW(root.left, NodeT).balance:= 0 END; root.left:= right.right; right.right:= root; root:= right; NARROW(right, NodeT).balance:= 0; END RrR; PROCEDURE RrL (VAR root: BinTreeRep.NodeT) = (*double rotation left*) VAR left:= root.right.left; BEGIN root.right.left:= left.right; left.right:= root.right; IF NARROW(left, NodeT).balance = +1 THEN NARROW(root, NodeT).balance:= -1 ELSE NARROW(root, NodeT).balance:= 0 END; IF NARROW(left, NodeT).balance = -1 THEN NARROW(root.right, NodeT).balance:= +1 ELSE NARROW(root.right, NodeT).balance:= 0 END; root.right:= left.left; left.left:= root; root:= left; NARROW(left, NodeT).balance:= 0; END RrL; PROCEDURE BalanceLeft(VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = BEGIN WITH done = NARROW(root, NodeT).balance DO CASE done OF |-1=> done:= 0; (*new depth: continue*)

| 0=> done:= 1; bal:= TRUE; (*balanced ->ok*) |+1=> (*balancing needed*) IF NARROW(root.right, NodeT).balance >= 0 THEN RL(root, bal) ELSE RrL(root) END END (*CASE*) END (*WITH*) END BalanceLeft; PROCEDURE BalanceRight(VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN) = BEGIN WITH done = NARROW(root, NodeT).balance DO CASE done OF |+1=> done:= 0; (*new depth: continue*) | 0=> done:= -1; bal:= TRUE; (*balanced ->ok*) |-1=> (*balancing needed*) IF NARROW(root.left, NodeT).balance <= 0 THEN RR(root, bal) ELSE RrR(root) END END (*CASE*) END (*WITH*) END BalanceRight; PROCEDURE DeleteSmallest(VAR root: BinTreeRep.NodeT; VAR bal: BOOLEAN): REFANY = VAR deleted: REFANY; BEGIN IF root.left = NIL THEN deleted:= root.info; root:= root.right; RETURN deleted; ELSE deleted:= DeleteSmallest(root.left, bal); IF NOT bal THEN BalanceLeft(root, bal) END; RETURN deleted; END; END DeleteSmallest; PROCEDURE Delete(VAR root: BinTreeRep.NodeT; elm: REFANY; VAR bal: BOOLEAN): REFANY = VAR deleted: REFANY; BEGIN IF root = NIL THEN RETURN NIL ELSIF tree.compare(root.info, elm)>0 THEN deleted:= Delete(root.left, elm, bal); IF deleted # NIL THEN IF NOT bal THEN BalanceLeft(root, bal) END; RETURN deleted; ELSE RETURN NIL; END

ELSIF tree.compare(root.info, elm)<0 THEN deleted:= Delete(root.right, elm, bal); IF deleted # NIL THEN IF NOT bal THEN BalanceRight(root, bal) END; RETURN deleted; ELSE RETURN NIL; END ELSE deleted:= root.info; IF root.left = NIL THEN root:= root.right; ELSIF root.right = NIL THEN root:= root.left; ELSE root.info:= DeleteSmallest(root.right, bal); IF NOT bal THEN BalanceRight(root, bal) END; END; RETURN deleted; END; END Delete; VAR balanced:= FALSE; BEGIN (*Delete*) RETURN Delete(tree.root, e, balanced) END Delete; BEGIN END AVLTree.

Deletion from AVL Trees We have seen that AVL trees are But what about deletion? Don't ask! Actually, you can rebalance an AVL tree in more complicated than insertion. but it is for insertion and query.

We will later study B-trees, where deletion is simpler, so don't worry about the details of deletions form AVL trees.

Splay Trees Lecture 26


What about non-uniform access? AVL/red-black trees give us worst case query and update operations, by keeping a balanced search tree. But when I access with nonuniform probability, a skewed tree might be better:

I call Eve with probability .75 I call Lisa with probability .05 I call Wendy with probability .20

Expected cost of left tree: Expected cost of right tree: In real life, it is difficult to obtain the actual probabilities, and they keep changing. What can we do? Self-organizing Search Trees We can apply our self-organizing heuristics to search trees, as we did with linked lists. Whenever we access a node, we can either:

Move-forward-one (conservative heuristic) Move-to-front (liberal heuristic)

Once again, move-to-front proves better at adjusting to changing distributions. Moving a made to the front of a search tree means making it the root! To get a particular node to the root we can do a sequence of rotations! Splay trees use the move-to-front heuristic on each search / query. Splay Trees

To search or insert into a splay tree, we first perform the operation as if it was a random tree. After it is found or inserted, perform a splay operation to move the given key to the root. A splay operation consists of a sequence of double rotations until the node is within one level of the root, where at most onesingle rotation suffices to finish the job. The choice of which double rotation to do depends upon our relationship to our grandparent - a single rotation is performed only when we have no grandparent! The cases: Splay Tree Example Example: Splay(a) At the conclusion, a is the root and the tree is more balanced. Note that the tree would not have become more balanced had we just used single rotations to promote a to the root, instead of double rotations. How good are Splay Trees? Sleator and Tarjan showed that if the keys are accessed with a uniform distribution, the cost for any sequence of n splay operations is the amortized cost is per operation! , so and .

This is better than expected since there is no probability involved! If we get an expensive splay step (i.e. moving up an non-balanced tree) it meant we did enough cheap operations before this that we can pay for the differences out of our savings! Further, if the distribution is non-uniform, we get amortized costs within a constant factor of the best possible tree! All of this is done without keeping any balance or color information - amazing!

Graphs Lecture 27
Graphs A graph G consists of a set of vertices V together with a set E of vertex pairs or edges. Graphs are important because any binary relation is a graph, so graphs can be used to represent essentially any relationship. Example: A network of roads, with cities as vertices and roads between cities as edges. Example: An electronic circuit, with junctions as vertices as components as edges. To understand many problems, we must think of them in terms of graphs! The Friendship Graph Consider a graph where the vertices are people, and there is an edge between two people if and only if they are friends. This graph is well-defined on any set of people: SUNY SB, New York, or the world. What questions might we ask about the friendship graph?

If I am your friend, does that mean you are my friend? A graph is undirected if (x,y) implies (y,x). Otherwise the graph is directed. The ``heard-of'' graph is directed since countless famous people have never heard of me! The ``had-sex-with'' graph is presumably undirected, since it requires a partner.

Am I my own friend?

An edge of the form (x,x) is said to be a loop. If x is y's friend several times over, that could be modeled usingmultiedges, multiple edges between the same pair of vertices. A graph is said to be simple if it contains no loops and multiple edges.

Am I linked by some chain of friends to the President? A path is a sequence of edges connecting two vertices. Since Mel Brooks is my father's-sister's-husband's cousin, there is a path between me and him!

How close is my link to the President? If I were trying to impress you with how tight I am with Mel Brooks, I would be much better off saying that Uncle Lenny knows him than to go into the details of how connected I am to Uncle Lenny. Thus we are often interested in the shortest path between two nodes.

Is there a path of friends between any two people? A graph is connected if there is a path between any two vertices. A directed graph is strongly connected if there is a directed path between any two vertices.

Who has the most friends? The degree of a vertex is the number of edges adjacent to it.

What is the largest clique? A social clique is a group of mutual friends who all hang around together. A graph theoretic clique is a complete subgraph, where each vertex pair has an edge between them. Cliques are the densest possible subgraphs. Within the friendship graph, we would expect that large cliques correspond to workplaces, neighborhoods, religious organizations, schools, and the like.

How long will it take for my gossip to get back to me? A cycle is a path where the last vertex is adjacent to the first. A cycle in which no vertex repeats (such as 1-2-3-1 verus 1-2-3-2-1) is said to be simple. The shortest cycle in the graph defines its girth, while a

simple cycle which passes through each vertex is said to be a Hamiltonian cycle. Data Structures for Graphs There are two main data structures used to represent graphs. Adjacency MatricesAn adjacency matrix is an iff there is no edge from vertex i to vertex j It takes matrix. matrix, where M[i,j] = 0

time to test if (i,j) is in a graph represented by an adjacency

Can we save space if (1) the graph is undirected? (2) if the graph is sparse? Adjacency ListsAn adjacency list consists of a array of pointers, where the ith element points to a linked list of the edges incident on vertex i. To test if edge (i,j) is in the graph, we search the ith list for j, which takes , where is the degree of the ith vertex.

Note that can be much less than n when the graph is sparse. If necessary, the two copies of each edge can be linked by a pointer to facilitate deletions. Tradeoffs Between Adjacency Lists and Adjacency Matrices

Both representations are very useful and have different properties, although adjacency lists are probably better for most problems.

Traversing a Graph One of the most fundamental graph problems is to traverse every edge and vertex in a graph. Applications include:

Printing out the contents of each edge and vertex. Counting the number of edges. Identifying connected components of a graph.

For efficiency, we must make sure we visit each edge at most twice. For correctness, we must do the traversal in a systematic way so that we don't miss anything. Since a maze is just a graph, such an algorithm must be powerful enough to enable us to get out of an arbitrary maze. Marking Vertices The idea in graph traversal is that we must mark each vertex when we first visit it, and keep track of what have not yet completely explored. For each vertex, we can maintain two flags:

discovered - have we ever encountered this vertex before? completely-explored - have we finished exploring this vertex yet?

We must also maintain a structure containing all the vertices we have discovered but not yet completely explored. Initially, only a single start vertex is considered to be discovered. To completely explore a vertex, we look at each edge going out of it. For each edge which goes to an undiscovered vertex, we mark it discovered and add it to the list of work to do. Note that regardless of what order we fetch the next vertex to explore, each edge is considered exactly twice, when each of its endpoints are explored. Correctness of Graph Traversal Every edge and vertex in the connected component is eventually visited.

Suppose not, ie. there exists a vertex which was unvisited whose neighbor was visited. This neighbor will eventually be explored so we would visit it: Traversal Orders The order we explore the vertices depends upon what kind of data structure is used:

Queue - by storing the vertices in a first-in, first out (FIFO) queue, we explore the oldest unexplored vertices first. Thus our explorations radiate out slowly from the starting vertex, defining a so-called breadth-first search. Stack - by storing the vertices in a last-in, first-out (LIFO) stack, we explore the vertices by lurching along a path, constantly visiting a new neighbor if one is available, and backing up only if we are surrounded by previously discovered vertices. Thus our explorations quickly wander away from our starting point, defining a so-called depth-first search.

The three possible colors of each node reflect if it is unvisited (white), visited but unexplored (grey) or completely explored (black). Breadth-First Search

BFS(G,s)

for each vertex

do

color[u] = white

, ie. the distance from s

p[u] = NIL, ie. the parent in the BFS tree

color[u] = grey

d[s] = 0

p[s] = NIL

while

do

u = head[Q]

for each

do

if color[v] = white then

color[v] = gray

d[v] = d[u] + 1

p[v] = u

enqueue[Q,v]

dequeue[Q]

color[u] = black

Depth-First Search DFS has a neat recursive implementation which eliminates the need to explicitly use a stack. Discovery and final times are sometimes a convenience to maintain.

DFS(G)

for each vertex

do

color[u] = white

parent[u] = nil

time = 0

for each vertex

do

if color[u] = white then DFS-VISIT[u]

Initialize each vertex in the main routine, then do a search from each connected component. BFS must also start from a vertex in each component to completely visit the graph.

DFS-VISIT[u]

color[u] = grey (*u had been white/undiscovered*)

discover[u] = time

time = time+1

for each

do

if color[v] = white then

parent[v] = u

DFS-VISIT(v)

color[u] = black (*now finished with u*)

finish[u] = time time = time+1

Sorting algorithm
From Wikipedia, the free encyclopedia

In computer science, a sorting algorithm is an algorithm that puts elements of a list in a certainorder. The most-used orders are numerical order and lexicographical order. Efficient sorting is important for optimizing the use of other algorithms (such as search and merge algorithms) that require sorted lists to work correctly; it is also often useful for canonicalizing data and for producing human-readable output. More formally, the output must satisfy two conditions:

1. 2.

The output is in nondecreasing order (each element is no smaller than the

previous element according to the desired total order); The output is a permutation, or reordering, of the input.

Since the dawn of computing, the sorting problem has attracted a great deal of research, perhaps due to the complexity of solving it efficiently despite its simple, familiar statement. For example, bubble sort was analyzed as early as 1956.[1] Although many consider it a solved problem, useful new sorting algorithms are still being invented (for example, library sort was first published in 2004). Sorting algorithms are prevalent in introductory computer science classes, where the abundance of algorithms for the problem provides a gentle introduction to a variety of core algorithm concepts, such as big O notation, divide and conquer algorithms, data structures, randomized algorithms, best, worst and average case analysis, time-space tradeoffs, and lower bounds.

Contents
[hide]

o o o o o o o o o o o o o o

1 Classification 1.1 Stability 2 Comparison of algorithms 3 Summaries of popular sorting algorithms 3.1 Bubble sort 3.2 Selection sort 3.3 Insertion sort 3.4 Shell sort 3.5 Comb sort 3.6 Merge sort 3.7 Heapsort 3.8 Quicksort 3.9 Counting sort 3.10 Bucket sort 3.11 Radix sort 3.12 Distribution sort 3.13 Timsort 4 Memory usage patterns and index sorting 5 Inefficient/humorous sorts 6 See also 7 References 8 External links

[edit]Classification
Sorting algorithms used in computer science are often classified by: Computational complexity (worst, average and best behaviour) of element comparisons in

terms of the size of the list (n). For typical sorting algorithms good behavior is O(n log n) and bad behavior is O(n2). (See Big O notation.) Ideal behavior for a sort is O(n), but this is not possible in the average case. Comparison-based sorting algorithms, which evaluate the

elements of the list via an abstract key comparison operation, need at least O(n log n) comparisons for most inputs. Computational complexity of swaps (for "in place" algorithms). Memory usage (and use of other computer resources). In particular, some sorting

algorithms are "in place". Strictly, an in place sort needs only O(1) memory beyond the items being sorted; sometimes O(log(n)) additional memory is considered "in place". Recursion. Some algorithms are either recursive or non-recursive, while others may be

both (e.g., merge sort). Stability: stable sorting algorithms maintain the relative order of records with equal keys

(i.e., values). Whether or not they are a comparison sort. A comparison sort examines the data only by

comparing two elements with a comparison operator. General method: insertion, exchange, selection, merging, etc.. Exchange sorts include

bubble sort and quicksort. Selection sorts include shaker sort and heapsort. Adaptability: Whether or not the presortedness of the input affects the running time.

Algorithms that take this into account are known to be adaptive.

[edit]Stability
Stable sorting algorithms maintain the relative order of records with equal keys. If all keys are different then this distinction is not necessary. But if there are equal keys, then a sorting algorithm is stable if whenever there are two records (let's say R and S) with the same key, and R appears before S in the original list, then R will always appear before S in the sorted list. When equal elements are indistinguishable, such as with integers, or more generally, any data where the entire element is the key, stability is not an issue. However, assume that the following pairs of numbers are to be sorted by their first component: (4, 2) (3, 7) (3, 1) (5, 6)

In this case, two different results are possible, one which maintains the relative order of records with equal keys, and one which does not: (3, 7) (3, 1) (3, 1) (3, 7) (4, 2) (4, 2) (5, 6) (5, 6) (order maintained) (order changed)

Unstable sorting algorithms may change the relative order of records with equal keys, but stable sorting algorithms never do so. Unstable sorting algorithms can be specially implemented to be

stable. One way of doing this is to artificially extend the key comparison, so that comparisons between two objects with otherwise equal keys are decided using the order of the entries in the original data order as a tie-breaker. Remembering this order, however, often involves an additional computational cost. Sorting based on a primary, secondary, tertiary, etc. sort key can be done by any sorting method, taking all sort keys into account in comparisons (in other words, using a single composite sort key). If a sorting method is stable, it is also possible to sort multiple times, each time with one sort key. In that case the keys need to be applied in order of increasing priority. Example: sorting pairs of numbers as above by second, then first component: (4, 2) (3, 1) (3, 1) (3, 7) (4, 2) (3, 7) (3, 1) (5, 6) (4, 2) (5, 6) (original) (3, 7) (after sorting by second component) (5, 6) (after sorting by first component)

On the other hand: (3, 7) (3, 1) (3, 1) (4, 2) (4, 2) (5, 6) (5, 6) (after sorting by first component) (3, 7) (after sorting by second component, order by first component is disrupted).

[edit]Comparison of algorithms
In this table, n is the number of records to be sorted. The columns "Average" and "Worst" give the time complexity in each case, under the assumption that the length of each key is constant, and that therefore all comparisons, swaps, and other needed operations can proceed in constant time. "Memory" denotes the amount of auxiliary storage needed beyond that used by the list itself, under the same assumption. These are all comparison sorts. The run time and the memory of algorithms could be measured using various notations like theta, omega, Big-O, small-o, etc. The memory and the run times below are applicable for all the 5 notations.

Comparison sorts

Name

Best

Averag e

Worst

Mem ory

Stab le

Meth od

Other notes

Quicksort

Quicksort is usually done in place with O(log(n)) stack space. [citation needed] Most implementations are Partitioni unstable, as stable inDepends ng place partitioning is more complex. Navevarian ts use an O(n) space array to store the partition. [citation needed]

Merge sort

Depends; worst case is

Yes

Merging

Used to sort this table in Firefox [2].

In-place Merg e sort

Yes

Implemented in Standard Template Library (STL): [3]; Merging can be implemented as a stable sort based on stable in-place merging: [4]

Heapsort

No

Selection

Insertion sort

Yes

O(n + d), where d is Insertion the number ofinversions

Introsort

No

Partitioni Used ng & in SGI STLimplement Selection ations

Selection sort

No

Selection Stable with O(n) extra space, for example

Comparison sorts

Name

Best

Averag e

Worst

Mem ory

Stab le

Meth od

Other notes

using lists [5]. Used to sort this table in Safari or other Webkit web browser [6].

Timsort

Yes

comparisons when Insertion the data is already & sorted or reverse Merging sorted.

Shell sort

or

Depends on gap sequence; best known is

No

Insertion

Bubble sort

Yes

Exchangi Tiny code size ng

Binary tree sort

Yes

When using a selfInsertion balancing binary search tree

Cycle sort

No

In-place with Insertion theoretically optimal number of writes

Library sort

Yes

Insertion

Patience sorting

No

Finds all the longest Insertion increasing & subsequences within Selection O(n log n)

Comparison sorts

Name

Best

Averag e

Worst

Mem ory

Stab le

Meth od

Other notes

Smoothsort

No

An adaptive sort comparisons when the Selection data is already sorted, and 0 swaps.

Strand sort

Yes

Selection

Tournament sort

Selection

Cocktail sort

Yes

Exchangi ng

Comb sort

No

Exchangi Small code size ng

Gnome sort

Yes

Exchangi Tiny code size ng

Bogosort

No

Luck

Randomly permute the array and check if sorted.

Slowsort

No

Remarkably Selection inefficient sorting algorithm [7]

The following table describes integer sorting algorithms and other sorting algorithms that are notcomparison sorts. As such, they are not limited by a lower bound. Complexities below are in terms of n, the number of items to be sorted, k, the size of each key, and d, the digit size used by the implementation. Many of them are based on the assumption that the key size is

large enough that all entries have unique key values, and hence that n << 2k, where << means "much less than." Non-comparison sorts

Name

Best

Average

Worst

Memory

Stable

n<< 2k

Notes

Pigeonhole sort

Yes

Yes

Bucket sort(uniform keys)

Yes

No

Assumes uniform distribution of elements from the domain in the array.[2]

Bucket sort(integer keys)

r is the range of numbers to be sorted. If r Yes Yes = = then Avg RT


[3]

Counting sort

r is the range of numbers to be sorted. If r Yes Yes = = then Avg RT


[2]

LSD Radix Sort

Yes

No

[3][2]

MSD Radix Sort

Yes

No

Stable version uses an external array of size n to hold all of the bins

Non-comparison sorts

Name

Best

Average

Worst

Memory

Stable

n<< 2k

Notes

MSD Radix Sort

No

No

In-Place. k / d recursion levels, 2d for count array

Spreadsort

No

No

Asymptotics are based on the assumption that n << 2k, but the algorithm does not require this.

The following table describes some sorting algorithms that are impractical for real-life use due to extremely poor performance or a requirement for specialized hardware. Na me Be st Aver age Wo rst Memor y Sta ble Compar ison

Other notes

Bead sort

N/A

N/A

N/A

No

Requires specialized hardware

Simple pancak e sort

No

Yes

Count is number of flips.

Spaghe tti (Poll) sort

Yes

Polling

This A linear-time, analog algorithm for sorting a sequence of items, requiring O(n) stack space, and the sort is stable. This requires parallel processors. Spaghetti sort#Analysis

Sorting networ ks

Yes

No

Requires a custom circuit of size

Additionally, theoretical computer scientists have detailed other sorting algorithms that provide better than time complexity with additional constraints, including:

Han's algorithm, a deterministic algorithm for sorting keys from a domain of finite size, time and space.[4]

taking

Thorup's algorithm, a randomized algorithm for sorting keys from a domain of finite size, time and space.[5] expected time and

taking

An integer sorting algorithm taking


[6]

space.

Algorithms not yet compared above include: Odd-even sort Flashsort Burstsort Postman sort Stooge sort Samplesort Bitonic sorter

[edit]Summaries of popular sorting algorithms


[edit]Bubble sort

A bubble sort, a sorting algorithm that continuously steps through a list, swappingitems until they appear in the correct order.

Main article: Bubble sort

Bubble sort is a simple sorting algorithm. The algorithm starts at the beginning of the data set. It compares the first two elements, and if the first is greater than the second, it swaps them. It continues doing this for each pair of adjacent elements to the end of the data set. It then starts again with the first two elements, repeating until no swaps have occurred on the last pass. This algorithm's average and worst case performance is O(n2), so it is rarely used to sort large, unordered, data sets. Bubble sort can be used to sort a small number of items (where its inefficiency is not a high penalty). Bubble sort may also be efficiently used on a list that is already sorted except for a very small number of elements. For example, if only one element is not in order, bubble sort will take only 2n time. If two elements are not in order, bubble sort will take only at most 3n time...

[edit]Selection sort
Main article: Selection sort Selection sort is an in-place comparison sort. It has O(n2) complexity, making it inefficient on large lists, and generally performs worse than the similar insertion sort. Selection sort is noted for its simplicity, and also has performance advantages over more complicated algorithms in certain situations. The algorithm finds the minimum value, swaps it with the value in the first position, and repeats these steps for the remainder of the list. It does no more than n swaps, and thus is useful where swapping is very expensive.

[edit]Insertion sort
Main article: Insertion sort Insertion sort is a simple sorting algorithm that is relatively efficient for small lists and mostly sorted lists, and often is used as part of more sophisticated algorithms. It works by taking elements from the list one by one and inserting them in their correct position into a new sorted list. In arrays, the new list and the remaining elements can share the array's space, but insertion is expensive, requiring shifting all following elements over by one. Shell sort (see below) is a variant of insertion sort that is more efficient for larger lists.

[edit]Shell sort

A Shell sort, different from bubble sort in that it moves elements numerous positionsswapping

Main article: Shell sort Shell sort was invented by Donald Shell in 1959. It improves upon bubble sort and insertion sort by moving out of order elements more than one position at a time. One implementation can be described as arranging the data sequence in a two-dimensional array and then sorting the columns of the array using insertion sort.

[edit]Comb sort
Main article: Comb sort Comb sort is a relatively simplistic sorting algorithm originally designed by Wlodzimierz Dobosiewicz in 1980. Later it was rediscovered and popularized by Stephen Lacey and Richard Box with a Byte Magazine article published in April 1991. Comb sort improves on bubble sort, and rivals algorithms like Quicksort. The basic idea is to eliminate turtles, or small values near the end of the list, since in a bubble sort these slow the sorting down tremendously. (Rabbits, large values around the beginning of the list, do not pose a problem in bubble sort.)

[edit]Merge sort
Main article: Merge sort Merge sort takes advantage of the ease of merging already sorted lists into a new sorted list. It starts by comparing every two elements (i.e., 1 with 2, then 3 with 4...) and swapping them if the first should come after the second. It then merges each of the resulting lists of two into lists of four, then merges those lists of four, and so on; until at last two lists are merged into the final sorted list. Of the algorithms described here, this is the first that scales well to very large lists, because its worst-case running time is O(n log n). Merge sort has seen a relatively recent surge in popularity for practical implementations, being used for the standard sort routine in the programming languages Perl,[7]Python (as timsort[8]), and Java (also uses timsort as of JDK7[9]), among others. Merge sort has been used in Java at least since 2000 in JDK1.3.[10][11]

[edit]Heapsort
Main article: Heapsort Heapsort is a much more efficient version of selection sort. It also works by determining the largest (or smallest) element of the list, placing that at the end (or beginning) of the list, then continuing with the rest of the list, but accomplishes this task efficiently by using a data structure called a heap, a special type of binary tree. Once the data list has been made into a heap, the root node is guaranteed to be the largest (or smallest) element. When it is removed and placed at the end of the list, the heap is rearranged so the largest element remaining moves to the root. Using the heap, finding the next largest element takes O(log n) time, instead of O(n) for a linear scan as in simple selection sort. This allows Heapsort to run in O(n log n) time, and this is also the worst case complexity.

[edit]Quicksort
Main article: Quicksort Quicksort is a divide and conquer algorithm which relies on a partition operation: to partition an array an element called a pivot is selected. All elements smaller than the pivot are moved before it and all greater elements are moved after it. This can be done efficiently in linear time and in-place. The lesser and greater sublists are then recursively sorted. Efficient implementations of quicksort (with in-place partitioning) are typically unstable sorts and somewhat complex, but are among the fastest sorting algorithms in practice. Together with its modest O(log n) space usage, quicksort is one of the most popular sorting algorithms and is available in many standard programming libraries. The most complex issue in quicksort is choosing a good pivot element; consistently poor choices of pivots can result in drastically slower O(n) performance, if at each step the median is chosen as the pivot then the algorithm works in O(n log n). Finding the median however, is an O(n) operation on unsorted lists and therefore exacts its own penalty with sorting.

[edit]Counting sort
Main article: Counting sort Counting sort is applicable when each input is known to belong to a particular set, S, of possibilities. The algorithm runs in O(|S| + n) time and O(|S|) memory where n is the length of the input. It works by creating an integer array of size |S| and using the ith bin to count the occurrences of the ith member ofS in the input. Each input is then counted by incrementing the value of its corresponding bin. Afterward, the counting array is looped through to arrange all of the inputs in order. This sorting algorithm cannot often be used because S needs to be reasonably small for it to be efficient, but the algorithm is extremely fast and demonstrates great asymptotic behavior as n increases. It also can be modified to provide stable behavior.

[edit]Bucket sort
Main article: Bucket sort Bucket sort is a divide and conquer sorting algorithm that generalizes Counting sort by partitioning an array into a finite number of buckets. Each bucket is then sorted individually, either using a different sorting algorithm, or by recursively applying the bucket sorting algorithm. A variation of this method called the single buffered count sort is faster than quicksort.[citation needed] Due to the fact that bucket sort must use a limited number of buckets it is best suited to be used on data sets of a limited scope. Bucket sort would be unsuitable for data such as social security numbers - which have a lot of variation.

[edit]Radix sort
Main article: Radix sort Radix sort is an algorithm that sorts numbers by processing individual digits. n numbers consisting ofk digits each are sorted in O(n k) time. Radix sort can process digits of each number either starting from the least significant digit (LSD) or starting from the most significant digit (MSD). The LSD algorithm first sorts the list by the least significant digit while preserving their relative order using a stable sort. Then it sorts them by the next digit, and so on from the least significant to the most significant, ending up with a sorted list. While the LSD radix sort requires the use of a stable sort, the MSD radix sort algorithm does not (unless stable sorting is desired). In-place MSD radix sort is not stable. It is common for the counting sort algorithm to be used internally by the radix sort. Hybrid sorting approach, such as using insertion sort for small bins improves performance of radix sort significantly.

[edit]Distribution sort
Distribution sort refers to any sorting algorithm where data is distributed from its input to multiple intermediate structures which are then gathered and placed on the output. See Bucket sort, Flashsort.

[edit]Timsort
Main article: Timsort Timsort finds runs in the data, creates runs with insertion sort if necessary, and then uses merge sort to create the final sorted list. It has the same complexity (O(nlogn)) in the average and worst cases, but with pre-sorted data it goes down to O(n).

[edit]Memory usage patterns and index sorting


When the size of the array to be sorted approaches or exceeds the available primary memory, so that (much slower) disk or swap space must be employed, the memory usage pattern of a sorting

algorithm becomes important, and an algorithm that might have been fairly efficient when the array fit easily in RAM may become impractical. In this scenario, the total number of comparisons becomes (relatively) less important, and the number of times sections of memory must be copied or swapped to and from the disk can dominate the performance characteristics of an algorithm. Thus, the number of passes and the localization of comparisons can be more important than the raw number of comparisons, since comparisons of nearby elements to one another happen at system bus speed (or, with caching, even at CPU speed), which, compared to disk speed, is virtually instantaneous. For example, the popular recursive quicksort algorithm provides quite reasonable performance with adequate RAM, but due to the recursive way that it copies portions of the array it becomes much less practical when the array does not fit in RAM, because it may cause a number of slow copy or move operations to and from disk. In that scenario, another algorithm may be preferable even if it requires more total comparisons. One way to work around this problem, which works well when complex records (such as in a relational database) are being sorted by a relatively small key field, is to create an index into the array and then sort the index, rather than the entire array. (A sorted version of the entire array can then be produced with one pass, reading from the index, but often even that is unnecessary, as having the sorted index is adequate.) Because the index is much smaller than the entire array, it may fit easily in memory where the entire array would not, effectively eliminating the disk-swapping problem. This procedure is sometimes called "tag sort".[12] Another technique for overcoming the memory-size problem is to combine two algorithms in a way that takes advantages of the strength of each to improve overall performance. For instance, the array might be subdivided into chunks of a size that will fit easily in RAM (say, a few thousand elements), the chunks sorted using an efficient algorithm (such as quicksort or heapsort), and the results merged as per mergesort. This is less efficient than just doing mergesort in the first place, but it requires less physical RAM (to be practical) than a full quicksort on the whole array. Techniques can also be combined. For sorting very large sets of data that vastly exceed system memory, even the index may need to be sorted using an algorithm or combination of algorithms designed to perform reasonably with virtual memory, i.e., to reduce the amount of swapping required.

[edit]Inefficient/humorous sorts
These are algorithms that are extremely slow compared to those discussed above Bogosort , Stooge sort .

[edit]See also
External sorting Sorting networks (compare) Cocktail sort Collation Schwartzian transform Shuffling algorithms Search algorithms

[edit]References
This article includes a list of references, but its sources remain unclear because it has insufficient inline citations. Please help to improve this article by introducing more precise citations. (September
2009)

1. 2. 3.

^ Demuth, H. Electronic Data Sorting. PhD thesis, Stanford University, 1956. ^ a b c Cormen, Thomas H.; Leiserson, Charles E., Rivest, Ronald L., Stein, Clifford (2001)

[1990].Introduction to Algorithms (2nd ed.). MIT Press and McGraw-Hill. ISBN 0-262-03293-7. ^ a b Goodrich, Michael T.; Tamassia, Roberto (2002). "4.5 Bucket-Sort and Radix-

Sort". Algorithm Design: Foundations, Analysis, and Internet Examples. John Wiley & Sons. pp. 241243.

4.

^ Y. Han. Deterministic sorting in

time and linear space.

Proceedings of the thirty-fourth annual ACM symposium on Theory of computing, Montreal, Quebec, Canada, 2002,p.602-608.

5.

^ M. Thorup. Randomized Sorting in

Time and Linear Space Using

Addition, Shift, and Bit-wise Boolean Operations. Journal of Algorithms, Volume 42, Number 2, February 2002, pp. 205-230(26)

6.

^ Han, Y. and Thorup, M. 2002. Integer Sorting in

Expected

Time and Linear Space. In Proceedings of the 43rd Symposium on Foundations of Computer Science (November 1619, 2002). FOCS. IEEE Computer Society, Washington, DC, 135-144.

7. 8. 9. 10.

^ Perl sort documentation ^ Tim Peters's original description of timsort ^ [1] ^ Merge sort in Java 1.3, Sun.

11. 12.

^ Java 1.3 live since 2000 ^ Definition of "tag sort" according to PC Magazine

D. E. Knuth, The Art of Computer Programming, Volume 3: Sorting and Searching.

[edit]External links
The Wikibook Algorithm implementation has a page on the topic of Sorting algorithms The Wikibook A-level Mathematics has a page on the topic of Sorting algorithms

Sorting Algorithm Animations - Graphical illustration of how different algorithms handle

different kinds of data sets. Sequential and parallel sorting algorithms - Explanations and analyses of many sorting

algorithms. Dictionary of Algorithms, Data Structures, and Problems - Dictionary of algorithms,

techniques, common functions, and problems. Slightly Skeptical View on Sorting Algorithms Discusses several classic algorithms and

promotes alternatives to the quicksort algorithm.


[hide]

Sorting algorithms
Theory

Computational complexity theory

Big O notation

Total order

Lists

Stability Comparison sort

Adaptive sort

Sorting network

Integer sorting

Bubble sort

Cocktail sort

Oddeven sort

Exchange sorts

Comb sort

Gnome sort

Quicksort


Selection sorts

Stooge sort Bogosort

Selection sort

Heapsort

Smoothsort

Cartesian tree sort

Tournament sort

Cycle sort

Insertion sort

Insertion sorts

Shellsort

Tree sort

Library sort


Merge sorts

Patience sorting

Merge sort

Polyphase merge sort

Strand sort

American flag sort

Bead sort

Bucket sort

Distribution sorts

Burstsort

Counting sort

Pigeonhole sort Proxmap sort

Radix sort

Flashsort


Concurrent sorts

Bitonic sorter

Batcher oddeven mergesort

Pairwise sorting network

Timsort

Hybrid sorts

Introsort

Spreadsort

UnShuffle sort

JSort

Bogosort

Other

Topological sorting

Pancake sorting

Spaghetti sort

You might also like