Compiler Notes - Ullman

Compiler Construction Lecture Notes
Introduction
o Lecture 1 (printable)
Lexical Analysis
Syntax Analysis
Semantic Analysis
Intermediate Code
Generation
Final Code Generation
lecture #1 began here

Why study compilers?
Most CS students do not go on to write a commercial compiler
someday, but that's not why we study compilers. We study
compiler construction for the following reasons:
Writing a compiler gives experience with large-scale
applications development. Your compiler program may be
the largest program you write as a student. Experience
working with really big data structures and complex
interactions between algorithms will help you out on your

next big programming project.
Compiler writing is one of the shining triumphs of CS
theory. It demonstrates the value of theory over the impulse
to just "hack up" a solution.
Compiler writing is a basic element of programming
language research. Many language researchers write
compilers for the languages they design.
Many applications have similar properties to one or more
phases of a compiler, and compiler expertise and tools can
help an application programmer working on other projects
besides compilers.
CS 370 is labor intensive. Famous computer scientist Dan Berry
of the University of Waterloo has argued convincingly that there
is no software development method for writing large programs
that doesn't involve pain: pain is inevitable in software
development (Berry's Theorem). From my own experience as a
student, I posulate Jeffery's Corollary: there is no way to learn
the skills necessary for writing big programs without pain. A
good CS course includes pain, and teaches pain management
and minimization.
The questions we should ask, then, are: (a) should CS majors be
required to spend a lot of time becoming really good
programmers? and (b) are we providing students with the
assistance and access to the tools and information they need to
accomplish their goals with the minimal doses of inevitable pain
that are required?
Some Tools we will use
Labs and lectures will discuss all of these, but if you do not
know them already, the sooner you go learn them, the better.
C and "make".
If you are not expert with these yet, you will be a lot closer
by the time you pass this class.
lex and yacc
These are compiler-writers tools, but they are useful for
other kinds of applications, almost anything with a complex
file format to read in can benefit from them.
gdb
If you do not know a source-level debugger well, start
learning. You will need one to survive this class.
e-mail
Regularly e-mailing your instructor is a crucial part of class
participation. If you aren't asking questions, you aren't
doing your job as a student.
web
This is where you get your lecture notes, homeworks, and
labs, and turnin all your work.
virtual environment
We have a 3D video game / chat tool available that can
help us handle questions when one of us is not on campus.
Compilers - What Are They and What Kinds of Compilers
are Out There?
The purpose of a compiler is: to translate a program in some
language (the source language) into a lower-level language
(the target language). The compiler itself is written in some
language, called the implementation language. To write a
compiler you have to be very good at programming in the
implementation language, and have to think about and

understand the source language and target language.
There are several major kinds of compilers:
Native Code Compiler
Translates source code into hardware (assembly or machine
code) instructions. Example: gcc.
Virtual Machine Compiler
Translates source code into an abstract machine code, for
execution by a virtual machine interpreter. Example: javac.
JIT Compiler
Translates virtual machine code to native code. Operates
within a virtual machine. Example: Sun's HotSpot java
machine.
Preprocessor
Translates source code into simpler or slightly lower level
source code, for compilation by another compiler.
Examples: cpp, m4.
Pure interpreter
Executes source code on the fly, without generating
machine code. Example: Lisp.
Phases of a Compiler
Lexical Analysis:
Converts a sequence of characters into words, or tokens
Syntax Analysis:
Converts a sequence of tokens into a parse tree
Semantic Analysis:
Manipulates parse tree to verify symbol and type
information
Intermediate Code Generation:

Converts parse tree into a sequence of intermediate code
instructions
Optimization:
Manipulates intermediate code to produce a more efficient
program
Final Code Generation:
Translates intermediate code into final (machine/assembly)
code
Example of the Compilation Process
Consider the example statement; its translation to machine code
illustrates some of the issues involved in compiling.
position = initial + rate * 60
30 or so characters, from a single line of source code, are first
transformed by lexical analysis into a sequence of 7 tokens.
Those tokens are then used to build a tree of height 4 during
syntax analysis. Semantic analysis may transform the tree into
one of height 5, that includes a type conversion necessary for
real addition on an integer operand. Intermediate code
generation uses a simple traversal algorithm to linearize the tree
back into a sequence of machine-independent three-addresscode instructions.
t1 = inttoreal(60)
t2 = id3 * t1
t3 = id2 + t2
id1 = t3
Optimization of the intermediate code allows the four

instructions to be reduced to two machine-independent
instructions. Final code generation might implement these two
instructions using 5 machine instructions, in which the actual
registers and addressing modes of the CPU are utilized.
MOVF id3, R2
MULF #60.0, R2
MOVF id2, R1
ADDF R2, R1
MOVF R1, id1
Announcements
Reading!
I hope you have already been reading! Make sure you read the
class lecture notes, the related sections of the text, and please
ask questions about whatever is not totally clear. You can Ask
Questions in class, via e-mail, in the virtual environment, or on
the class message board.
Note: although last year's CS 370 lecture notes are ALL
available to you up front, I generally revise each lecture's notes,
making additions, corrections and adaptations to this year's
homeworks, the night before each lecture. The best time to print
hard copies of the lecture notes is one day at a time, right before
the lecture is given.
Overview of Lexical Analysis
A lexical analyzer, also called a scanner, typically has the

following functionality and characteristics.
Its primary function is to convert from a (often very long)
sequence of characters into a (much shorter, perhaps 10X
shorter) sequence of tokens. This means less work for
subsequent phases of the compiler.
The scanner must Identify and Categorize specific
character sequences into tokens. It must know whether
every two adjacent characters in the file belong together in
the same token, or whether the second character must be in
a different token.
Most lexical analyzers discard comments & whitespace. In
most languages these characters serve to separate tokens
from each other, but once lexical analysis is completed they
serve no purpose. On the other hand, the exact line # and/or
column # may be useful in reporting errors, so some record
of what whitespace has occurred may be retained. Note: in
some languages, even popular ones, whitespace is
significant.
Handle lexical errors (illegal characters, malformed tokens)
by reporting them intelligibly to the user.
Efficiency is crucial; a scanner may perform elaborate input
buffering
Token categories can be (precisely, formally) specified
using regular expressions, e.g.
IDENTIFIER=[a-zA-Z][a-zA-Z0-9]*
Lexical Analyzers can be written by hand, or implemented
automatically using finite automata.
What is a "token" ?
In compilers, a "token" is:

1. a single word of source code input (a.k.a. "lexeme")
2. an integer code that refers to a single word of input
3. a set of lexical attributes computed from a single word of
input
Programmers think about all this in terms of #1. Syntax
checking uses #2. Error reporting, semantic analysis, and code
generation require #3. In a compiler written in C, for each token
you allocate a C struct to store (3) for each token.
Worth Mentioning
Here are the names of several important tools closely related to
compilers. You should learn those of these terms that you don't
already know.
interpreter
a language processor program that translates and executes
source code directly, without compiling it ot machine code.
assembler
a translator from human readable (ASCII text) files of
machine instructions into the actual binary code (object
files) of a machine.
linker
a program that combines (multiple) object files to make an
executable. Converts names of variables and functions to
numbers (machine addresses).
loader
Program to load code. On some systems, different
executables start at different base addresses, so the loader
must patch the executable with the actual base address of
the executable.
preprocessor
Program that processes the source code before the compiler
sees it. Usually, it implements macro expansion, but it can
do much more.
editor
Editors may operate on plain text, or they may be wired
into the rest of the compiler, highlighting syntax errors as
you go, or allowing you to insert or delete entire syntax
constructs at a time.
debugger
Program to help you see what's going on when your
program runs. Can print the values of variables, show what
procedure called what procedure to get where you are, run
up to a particular line, run until a particular variable gets a
special value, etc.
profiler
Program to help you see where your program is spending
its time, so you can tell where you need to speed it up.
Auxiliary data structures
You were presented with the phases of the compiler, from
lexical and syntax analysis, through semantic analysis, and
intermediate and final code generation. Each phase has an input
and an output to the next phase. But there are a few data
structures we will build that survive across multiple phases: the
literal table, the symbol table, and the error handler.
lexeme table
a table that stores lexeme values, such as strings and
variable names, that may occur in many places. Only one
copy of each unique string and name needs to be allocated

in memory.
symbol table
a table that stores the names defined (and visible with) each
particular scope. Scopes include: global, and procedure
(local). More advanced languages have more scopes such
as class (or record) and package.
error handler
errors in lexical, syntax, or semantic analysis all need a
common reporting mechanism, that shows where the error
occurred (filename, line number, and maybe column
number are useful).
Reading Named Files in C using stdio
In this class you are opening and reading files. Hopefully this is
review for you; if not, you will need to learn it quickly. To do
any "standard I/O" file processing, you start by including the
header:
#include <stdio.h>
This defines a data type (FILE *) and gives prototypes for
relevant functions. The following code opens a file using a
string filename, reads the first character (into an int variable, not
a char, so that it can detect end-of-file; EOF is not a legal char
value).
FILE *f = fopen(filename, "r");
int i = fgetc(f);
if (i == EOF) /* empty file... */
Command line argument handling and file processing in C
The following example is from Kernighan & Ritchie's "The C

Programming Language", page 162.
#include <stdio.h>
/* cat: concatenate files, version 1 */
int main(int argc, char *argv[])
{
FILE *fp;
void filecopy(FILE *, FILE *);
if (argc == 1)
filecopy(stdin, stdout);
else
while (--argc > 0)
if ((fp = fopen(*++argv, "r")) == NULL) {
printf("cat: can't open %s\n", *argv);
return 1;
}
else {
filecopy(fp, stdout);
fclose(fp);
}
return 0;
}
void filecopy(FILE *ifp, FILE *ofp)
{
int c;
while ((c = getc(ifp)) != EOF)
putc(c, ofp);
}
Warning: while using and adapting the above code is fair game
in this class, the yylex() function is very different than the
filecopy() function! It takes no parameters! It returns an integer
every time it finds a token! So if you "borrow" from this
example, delete filecopy() and write yylex() from scratch.
Multiple students have fallen into this trap before you.
A Brief Introduction to Make
It is not a good idea to write a large program like a compiler as a
single source file. For one thing, every time you make a small
change, you would need to recompile the whole program, which
will end up being many thousands of lines. For another thing,
parts of your compiler may be generated by "compiler
construction tools" which will write separate files. In any case,
this class will require you to use multiple source files, compiled
separately, and linked together to form your executable program.
This would be a pain, except we have "make" which takes care
of it for us. Make uses an input file named "makefile", which
stores in ASCII text form a collection of rules for how to build a
program from its pieces. Each rule shows how to build a file
from its source files, or dependencies. For example, to compile a
file under C:
foo.o : foo.c
gcc -c foo.c
The first line says to build foo.o you need foo.c, and the second
line, which must being with a tab, gave a command-line to
execute whenever foo.o should be rebuilt, i.e. when it is missing

or when foo.c has been changed and need to be recompiled.
The first rule in the makefile is what "make" builds by default,
but note that make dependencies are recursive: before it checks
whether it needs to rebuild foo.o from foo.c it will check
whether foo.c needs to be rebuilt using some other rule. Because
of this post-order traversal of the "dependency graph", the first
rule in your makefile is usually the last one that executes when
you type "make". For a C program, the first rule in your
makefile would usually be the "link" step that assembles objects
files into an executable as in:
compiler: foo.o bar.o baz.o
gcc -o compiler foo.o bar.o baz.o
There is a lot more to "make" but we will take it one step at a
time. This article on Make may be useful to you. You can find
other useful on-line documentation on "make" (manual page,
Internet reference guides, etc) if you look.
A couple finer points for HW#1
extern vs. #include: when do you use the one, when the other?
public interface to yylex(): no, you can't add your own
parameters
Regular Expressions
The notation we use to precisely capture all the variations that a
given category of token may take are called "regular
expressions" (or, less formally, "patterns". The word "pattern" is
really vague and there are lots of other notations for patterns
besides regular expressions). Regular expressions are a

shorthand notation for sets of strings. In order to even talk about
"strings" you have to first define an alphabet, the set of
characters which can appear.
1. Epsilon () is a regular expression denoting the set
containing the empty string
2. Any letter in the alphabet is also a regular expression
denoting the set containing a one-letter string consisting of
that letter.
3. For regular expressions r and s,
r|s
is a regular expression denoting the union of r and s
4. For regular expressions r and s,
rs
is a regular expression denoting the set of strings consisting
of a member of r followed by a member of s
5. For regular expression r,
r*
of zero or more occurrences of r.
6. You can parenthesize a regular expression to specify
operator precedence (otherwise, alternation is like plus,
concatenation is like times, and closure is like
exponentiation)
Although these operators are sufficient to describe all regular
languages, in practice everybody uses extensions:
For regular expression r,
r+
of one or more occurrences of r. Equivalent to rr*
For regular expression r,

r?
of zero or one occurrence of r. Equivalent to r|
The notation [abc] is short for a|b|c. [a-z] is short for
a|b|...|z. [^abc] is short for: any character other than a, b, or
c.

What is a "lexical attribute" ?
A lexical attribute is a piece of information about a token. These
typically include:
category
an integer code used to check syntax
lexeme
actual string contents of the token
line, column, file where the lexeme occurs in source code
value
for literals, the binary data they represent
Homework #2
Avoid These Common Bugs in Your Homeworks!
1. yytext or yyinput were not declared global
2. main() does not have its required argc, argv parameters!
3. main() does not call yylex() in a loop or check its return
value
4. getc() EOF handling is missing or wrong! check EVERY
all to getc() for EOF!
5. opened files not (all) closed! file handle leak!
6. end-of-comment code doesn't check for */
7. yylex() is not doing the file reading
8. yylex() does not skip multiple spaces, mishandles spaces at

the front of input, or requires certain spaces in order to
function OK
9. extra or bogus output not in assignment spec
10.
= instead of ==
Some Regular Expression Examples
In a previous lecture we saw regular expressions, the preferred
notation for specifying patterns of characters that define token
categories. The best way to get a feel for regular expressions is
to see examples. Note that regular expressions form the basis for
pattern matching in many UNIX tools such as grep, awk, perl,
etc.
What is the regular expression for each of the different lexical
items that appear in C programs? How does this compare with
another, possibly simpler programming language such as
BASIC?
lexical
category
operators
BASIC
the characters
themselves
C
For operators that are regular
expression operators we need
mark them with double quotes or
backslashes to indicate you mean
the character, not the regular
expression operator. Note several
operators have a common prefix.
The lexical analyzer needs to look
ahead to tell whether an = is an
assignment, or is followed by
another = for example.

Reserved words are also matched
the concatenation
reserved
by the regular expression for
of characters;
words
identifiers, so a disambiguating
case insensitive
rule is needed.
no _; $ at ends of
some; 2
identifiers significant
[a-zA-Z_][a-zA-Z0-9]*
letters!?; case
insensitive
ints and reals,
numbers starting with [0- 0x[0-9a-fA-F]+ etc.
9]+
comments REM.*
C's comments are tricky regexp's
almost ".*"; no
strings
escaped quotes
escapes
what else?
lex(1) and flex(1)
These programs generally take a lexical specification given in a
.l file and create a corresponding C language lexical analyzer in
a file named lex.yy.c. The lexical analyzer is then linked with
the rest of your compiler.
The C code generated by lex has the following public interface.
Note the use of global variables instead of parameters, and the
use of the prefix yy to distinguish scanner names from your
program names. This prefix is also used in the YACC parser
generator.
FILE *yyin; /* set this variable prior to calling yylex() */

int yylex(); /* call this function once for each token */
char yytext[]; /* yylex() writes the token's lexeme to an array */
/* note: with flex, I believe extern declarations must
read
extern char *yytext;
*/
int yywrap(); /* called by lex when it hits end-of-file; see
below */
The .l file format consists of a mixture of lex syntax and C code
fragments. The percent sign (%) is used to signify lex elements.
The whole file is divided into three sections separated by %%:
header
%%
body
%%
helper functions
The header consists of C code fragments enclosed in %{ and %}
as well as macro definitions consisting of a name and a regular
expression denoted by that name. lex macros are invoked
explicitly by enclosing the macro name in curly braces.
Following are some example lex macros.
letter
digit
ident
[a-zA-Z]
[0-9]
{letter}({letter}|{digit})*
The body consists of of a sequence of regular expressions for

different token categories and other lexical entities. Each regular
expression can have a C code fragment enclosed in curly braces
that executes when that regular expression is matched. For most
of the regular expressions this code fragment (also called
a semantic action consists of returning an integer that identifies
the token category to the rest of the compiler, particularly for
use by the parser to check syntax. Some typical regular
expressions and semantic actions might include:
""
{ /* no-op, discard whitespace */ }
{ident}
{ return IDENTIFIER; }
"*"
{ return ASTERISK; }
"."
{ return PERIOD; }
You also need regular expressions for lexical errors such as
unterminated character constants, or illegal characters.
The helper functions in a lex file typically compute lexical
attributes, such as the actual integer or string values denoted by
literals. One helper function you have to write is yywrap(),
which is called when lex hits end of file. If you just want lex to
quit, have yywrap() return 1. If your yywrap() switches yyin to a
different file and you want lex to continue processing, have
yywrap() return 0. The lex or flex library (-ll or -lfl) have default
yywrap() function which return a 1, and flex has the
directive %option noyywrap which allows you to skip writing
this function.
A Short Comment on Lexing C Reals
C float and double constants have to have at least one digit,

either before or after the required decimal. This is a pain:
([0-9]+.[0-9]* | [0-9]*.[0-9]+) ...
You might almost be happier if you wrote
([0-9]*.[0-9]*) { return (strcmp(yytext,".")) ? REAL :
PERIOD; }
You-all know C's ternary e1 ? e2 : e3 operator, don't ya? Its an
if-then-else expression, very slick.
Lex extended regular expressions
Lex further extends the regular expressions with several helpful
operators. Lex's regular expressions include:
c
normal characters mean themselves
\c
backslash escapes remove the meaning from most operator
characters. Inside character sets and quotes, backslash
performs C-style escapes.
"s"
Double quotes mean to match the C string given as itself.
This is particularly useful for multi-byte operators and may
be more readable than using backslash multiple times.
[s]
This character set operator matches any one character
among those in s.
[^s]
A negated-set matches any one character not among those
in s.
.
The dot operator matches any one character except
newline: [^\n]
r*
match r 0 or more times.
r+
match r 1 or more times.
r?
match r 0 or 1 time.
r{m,n}
match r between m and n times.
r1r2
concatenation. match r1 followed by r2
r1|r2
alternation. match r1 or r2
(r)
parentheses specify precedence but do not match anything
r1/r2
lookahead. match r1 when r2 follows, without consuming r2
^r
match r only when it occurs at the beginning of a line
r$
match r only when it occurs at the end of a line
Announcements
Next homework I promise: I will ask the TA to run your
program with a nonexistent file as a command-line argument!
Lexical Attributes and Token Objects
Besides the token's category, the rest of the compiler may need
several pieces of information about a token in order to perform
semantic analysis, code generation, and error handling. These
are stored in an object instance of class Token, or in C, a struct.
The fields are generally something like:
struct token {
int category;
char *text;
int linenumber;
int column;
char *filename;
union literal value;
}
The union literal will hold computed values of integers, real
numbers, and strings. In your homework assignment, I am
requiring you to compute column #'s; not all compilers require
them, but they are easy. Also: in our compiler project we are not
worrying about optimizing our use of memory, so am not
requiring you to use a union.
Flex Manpage Examplefest
To read a UNIX "man page", or manual page, you type
"man command" where command is the UNIX program or
library function you need information on. Read the man page for
man to learn more advanced uses ("man man").
It turns out the flex man page is intended to be pretty complete,
enough so that we can draw our examples from it. Perhaps what
you should figure out from these examples is that flex is
actually... flexible. The first several examples use flex as a filter

from standard input to standard output.
sneaky string removal tool:

%%
"zap me"
excess whitespace trimmer
%%
[ \t]+
putchar( ' ' );
[ \t]+$
/* ignore this token */
sneaky string substitution tool:
%%
username printf( "%s", getlogin() );
Line Counter/Word Counter
int num_lines = 0, num_chars = 0;

%%
\n ++num_lines; ++num_chars;
.
++num_chars;
%%
main()
{
yylex();
printf( "# of lines = %d, # of chars =
%d\n",
num_lines, num_chars );
}
Toy compiler example
/* scanner for a toy Pascal-like language */

%{
/* need this for the call to atof() below */
#include <math.h>
%}
DIGIT [0-9]
ID
[a-z][a-z0-9]*
%%
{DIGIT}+ {
printf( "An integer: %s (%d)\n",
yytext,
atoi( yytext ) );
}
{DIGIT}+"."{DIGIT}*
{
printf( "A float: %s (%g)\n", yytext,
atof( yytext ) );
}
if|then|begin|end|procedure|function
{
printf( "A keyword: %s\n", yytext );
}
{ID}
printf( "An identifier: %s\n",
yytext );
"+"|"-"|"*"|"/" printf( "An operator: %s\n",

yytext );
"{"[^}\n]*"}"
/* eat up one-line
comments */
[ \t\n]+
.
%s\n", yytext );
/* eat up whitespace */
printf( "Unrecognized character:
%%
main( argc, argv )
int argc;
char **argv;
{
++argv, --argc; /* skip over program
name */
if ( argc > 0 )
yyin = fopen( argv[0], "r" );
else
yyin = stdin;
yylex();
}
On the use of character sets (square brackets) in lex and

similar tools
A student recently sent me an example regular expression for
comments that read:
COMMENT [/*][[^*/]*[*]*]]*[*/]
One problem here is that square brackets are not parentheses,
they do not nest, they do not support concatenation or other
regular expression operators. They mean exactly: "match any
one of these characters" or for ^: "match any one character that
is not one of these characters". Note also that you can't use ^ as
a "not" operator outside of square brackets: you can't write the
expression for "stuff that isn't */" by saying (^ "*/")
Finite Automata
A finite automaton (FA) is an abstract, mathematical machine,
also known as a finite state machine, with the following
components:
1. A set of states S
2. A set of input symbols E (the alphabet)
3. A transition function move(state, symbol) : new state(s)
4. A start state S0
5. A set of final states F
The word finite refers to the set of states: there is a fixed size to
this machine. No "stacks", no "virtual memory", just a known
number of states. The word automaton refers to the execution
mode: there is no instruction set, there is no sequence of
instructions, there is just a hardwired short loop that executes the

same instruction over and over:
while ((c=getchar()) != EOF) S := move(S, c);
DFAs
The type of finite automata that is easiest to understand and
simplest to implement (say, even in hardware) is called a
deterministic finite automaton (DFA). The
word deterministic here refers to the return value of function
move(state, symbol), which goes to at most one state. Example:
S = {s0, s1, s2}
E = {a, b, c}
move = { (s0,a):s1; (s1,b):s2; (s2,c):s2 }
S0 = s0
F = {s2}
Finite automata correspond in a 1:1 relationship to transition
diagrams; from any transition diagram one can write down the
formal automaton in terms of items #1-#5 above, and vice versa.
To draw the transition diagram for a finite automaton:
draw a circle for each state s in S; put a label inside the

circles to identify each state by number or name
draw an arrow between Si and Sj, labeled with x whenever
the transition says to move(Si, x) : Sj
draw a "wedgie" into the start state S0 to identify it
draw a second circle inside each of the final states in F
The Automaton Game
If I give you a transition diagram of a finite automaton, you can

hand-simulate the operation of that automaton on any input I
give you.
DFA Implementation
The nice part about DFA's is that they are efficiently
implemented on computers. What DFA does the following code
correspond to? What is the corresponding regular expression?
You can speed this code fragment up even further if you are
willing to use goto's or write it in assembler.
state := S0
for(;;)
switch (state) {
case 0:
switch (input) {
'a': state = 1; input = getchar(); break;
'b': input = getchar(); break;
default: printf("dfa error\n"); exit(1);
}
case 1:
switch (input) {
EOF: printf("accept\n"); exit(0);
default: printf("dfa error\n"); exit(1);
}
}
Deterministic Finite Automata Examples
A lexical analyzer might associate different final states with
different token categories:
C Comments:
Nondeterministic Finite Automata (NFA's)

Notational convenience motivates more flexible machines in
which function move() can go to more than one state on a given
input symbol, and some states can move to other states even
without consuming an input symbol (-transitions).
Fortunately, one can prove that for any NFA, there is an
equivalent DFA. They are just a notational convenience. So,
finite automata help us get from a set of regular expressions to a
computer program that recognizes them efficiently.
NFA Examples
-transitions make it simpler to merge automata:
multiple transitions on the same symbol handle common

prefixes:
factoring may optimize the number of states. Is this picture

OK/correct?
C Pointers, malloc, and your future

For most of you success as a computer scientist may boil down
to whether you can master the concept of dynamically allocated
memory. In C this means pointers and the malloc() family of
functions. Here are some tips:
Draw "memory box" pictures of your variables. Pencil and
paper understanding of memory leads to correct running
programs.
Always initialize local pointer variables. Consider this
code:
void f() {
int i = 0;
struct tokenlist *current, *head;
...
foo(current)
}
Here, current is passed in as a parameter to foo, but it is a
pointer that hasn't been pointed at anything. I cannot tell
you how many times I personally have written bugs myself
or fixed bugs in student code, caused by reading or writing
to pointers that weren't pointing at anything in particular.
Local variables that weren't initialized point at random
garbage. If you are lucky this is a coredump, but you might
not be lucky, you might not find out where the mistake
was, you might just get a wrong answer. This can all be
fixed by
struct tokenlist *current = NULL, *head = NULL;
Avoid this common C bug:
struct token *t = (struct token
*)malloc(sizeof(struct token *)));
This compiles, but causes coredumps during program
execution. Why?
Check your malloc() return value to be sure it is not NULL.
Sure, modern programs will "never run out of memory".
Wrong. malloc() can return NULL even on big machines.
Operating systems often place limits on memory so as to
protect themselves from runaway programs or hacker
attacks.
Regular expression examples

Can you draw an NFA corresponding to the following?
(a|c)*b(a|c)*
(a|c)*|(a|c)*b(a|c)*
(a|c)*(b|)(a|c)*
Regular expressions can be converted automatically to
NFA's
Each rule in the definition of regular expressions has a
corresponding NFA; NFA's are composed using transitions.
This is called "Thompson's construction" ). We will work
examples such as (a|b)*abb in class and during lab.
1. For , draw two states with a single transition.
2. For any letter in the alphabet, draw two states with a single
transition labeled with that letter.
3. For regular expressions r and s, draw r | s by adding a new

start state with transitions to the start states of r and s, and
a new final state with transitions from each final state in r
and s.
4. For regular expressions r and s, draw rs by adding

transitions from the final states of r to the start state of s.
5. For regular expression r, draw r* by adding new start and

final states, and transitions
o from the start state to the final state,
o from the final state back to the start state,
o from the new start to the old start and from the old
final states to the new final state.
6. For parenthesized regular expression (r) you can use the

NFA for r.
NFA's can be converted automatically to DFA's
In: NFA N
Out: DFA D
Method: Construct transition table Dtran (a.k.a. the "move
function"). Each DFA state is a set of NFA states. Dtran
simulates in parallel all possible moves N can make on a given
string.
Operations to keep track of sets of NFA states:
_closure(s)
set of states reachable from state s via
_closure(T)
set of states reachable from any state in set T via
move(T,a)
set of states to which there is an NFA transition from states
in T on symbol a
NFA to DFA Algorithm:

Dstates := {_closure(start_state)}
while T := unmarked_member(Dstates) do {
mark(T)
for each input symbol a do {
U := _closure(move(T,a))
if not member(Dstates, U) then
insert(Dstates, U)
Dtran[T,a] := U
}
}
Practice converting NFA to DFA
OK, you've seen the algorithm, now can you use it?
...
...did you get:
OK, how about this one:

Some Remarks
I have a collection of compiler textbooks in my office,

which I will make avaliable as "loaners" from class period
to class period, all you have to do is sign a return contract
in blood.
If you checked out the class web page, you saw a solution
to HW#1 was posted awhile ago... I will try to do this for
future assignments also, but not immediately, so as to allow

students a few days of lateness without a heavy penalty.
Whether we return the same or a different category for
integer constants and for line numbers depends very much
on the grammar we use to parse our language.
Lexical Analysis and the Literal Table

In many compilers, the memory management components of the
compiler interact with several phases of compilation, starting
with lexical analysis.
Efficient storage is necessary to handle large input files.
There is a colossal amount of duplication in lexical data:
variable names, strings and other literal values duplicate
frequently
What token type to use may depend on previous
declarations.
A hash table or other efficient data structure can avoid this
duplication. The software engineering design pattern to use is
called the "flyweight".
Major Data Structures in a Compiler
token
contains an integer category, lexeme, line #, column #,
filename... We could build these into a link list, but instead
we'll use them as leaves in a tree structure.
syntax tree
contains grammar information about a sequence of related
tokens. leaves contain lexical information (tokens). internal
nodes contain grammar rules and pointers to tokens or

other tree nodes.
symbol table
contains variable names, types, and information needed to
generate code for a name (such as its address, or constant
value). Look ups are by name, so we'll need a hash table.
intermediate & final code
We'll need link lists or similar structures to hold sequences
of machine instructions
Literal Table: Usage Example
Example abbreviated from [ASU86]: Figure 3.18, p. 109. Use
"install_id()" instead of "strdup()" to avoid duplication in the
lexical data.
%{
/* #define's for token categories LT, LE, etc.
%}
white [ \t\n]+
digit [0-9]
id
[a-zA-Z_][a-zA-Z_0-9]*
num {digit}+(\.{digit}+)?
%%
{ws} { /* discard */ }
if
{ return IF; }
then { return THEN; }
else { return ELSE; }
{id} { yylval.id = install_id(); return ID; }

{num} { yylval.num = install_num(); return NUMBER; }
"<" { yylval.op = LT; return RELOP; }
">" { yylval.op = GT; return RELOP; }
%%
install_id()
{
/* insert yytext into the literal table */
}
install_num()
{
/* insert (binary number corresponding to?) yytext into the
literal table */
}
So how would you implement a literal table using a hash table?
We will see more hash tables when it comes time to construct
the symbol tables with which variable names and scopes are
managed, so you had better become fluent.
Constructing your Token inside yylex()
A student recently asked if it was OK to allocate a token
structure inside main() after yylex() returns the token. This is not
OK because in the next phase of your compiler, you are not
calling yylex(), the automatically generated parser will call
yylex(). There is a way for the parser to grab your token if
you've stored it in a global variable, but there is not a way for

the parser to build the token structure itself.
Syntax Analysis
Parsing is the act of performing syntax analysis to verify an
input program's compliance with the source language. A byproduct of this process is typically a tree that represents the
structure of the program.
Context Free Grammars
A context free grammar G has:
A set of terminal symbols, T
A set of nonterminal symbols, N
A start symbol, s, which is a member of N
A set of production rules of the form A -> w, where A is a
nonterminal and w is a string of terminal and nonterminal
symbols.
A context free grammar can be used to generate strings in the
corresponding language as follows:
let X = the start symbol s
while there is some nonterminal Y in X do
apply any one production rule using Y, e.g. Y -> w
When X consists only of terminal symbols, it is a string of the
language denoted by the grammar. Each iteration of the loop is
a derivation step. If an iteration has several nonterminals to
choose from at some point, the rules of derviation would allow
any of these to be applied. In practice, parsing algorithms tend to
always choose the leftmost nonterminal, or the rightmost
nonterminal, resulting in strings that are leftmost
derivations or rightmost derivations.
Context Free Grammar Examples

Well, OK, so how much of the C language grammar can we
come up with in class today? Start with expressions, work on up
to statements, and work there up to entire functions, and
programs.
Dr. Pontelli is looking for a web developer, did everyone see
that ad? I too am looking for student research assistants.
Grammar Ambiguity
The grammar
E -> E + E
E -> E * E
E -> ( E )
E -> ident
allows two different derivations for strings such as "x + y * z".
The grammar is ambiguous, but the semantics of the language
dictate a particular operator precedence that should be used. One
way to eliminate such ambiguity is to rewrite the grammar. For
example, we can force the precedence we want by adding some
nonterminals and production rules.
E -> E + T
E -> T
T -> T * F
T -> F
F -> ( E )
F -> ident
Given the arithmetic expression grammar from last lecture:
How can a program figure that x + y * z is legal?
How can a program figure out that x + y (* z) is illegal?
A brief aside on casting your mallocs
If you don't put a prototype for malloc(), C thinks it returns an
int.
#include <stdlib.h>
includes prototypes for malloc(), free(), etc. malloc() returns a
void *.
void * means "pointer that points at nothing", or "pointer that
points at anything". You need to cast it to what you are really
pointing at, as in:
union lexval *l = (union lexval *)malloc(sizeof(union lexval));
Note the stupid duplication of type information; no language is
perfect! Anyhow, always cast your mallocs. The program may
work without the cast, but you need to fix every warning, so you
don't accidentally let a serious one through.
Recursive Descent Parsing
Perhaps the simplest parsing method, for a large subset of
context free grammars, is called recursive descent. It is simple
because the algorithm closely follows the production rules of
nonterminal symbols.
Write 1 procedure per nonterminal rule
Within each procedure, a) match terminals at appropriate

positions, and b) call procedures for non-terminals.
Pitfalls:
1. left recursion is FATAL
2. must distinguish between several production rules, or
potentially, one has to try all of them via backtracking.
Recursive Descent Parsing Example #1

Consider the grammar we gave above. There will be functions
for E, T, and F. The function for F() is the "easiest" in some
sense: based on a single token it can decide which production
rule to use. The parsing functions return 0 (failed to parse) if the
nonterminal in question cannot be derived from the tokens at the
current point. A nonzero return value of N would indicate
success in parsing using production rule #N.
int F()
{
int t = yylex();
if (t == IDENT) return 6;
else if (t == LP) {
if (E() && (yylex()==RP) return 5;
}
return 0;
}
Comment #1: if F() is in the middle of a larger parse of E() or
T(), F() may succeed, but the subsequent parsing may fail. The
parse may have to backtrack, which would mean we'd have to
be able to put tokens back for later parsing. Add a memory (say,
a gigantic array or link list for example) of already-parsed
tokens to the lexical analyzer, plus backtracking logic to E() or

T() as needed. The call to F() may get repeated following a
different production rule for a higher nonterminal.
Comment #2: in a real compiler we need more than "yes it
parsed" or "no it didn't": we need a parse tree if it succeeds, and
we need a useful error message if it didn't.
Question: for E() and T(), how do we know which production
rule to try? Option A: just blindly try each one in turn. Option B:
look at the first (current) token, only try those rules that start
with that token (1 character lookahead). If you are lucky, that
one character will uniquely select a production rule. If that is
always true through the whole grammar, no backtracking is
needed.
Question: how do we know which rules start with whatever
token we are looking at? Can anyone suggest a solution, or are
we stuck?
Announcements
Homework #3 minor extension

Midterm exam: Thursday March 16
The first midterm exam will cover lexical analysis and
syntax analysis
Removing Left Recursion

E -> E + T | T
T -> T * F | F
F -> ( E ) | ident
We can remove the left recursion by introducing new
nonterminals and new production rules.
E -> T E'
E' -> + T E' |
T -> F T'
T' -> * F T' |
F -> ( E ) | ident
Getting rid of such immediate left recursion is not enough, one
must get rid of indirect left recursion, where two or more
nonterminals are mutually left-recursive. One can
rewrite any CFG to remove left recursion (Algorithm 4.1).
for i := 1 to n do
for j := 1 to i-1 do begin
replace each Ai -> Aj gamma with productions
Ai -> delta1gamma | delta2gamma
end
eliminate immediate left recursion
Removing Left Recursion, part 2
Left recursion can be broken into three cases
case 1: trivial
A:A|
The recursion must always terminate by A finally deriving so
you can rewrite it to the equivalent
A : A'
A' : A' |
Example:
E : E op T | T
can be rewritten
E : T E'
E' : op T E' |
case 2: non-trivial, but immediate
In the more general case, there may be multiple recursive
productions and/or multiple non-recursive productions.
A : A 1 | A 2 | ... | 1 | 2
As in the trivial case, you get rid of left-recursing A and
introduce an A'
A : 1 A' | 2 A' | ...
A' : 1 A' | 2 A' | ... |
case 3: mutual recursion
1. Order the nonterminals in some order 1 to N.
2. Rewrite production rules to eliminate all nonterminals in
leftmost positions that refer to a "previous" nonterminal.
When finished, all productions' right hand symbols start
with a terminal or a nonterminal that is numbered equal or
higher than the nonterminal no the left hand side.
3. Eliminate the direct left recusion as per cases 1-2.
Left Recursion Versus Right Recursion: When does it

Matter?
A student came to me once with what they described as an
operator precedence problem where 5-4+3 was computing the
wrong value (-2 instead of 4). What it really was, was an
associativity problem due to the grammar:
E:T+E|T-E|T
The problem here is that right recursion is forcing right
associativity, but normal arithmetic requires left associativity.
Several solutions are: (a) rewrite the grammar to be left
recursive, or (b) rewrite the grammar with more nonterminals to
force the correct precedence/associativity, or (c) if using YACC
or Bison, there are "cheat codes" we will discuss later to allow it
to be majorly ambiguous and specify associativity separately
(look for %left and %right in YACC manuals).
Recursive Descent Parsing Example #2
The grammar
S -> A B C
A -> a A
A ->
B -> b
C -> c
maps to pseudocode like the following. (:= is an assignment
operator)
procedure S()
if A() & B() & C() then succeed # matched S, we win
end
procedure A()
if yychar == a then { # use production 2
yychar := scan()
return A()
}
else
succeed # production rule 3, match
end
procedure B()
if yychar == b then {
yychar := scan()
succeed
}
else fail
end
procedure C()
if yychar == c then {
yychar := scan()
succeed
}
else fail
end
Backtracking?
Could your current token begin more than one of your possible
production rules? Try all of them, remember and reset state for
each try.
S -> cAd
A -> ab
A -> a
Left factoring can often solve such problems:
S -> cAd
A -> a A'
A'-> b
A'-> ()
One can also perform left factoring to reduce or eliminate the
lookahead or backtracking needed to tell which production rule
to use. If the end result has no lookahead or backtracking
needed, the resulting CFG can be solved by a "predictive parser"
and coded easily in a conventional language. If backtracking is
needed, a recursive descent parser takes more work to
implement, but is still feasible. As a more concrete example:
S -> if E then S
S -> if E then S1 else S2
can be factored to:
S -> if E then S S'
S'-> else S2 |
Some More Parsing Theory
Automatic techniques for constructing parsers start with

computing some basic functions for symbols in the grammar.
These functions are useful in understanding both recursive
descent and bottom-up LR parsers.
First(a)
First(a) is the set of terminals that begin strings derived from a,
which can include .
1. First(X) starts with the empty set.
2. if X is a terminal, First(X) is {X}.
3. if X -> is a production, add to First(X).
4. if X is a non-terminal and X -> Y1 Y2 ... Yk is a production,
add First(Y1) to First(X).
5.
for (i = 1; if Yi can derive ; i++)
6.
add First(Yi+1) to First(X)
7.
First(a) examples
by the way, this stuff is all in section 4.3 in your text.
Last time we looked at an example with E, T, and F, and + and
*. The first-set computation was not too exciting and we need
more examples.
stmt : if-stmt | OTHER
if-stmt: IF LP expr RP stmt else-part
else-part: ELSE stmt |
expr: IDENT | INTLIT
What are the First() sets of each nonterminal?
Follow(A)
Follow(A) for nonterminal A is the set of terminals that can
appear immediately to the right of A in some sentential form S > aAxB... To compute Follow, apply these rules to all
nonterminals in the grammar:
1. Add $ to Follow(S)
2. if A -> aBb then add First(b) - to Follow(B)
3. if A -> aB or A -> aBb where is in First(b), then add
Follow(A) to Follow(B).
On resizing arrays in C
The sval attribute in homework #2 is a perfect example of a
problem which a BCS major might not be expected to manage,
but a CS major should be able to do by the time they graduate.
This is not to encourage any of you to consider BCS, but rather,
to encourage you to learn how to solve problems like these.
The problem can be summarized as: step through yytext,
copying each piece out to sval, removing doublequotes and
plusses between the pieces, and evaluating CHR$() constants.
Space allocated with malloc() can be increased in size by
realloc(). realloc() is awesome. But, it COPIES and MOVES the
old chunk of space you had to the new, resized chunk of space,
and frees the old space, so you had better not have any other
pointers pointing at that space if you realloc(), and you have to
update your pointer to point at the new location realloc() returns.
i = 0; j = 0;
while (yytext[i] != '\0') {
if (yytext[i] == '\"') {
/* copy string into sval */
i++;
while (yytext[i] != '\"') {
sval[j++] = yytext[i++];
}
}
else if ((yytext[i] == 'C') || (yytext[i] == 'c')) {
/* handle CHR$(...) */
i += 5;
k = atoi(yytext + i);
sval[j++] = k;
/* might check for 0-255 */
while (yytext[i] != ')') i++;
}
/* else we can just skip it */
i++;
}
sval[j] = '\0'; /* NUL-terminate our string */
There is one more problem: how do we allocate memory for
sval, and how big should it be?
Solution #1: sval = malloc(strlen(yytext)+1) is very safe,
but wastes space.
Solution #2: you could malloc a small amount and grow the
array as needed.
sval = strdup("");
...
sval = appendstring(sval, yytext[i]); /* instead of

sval[j++] = yytext[i] */
where the function appendstring could be:
char *appendstring(char *s, char c)

{
i = strlen(s);
s = realloc(s, i+2);
s[i] = c;
s[i+1] = '\0';
return s;
}
Note: it is very inefficient to grow your array one character
at a time; in real life people grow arrays in large chunks at
a time.
Solution #3: use solution one and then shrink your array
when you find out how big it actually needs to be.
sval = malloc(strlen(yytext)+1);
/* ... do the code copying into sval; be sure to
NUL-terminate */
sval = realloc(sval, strlen(sval)+1);
YACC
YACC ("yet another compiler compiler") is a popular tool

which originated at
AT&T Bell Labs. YACC takes a context free grammar as input,
and generates a
parser as output. Several independent, compatible
implementations (AT&T
yacc, Berkeley yacc, GNU Bison) for C exist, as well as many
implementations
for other popular languages.
YACC files end in .y and take the form

declarations
%%
grammar
%%
subroutines
The declarations section defines the terminal symbols (tokens)
and
nonterminal symbols. The most useful declarations are:
%token a
declares terminal symbol a; YACC can generate a set of
#define's
that map these symbols onto integers, in a y.tab.h
file. Note: don't
#include your y.tab.h file from your grammar .y file,
YACC generates the
same definitions and declarations directly in the .c file,

and including
the .tab.h file will cause duplication errors.
%start A
specifies the start symbol for the grammar (defaults to
nonterminal
on left side of the first production rule).
The grammar gives the production rules, interspersed with

program code
fragments called semantic actions that let the programmer do
what's
desired when the grammar productions are reduced. They
follow the
syntax
A : body ;
Where body is a sequence of 0 or more terminals, nonterminals,
or semantic
actions (code, in curly braces) separated by spaces. As a
notational
convenience, multiple production rules may be grouped together
using the
vertical bar (|).
Bottom Up Parsing
Bottom up parsers start from the sequence of terminal symbols

and work
their way back up to the start symbol by repeatedly replacing
grammar
rules' right hand sides by the corresponding non-terminal. This
is
the reverse of the derivation process, and is called "reduction".
Example. For the grammar

(1)
(2)
(3)
(4)
S->aABe
A->Abc
A->b
B->d
the string "abbcde" can be parsed bottom-up by the following

reduction
steps:
abbcde
aAbcde
aAde
aABe
S
Handles
Definition: a handle is a substring that

1.
matches a right hand side of a production rule in the

grammar and
2.
3.
whose reduction to the nonterminal on the left hand

side of that
4.
grammar rule is a step along the reverse of a
rightmost derivation.
5.
Shift Reduce Parsing
A shift-reduce parser performs its parsing using the following

structure
Stack
$
Input
w$
At each step, the parser performs one of the following actions.

1.
Shift one symbol from the input onto the parse stack
2.
3.
Reduce one handle on the top of the parse stack. The

symbols
4.
from the right hand side of a grammar rule are
popped of the
5.
stack, and the nonterminal symbol is pushed on
the stack in their place.
6.
7.
Accept is the operation performed when the start
symbol is alone
8.
on the parse stack and the input is empty.
9.
10.
Error actions occur when no successful parse is
possible.
11.
The YACC Value Stack
YACC's parse stack contains only "states"

YACC maintains a parallel set of values
$ is used in semantic actions to name elements on
the value stack
$$ denotes the value associated with the LHS

(nonterminal) symbol
$n denotes the value associated with RHS symbol
at position n.
Value stack typically used to construct the parse

tree
Typical rule with semantic action: A : b C d { $$ =

tree(R,3,$1,$2,$3); }
The default value stack is an array of integers
The value stack can hold arbitrary values in an
array of unions
The union type is declared with %union and is
named YYSTYPE
Getting Lex and Yacc to talk
YACC uses a global variable named yylval, of type YYSTYPE,

to receive
lexical information from the scanner. Whatever is in this
variable
each time yylex() returns to the parser will get copied over to the
top of the value stack when the token is shifted onto the parse
stack.
You can either declare that struct token may appear in the
%union,
and put a mixture of struct node and struct token on the value
stack,
or you can allocate a "leaf" tree node, and point it at your struct
token. Or you can use a tree type that allows tokens to include
their lexical information directly in the tree nodes. If you have
more than one %union type possible, be prepared to see type
conflicts
and to declare the types of all your nonterminals.
Getting all this straight takes some time; you can plan on
it. Your best
bet is to draw pictures of how you want the trees to look, and
then make the
code match the pictures. No pictures == "Dr. J will ask to see
your
pictures and not be able to help if you can't describe your trees."
Declaring value stack types for terminal and nonterminal
symbols
Unless you are going to use the default (integer) value stack, you
will
have to declare the types of the elements on the value
stack. Actually,
you do this by declaring which
union member is to be used for each terminal and nonterminal in
the
grammar.
Example: in the cocogram.y that I gave you we could add a
%union declaration
with a union member named treenode:
%union {
nodeptr treenode;
}
This will produce a compile error if you haven't declared a

nodeptr type
using a typedef, but that is another story. To declare that a
nonterminal
uses this union member, write something like:
%type < treenode > function_definition
Terminal symbols use %token to perform the corresponding

declaration.
If you had a second %union member (say struct token *tokenptr)
you
might write:
%token < tokenptr > SEMICOL
Announcements
Having trouble debugging your grammar? "bison -v"

generates a .output
file that gives the gory details of conflicts and such.
Announcements
In honor of Dr. Jeffery's 10th anniversary, a minor extension

in Homework #3.
Conflicts in Shift-Reduce Parsing
"Conflicts" occur when an ambiguity in the grammar creates a

situation
where the parser does not know which step to perform at a given
point
during parsing. There are two kinds of conflicts that occur.
shift-reduce
a shift reduce conflict occurs when the grammar indicates
that
different successful parses might occur with either a
shift or a reduce
at a given point during parsing. The vast majority of
situations where
this conflict occurs can be correctly resolved by shifting.
reduce-reduce
a reduce reduce conflict occurs when the parser has two or
more
handles at the same time on the top of the
stack. Whatever choice
the parser makes is just as likely to be wrong as not. In
this case
it is usually best to rewrite the grammar to eliminate the
conflict,
possibly by factoring.
Example shift reduce conflict:

S->if E then S
S->if E then S else S
In many languages two nested "if" statements produce a

situation where
an "else" clause could legally belong to either "if". The usual
rule
(to shift) attaches the else to the nearest (i.e. inner) if statement.
Example reduce reduce conflict:

(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
S -> id LP plist RP
S -> E GETS E
plist -> plist, p
plist -> p
p -> id
E -> id LP elist RP
E -> id
elist -> elist, E
elist -> E
By the point the stack holds ...id LP id
the parser will not know which rule to use to reduce the id: (5)
or (7).
Further Discussion of Reduce Reduce and Shift Reduce
Conflicts
The following grammar, based loosely on our expression

grammar from
last time, illustrates a reduce reduce conflict, and how you have
to
exercise care when using epsilon productions. Epsilon
productions
were helpful for some of the grammar rewriting methods, such
as removing
left recursion, but used indiscriminately, they can cause much
trouble.
T : F | F T2 ;
T2 : p F T2 | ;
F:lTr|v;
The reduce-reduce conflict occurs after you have seen an F. If

the next
symbol is a p there is no question of what to do, but if the next
symbol
is the end of file, do you reduce by rule #1 or #4 ?
A slightly different grammar is needed to demonstrate a shiftreduce conflict:
T : F g;
T : F T2 g;
T2 : t F T2 ;
T2 : ;
F:lTr;
F:v;
This grammar is not much different than before, and has the
same problem,
but the surrounding context (the "calling environments") of F
cause the
grammar to have a shift-reduce instead of reduce-reduce. Once
again,
the trouble is after you have seen an F and dwells on the
question of
whether to reduce the epsilon production, or instead to shift,
upon
seeing a token g.
The .output file generated by "bison -v" explains these conflicts

in
considerable detail. Part of what you need to interpret them are
the
concepts of "items" and "sets of items" discussed below.
YACC precedence and associativity declarations
YACC headers can specify precedence and associativity rules

for otherwise
heavily ambiguous grammars. Precedence is determined by
increasing order
of these declarations. Example:
%right ASSIGN
%left PLUS MINUS
%left TIMES DIVIDE
%right POWER
%%
expr: expr ASSIGN expr
| expr PLUS expr
| expr MINUS expr
| expr TIMES expr
| expr DIVIDE expr
| expr POWER expr
;
YACC error handling and recovery
Use special predefined token error where errors

expected
On an error, the parser pops states until it enters

one that has an
action on the error token.
For example: statement: error ';' ;
The parser must see 3 good tokens before it

decides it has recovered.
yyerrok tells parser to skip the 3 token recovery

rule
yyclearin throws away the current (error-causing?)

token
yyerror(s) is called when a syntax error occurs (s

is the error message)
Improving YACC's Error Reporting
yyerror(s) overrides the default error message, which usually

just says either
"syntax error" or "parse error", or "stack overflow".
You can easily add information in your own yyerror() function,

for example
GCC emits messages that look like:
goof.c:1: parse error before '}' token
using a yyerror function that looks like
void yyerror(char *s)
{
fprintf(stderr, "%s:%d: %s before '%s' token\n",
yyfilename, yylineno, s, yytext);
}
You could instead, use the error recovery mechanism to produce

better messages.
For example
lbrace : LBRACE | { error_code=MISSING_LBRACE; } error ;
Where LBRACE is an expected token {
This uses a global variable error_code to pass parse information

to yyerror().
Another related option is to call yyerror() explicitly with a better

message
string, and tell the parser to recover explicitly:
package_declaration: PACKAGE_TK error
{ yyerror("Missing name"); yyerrok; } ;
But, using error recovery to perform better error reporting runs

against
conventional wisdom that you should use error tokens very
sparingly.
What information from the parser determined we had an error in
the first
place? Can we use that information to produce a better error
message?
LR Syntax Error Messages: Advanced Methods
The pieces of information that YACC/Bison use to determine

that there
is an error in the first place are the parse state (yystate) and the
current input token (yychar). These are exactly the pieces of
information
one might use to produce better diagnostic error messages
without
relying on the error recovery mechanism and mucking up the

grammar
with a lot of extra production rules that feature the error token.
Even just the parse state is enough to do pretty good error

messages.
yystate is not part of YACC's public interface, though, so you
may
have to play some tricks to pass it as a parameter into yyerror()
from
yyparse(). Say, for example:
#define yyerror(s) __yyerror(s,yystate)
Inside __yyerror(msg, yystate) you can use a switch statement or

a global
array to associate messages with specific parse states. But,
figuring
out which parse state means which syntax error message would
be by trial
and error.
A tool called Merr is available that let's you generate this

yyerror
function from examples: you supply the sample syntax errors
and messages,
and Merr figures out which parse state integer goes with which
message.
Merr also uses the yychar (current input token) to refine the
diagnostics
in the event that two of your example errors occur on the same
parse state.
See the Merr web page.
Announcements
The TA's HW2 grades are available from the TA. The
distribution (out of 80) was
76, 74, 74, 74, 73, 72, 66, 65, 55, 52, 46, 35, 30, 30, 30, 15, 14
1/3rd of the class got an "A". The rest of you need to visit the
TA, see how
the grades were measured, see the professor, and most
important, get a lexical
analyzer working well enough to complete the later assignments
in this course.
If your grade was below 70, you probably want to get it working
and resubmit
it, I have asked the TA to accept resubmissions and average the

grades
(example: you got a 30, fixed it and resubmitted it and got a 70;
your overall
grade is a 50). This option is valid until the due date for the next
homework.
After all of this adjustment, you are being graded relative to
your
peers, not on an absolute 90/80/... scale. Depending on your
peers'
performance, a 60% score at the end of the semester could be a
"B" for all I
know. The purpose of the late penalty is to encourage you not to
fall
further and further behind as the semester progresses, and to
encourage you
to in fact catch up if you do fall behind.
For HW3 (syntax checker), make sure your tar file
unpacks OK and that "make" just works for us out of the
box. In your paper
turnin, make sure you DO include the lex .l and yacc .y files,
and make
sure you do NOT include the .c files generated from the lex .l
and
yacc .y files (lex.yy.c, y.tab.c, whatever). Include all .h files and
your
makefile.
For HW3, test your work on as many test cases as possible.
Midterm Exam is coming up, March 16. Midterm review

March 14.
Three more lectures before that.
LR vs. LL vs. LR(0) vs. LR(1) vs. LALR(1)
The first char ("L") means input tokens are read from the left
(left to right). The second char ("R" or "L") means parsing
finds the rightmost, or leftmost, derivation. Relevant
if there is ambiguity in the grammar. (0) or (1) or (k) after
the main lettering indicates how many lookahead characters are
used. (0) means you only look at the parse stack, (1) means you
use the current token in deciding what to do, shift or reduce.
(k) means you look at the next k tokens before deciding what
to do at the current position.
LR Parsers
LR denotes a class of bottom up parsers that is capable of

handling virtually
all programming language constructs. LR is efficient; it runs in

linear time
with no backtracking needed. The class of languages handled
by LR is a proper
superset of the class of languages handled by top down
"predictive parsers".
LR parsing detects an error as soon as it is possible to do
so. Generally
building an LR parser is too big and complicated a job to do by
hand, we use
tools to generate LR parsers.
The LR parsing algorithm is given below.

ip = first symbol of input
repeat {
s = state on top of parse stack
a = *ip
case action[s,a] of {
SHIFT s': { push(a); push(s') }
REDUCE A->beta: {
pop 2*|beta| symbols; s' = new state on top
push A
push goto(s', A)
}
ACCEPT: return 0 /* success */
ERROR: { error("syntax error", s, a); halt }
}
}
Constructing SLR Parsing Tables:
Note: in Spring 2006 this material is FYI but you will not be
examined on it.
Definition: An LR(0) item of a grammar G is a production

of G with a dot at some position of the RHS.
Example: The production A->aAb gives the items:
A -> . a A b
A -> a . A b
A -> a A . b
A -> a A b .
Note: A production A-> generates
only one item:
A -> .
Intuition: an item A-> . denotes:
1.
2.
3.
4.
5.
6.
- we have already seen a string

derivable from
- we hope to see a string derivable
from
Functions on Sets of Items
Closure: if I is a set of items for a grammar G, then closure(I)

is the set of items constructed as follows:
1.
2.
3.
4.
5.
Every item in I is in closure(I).

If A-> . B
is in closure(I) and B->
is a production, then add B-> .
6.
7.
to closure(I).
These two rules are applied repeatedly until no new items can
be added.
Intuition: If A -> . B is in
closure(I) then we hope to see a string derivable from B in the
input. So if B-> is a production,
we should hope to see a string derivable from .
Hence, B->. is in closure(I).
Goto: if I is a set of items and X is a grammar symbol, then

goto(I,X)
is defined to be:
goto(I,X) = closure({[A->X.] | [A->.X]
is in I})
Intuition:
[A->.X]
is in I => we've seen a string derivable
from ; we hope to see a string derivable
from X.
Now suppose we see a string derivable from X
Example: Consider the grammar
Then, we should "goto" a state where we've seen

a string derivable from X, and where
we hope to see a string derivable from .
The item corresponding to this is [A->X.]
E -> E+T | T
T -> T*F | F
F -> (E) | id
Let I = {[E -> E . + T]} then:

goto(I,+) = closure({[E -> E+.T]})
= closure({[E -> E+.T], [E -> .T*F], [T -> .F]})
= closure({[E -> E+.T], [E -> .T*F], [T -> .F], [F->

.(E)], [F -> .id]})
= { [E -> E + .T],[T -> .T * F],[T -> .F],[F -> .(E)],[F > .id]}
The Sets of Items Construction
1.
Given a grammar G with start symbol S, construct

the augmented
2.
grammar by adding a special production S'->S where
S' does
3.
not appear in G.
4.
5.
Algorithm for constructing the canonical collection
of LR(0)
6.
items for an augmented grammar G':
7.
begin
C := { closure({[S' -> .S]}) };
repeat
for each set of items I in C:
for each grammar symbol X:

if goto(I,X) != 0 and goto(I,X) is not in C then
add goto(I,X) to C;
until no new sets of items can be added to C;
return C;
end
Valid Items: an item A ->

1. 2
is valid for a viable prefix
1 if
there is a derivation:
S' =>*rm A =>*rm 1 2
Suppose A -> 1. 2 is valid for 1,

and B1 is on the parsing
stack
1.
2.
3.
if 2 != ,
we should shift
4.
5.
6.
7.
if 2 = ,
A -> 1 is the handle,
and we should reduce by this production
Note: two valid items may tell us to do different things for the
same viable prefix. Some of these conflicts can be resolved
using
lookahead on the input string.
Constructing an SLR Parsing Table
1.
Given a grammar G, construct the augmented

grammar by adding
2.
the production S' -> S.
3.
4.
Construct C = {I0, I1, In},
5.
the set of sets of LR(0) items for G'.
6.
7.
State I is constructed from Ii, with parsing action
8.
determined as follows:
9.
10.
o
[A -> .aB] is in
o
Ii, where a is a terminal; goto(Ii,a) = Ij
o
: set action[i,a] = "shift j"
o
o
o
o
o
o
o
[A -> .] is in
Ii : set action[i,a] to "reduce A -> x"
for all a e FOLLOW(A), where A != S'
[S' -> S] is in Ii :
set action[i,$] to "accept"
11.
12.
13.
14.
goto transitions constructed as follows: for all nonterminals:
15.
if goto(Ii, A) = Ij, then goto[i,A] = j
16.
17.
All entries not defined by (3) & (4) are made "error".
18.
If there are any multiply defined entries, grammar is
not SLR.
19.
20.
Initial state S0 of parser: that constructed from
21.
I0 or [S' -> S]
22.
Example:
S -> aABe
A -> Abc
{b,d}
FIRST(S) = {a}
FIRST{A} = {b}
FOLLOW(S) = {$}
FOLLOW(A) =
A -> b
FIRST{B} = {d}
FOLLOW{B} =
B -> d
FIRST{S'}= {a}
FOLLOW{S'}=
{e}
{$}
I0 = closure([S'->.S]
= closure([S'->.S],[S->.aABe])
goto(I0,S) = closure([S'->S.]) = I1
goto(I0,a) = closure([S->a.Abe])
= closure([S->a.Abe],[A->.Abc],[A->.b]) = I2
goto(I2,A) = closure([S->aA.Be],[A->A.bc])
= closure([S->aA.Be],[A->A.bc],[B->.d]) = I3
goto(I2,B) = closure([A->b.]) = I4
goto(I3,B) = closure([S->aAB.e]) = I5
goto(I3,b) = closure([A->Ab.c]) = I6
goto(I3,d) = closure([B->d.]) = I7
goto(I5,e) = closure([S->aABe.]) = I8
goto(I6,c) = closure([A->Abc.]) = I9
On Tree Traversals
Trees are classic data structures. Trees have nodes and edges, so
they are
a special case of graphs. Tree edges are directional, with roles
"parent"
and "child" attributed to the source and destination of the edge.
A tree has the property that every node has zero or one
parent. A node
with no parents is called a root. A node with no children is
called a leaf.
A node that is neither a root nor a leaf is an "internal
node". Trees have
a size (total # of nodes), a height (maximum count of nodes
from root to a leaf),
and an "arity" (maximum number of children in any one node).
Parse trees are k-ary, where there is a

variable number of children bounded by a value k determined by
the grammar.
You may wish to consult your old data structures book, or look
at some books
from the library, to learn more about trees if you are not totally
comfortable with them.
#include <stdarg.h>
struct tree {
short label;
short nkids;
/* what production rule this came from */

/* how many children it really has */
struct tree *child[1];
/* array of children, size varies 0..k */
};
struct tree *alctree(int label, int nkids, ...)
{
int i;
va_list ap;
struct tree *ptr = malloc(sizeof(struct tree) +
(nkids-1)*sizeof(struct tree *));
if (ptr == NULL) {fprintf(stderr, "alctree out of memory\n");
exit(1); }
ptr->label = label;
ptr->nkids = nkids;
va_start(ap, nkids);
for(i=0; i < nkids; i++)
ptr->child[i] = va_arg(ap, struct tree *);
va_end(ap);
return ptr;
}
Besides a function to allocate trees, you need to write one or

more recursive
functions to visit each node in the tree, either top to bottom
(preorder),
or bottom to top (postorder). You might do many different
traversals on the
tree in order to write a whole compiler: check types, generate
machine-
independent intermediate code, analyze the code to make it

shorter, etc.
You can write 4 or more different traversal functions, or you can
write
1 traversal function that does different work at each node,
determined by
passing in a function pointer, to be called for each node.
void postorder(struct tree *t, void (*f)(struct tree *))

{
/* postorder means visit each child, then do work at the parent
*/
int i;
if (t == NULL) return;
/* visit each child */
for (i=0; i < t-> nkids; i++)
postorder(t->child[i], f);
/* do work at parent */
f(t);
}
You would then be free to write as many little helper functions

as you
want, for different tree traversals, for example:
void printer(struct tree *t)
{
if (t == NULL) return;
printf("%p: %d, %d children\n", t, t->label, t->nkids);
}
Semantic Analysis
Semantic ("meaning") analysis refers to a phase of compilation

in which the
input program is studied in order to determine what operations
are to be
carried out. The two primary components of a classic semantic
analysis
phase are variable reference analysis and type checking. These
components
both rely on an underlying symbol table.
What we have at the start of semantic analysis is a syntax tree

that
corresponds to the source program as parsed using the context
free grammar.
Semantic information is added by annotating grammar symbols
with
semantic attributes, which are defined by semantic rules.

A semantic rule is a specification of how to calculate a semantic
attribute
that is to be added to the parse tree.
So the input is a syntax tree...and the output is the same tree,
only
"fatter" in the sense that nodes carry more information.
Another output of semantic analysis are error messages
detecting many
types of semantic errors.
Two typical examples of semantic analysis include:

variable reference analysis
the compiler must determine, for each use of a variable,
which
variable declaration corresponds to that use. This
depends on
the semantics of the source language being translated.
type checking
the compiler must determine, for each operation in the
source code,
the types of the operands and resulting value, if any.
Notations used in semantic analysis:
syntax-directed definitions
high-level (declarative) specifications of semantic rules
translation schemes
semantic rules and the order in which they get evaluated
In practice, attributes get stored in parse tree nodes, and the

semantic rules are evaluated either (a) during parsing (for easy
rules) or
(b) during one or more (sub)tree traversals.
Two Types of Attributes:
synthesized
attributes computed from information contained within
one's children.
These are generally easy to compute, even on-the-fly
during parsing.
inherited
attributes computed from information obtained from one's

parent or siblings
These are generally harder to compute. Compilers may
be able to jump
through hoops to compute some inherited attributes
during parsing,
but depending on the semantic rules this may not be
possible in general.
Compilers resort to tree traversals to move semantic
information around
the tree to where it will be used.
Attribute Examples
Isconst and Value
Not all expressions have constant values; the ones that do may
allow
various optimizations.
CFG
Semantic Rule
E1.isconst = E2.isconst &&
E1 : E2 + T T.isconst
if (E1.isconst)
E:T
T:T*F
T:F
F:(E)
F : ident
F : intlit
E1.value = E2.value + T.value

E.isconst = T.isconst
if (E.isconst)
E.value = T.value
T1.isconst = T2.isconst && F.isconst
if (T1.isconst)
T1.value = T2.value * F.value
T.isconst = F.isconst
if (T.isconst)
T.value = F.value
F.isconst = E.isconst
if (F.isconst)
F.value = E.value
F.isconst = FALSE
F.isconst = TRUE
F.value = intlit.ival
Questions from the board and from the floor
Symbol Table Module
Symbol tables are used to resolve names within name spaces.

Symbol
tables are generally organized hierarchically according to the
scope rules of the language. Although initially concerned with
simply
storing the names of various that are visible in each scope,
symbol
tables take on additional roles in the remaining phases of the
compiler.
In semantic analysis, they store type information. And for code
generation,
they store memory addresses and sizes of variables.
mktable(parent)
creates a new symbol table, whose scope is local to (or
inside) parent
enter(table, symbolname, type, offset)
insert a symbol into a table
lookup(table, symbolname)
lookup a symbol in a table; returns structure pointer
including type and offset. lookup operations are often
chained together progressively from most local scope on
out to global scope.
addwidth(table)
sums the widths of all entries in the table. ("widths" =

#bytes, sum of
widths = #bytes needed for an "activation record" or
"global data section").
Worry not about this method until code generation you
wish to implement.
enterproc(table, name, newtable)
enters the local scope of the named procedure
Variable Reference Analysis
The simplest use of a symbol table would check:
for each variable, has it been

declared? (undeclared error)
for each declaration, is it already declared?
(redeclared error)
Reading Tree Leaves
In order to work with your tree, you must be able to tell,

preferably
trivially easily, which nodes are tree leaves and which are
internal nodes,
and for the leaves, how to access the lexical attributes.
Options:
1.
2.
3.
4.
encode in the parent what the types of children are

encode in each child what its own type is (better)
How do you do option #2 here?

Perhaps the best approach to all this is to unify the tokens and
parse tree
nodes with something like the following, where perhaps an
nkids value of -1
is treated as a flag that tells the reader to use
lexical information instead of pointers to children:
struct node {
int code;
/* terminal or nonterminal symbol */
int nkids;
union {
struct token { ... } leaf;
struct node *kids[9];
}u;
};
There are actually nonterminal symbols with 0 children

(nonterminal with
a righthand side with 0 symbols) so you don't necessarily want
to use
an nkids of 0 is your flag to say that you are a leaf.
Type Checking
Perhaps the primary component of semantic analysis in many

traditional
compilers consists of the type checker. In order to check types,
one first
must have a representation of those types (a type system) and
then one must
implement comparison and composition operators on those
types using the
semantic rules of the source language being compiled. Lastly,
type checking
will involve adding (mostly-) synthesized attributes through
those parts of
the language grammar that involve expressions and values.
Type Systems
Types are defined recursively according to rules defined by the

source
language being compiled. A type system might start with rules
like:
Base types (int, char, etc.) are types

Named types (via typedef, etc.) are types
Types composed using other types are types, for
example:
o
o
o
o
o
array(T, indices) is a type. In some

languages indices always start with
0, so array(T, size) works.
T1 x T2 is a type (specifying, more or

less, the tuple or sequence T1
followed by T2;
o
x is a so-called cross-product
operator).
o
o
record((f1 x T1) x (f2 x T2) x ... x (fn x
Tn)) is a type
o
in languages with pointers, pointer(T) is a

type
o
o
o
(T1 x ... Tn) -> Tn+1 is a

type denoting a function mapping
parameter types to a return type
In some language type expressions may contain

variables whose values
are types.
In addition, a type system includes rules for assigning these

types
to the various parts of the program; usually this will be
performed
using attributes assigned to grammar symbols.
Midterm Exam Review
The Midterm will cover lexical analysis, finite automatas,

context free
grammars, syntax analysis, and parsing. Sample problems:
1.
Write a regular expression for numeric quantities of

U.S. money
2.
that start with a dollar sign, followed by one or
more digits.
3.
Require a comma between every three digits, as in
$7,321,212.
4.
Also, allow but do not require a decimal point
followed by two
5.
digits at the end, as in $5.99
6.
7.
Use Thompson's construction to write a nondeterministic finite
8.
automaton for the following regular expression,
an abstraction
9.
of the expression used for real number literal
values in C.
10.
(d+pd*|d*pd+)(ed+)?
11.
Write a regular expression, or explain why you can't
write a
12.
regular expression, for Modula-2 comments which
use (* *) as
13.
their boundaries. Unlike C, Modula-2 comments
may be nested,
14.
as in (* this is a (* nested *) comment *)
15.
16.
Write a context free grammar for the subset of C
expressions
17.
that include identifiers and function calls with
parameters.
18.
Parameters may themselves be function calls, as in
f(g(x)),
19.
or h(a,b,i(j(k,l)))
20.
21.
What are the FIRST(E) and FOLLOW(T) in the
grammar:
22.
23.
E:E+T|T
24.
T:T*F|F
F : ( E ) | ident
25.
What is the -closure(move({2,4},b)) in the following
NFA?
26.
That is, suppose you might be in either state 2 or 4
at the time
27.
you see a symbol b: what NFA states might you
find yourself in
28.
after consuming b?
(automata to be written on the board)
29.
Q: What else is likely to appear on the midterm?
A: questions that allow you to demonstrate that you know the

difference
between an DFA and an NFA, questions about lex and flex

and tokens
and lexical attributes, questions about context free grammars:
ambiguity, factoring, removing left recursion, etc.
On the mysterious TYPE_NAME
The C language typedef construct is an example where all the

beautiful
theory we've used up to this point breaks down. Once a typedef
is
introduced (which can first be recognized at the syntax level),
certain
identifiers should be legal type names instead of identifiers. To
make
things worse, they are still legal variable names: the lexical
analyzer
has to know whether the syntactic context needs a type name or
an
identifier at each point in which it runs into one of these names.
This
sort of feedback from syntax or semantic analysis back into
lexical
analysis is not un-doable but it requires extensions added by
hand to
the machine generated lexical and syntax analyzer code.
typedef int foo;

foo x;
/* a normal use of typedef... */
foo foo;
/* try this on gcc! is it a legal global? */
void main() { foo foo; } /* what about this ? */
370-C does not support typedef's and without working typedef's

the
TYPE_NAME token simply will never occur. Typedef's are fair
game for
extra credit points.
Representing C (C++, Java, etc.) Types
The type system is represented using data structures in the

compiler's
implementation language.
In the symbol table and in the parse tree attributes used in type
checking,
there is a need to represent and compare source language
types. You might
start by trying to assign a numeric code to each type, kind of like
the
integers used to denote each terminal symbol and each
production rule of the
grammar. But what about arrays? What about structs? There

are an infinite
number of types; any attempt to enumerate them will
fail. Instead, you
should create a new data type to explicitly represent type
information.
This might look something like the following:
struct c_type {
int base_type; /* 1 = int, 2=float, ... */
union {
struct array {
int size;
struct c_type *elemtype;
} a;
struct ctype *p;
struct struc {
char *label;
struct field **f;
} s;
} u;
}
struct field {
char *name;
struct ctype *elemtype;
}
Given this representation, how would you initialize a variable to

represent each of the following types:
int [10][20]
struct foo { int x; char *s; }
Example Semantic Rules for Type Checking
grammar rule
semantic rule
E1 : E2 PLUS E3 E1.type = check_types(PLUS, E2.type, E3.type)
Where check_types() returns a (struct c_type *) value. One of

the values
it should be able to return is Error. The operator (PLUS) is
included in
the check types function because behavior may depend on the
operator -the result type for array subscripting works different than the
result
type for the arithmetic operators, which may work different (in
some
languages) than the result type for logical operators that return
booleans.
Type Promotion and Type Equivalence
When is it legal to perform an assignment x = y? When x and y

are
identical types, sure. Many languages such as C have automatic
promotion rules for scalar types such as shorts and longs.
The results of type checking may include not just a type
attribute,
they may include a type conversion, which is best represented
by
inserting a new node in the tree to denote the promoted value.
Example:
int x;
long y;
y = y + x;
For records/structures, some languages use name equivalence,

while
others use structure equivalence. Features like typedef
complicate
matters. If you have a new type name MY_INT that is defined
to be
an int, is it compatible to pass as a parameter to a function that
expects regular int's? Object-oriented languages also get

interesting
during type checking, since subclasses usually are allowed
anyplace
their superclass would be allowed.
Implementing Structs
1.
storing and retrieving structs by their label -- the

struct label is
2.
how structs are identified. You do not have to do
typedefs and such.
3.
The labels can be keys in a separate hash table,
similar to the global
4.
symbol table. You can put them in the global
symbol table so long as
5.
you can tell the difference between them and
variable names.
6.
7.
8.
You have to store fieldnames and their types, from
where the struct is
9.
declared. You could use a hash table for each
struct, but a link list
10.
is OK as an alternative.
11.
12.
13.
You have to use the struct information to check the
validity of each
14.
dot operator like in rec.foo. To do this you'll have
to lookup rec
15.
in the symbol table, where you store rec's
type. rec's type must be
16.
a struct type for the dot to be legal, and that struct
type should
17.
include a hash table or link list that gives the names
and types of
18.
the fields -- where you can lookup the name foo to
find its type.
19.
Run-time Environments
How does a compiler (or a linker) compute the addresses for the
various
instructions and references to data that appear in the program
source code?
To generate code for it, the compiler has to "lay out" the data as
it will
be used at runtime, deciding how big things are, and where they
will go.
Relationship between source code names and data

objects during execution
Procedure activations
Memory management and layout
Library functions
Announcements
Affinity Research Group Workshop this Saturday,

9-3 in SH 124.
Extra credit: 20 points will be added to your

midterm exam grade
for attending and providing sincere attention at this

workshop.
Lunch is also provided.

HW#5 is available
Scopes and Bindings
Variables may be declared explicitly or implicitly in some

languages
Scope rules for each language determine how to go from names

to declarations.
Each use of a variable name must be associated with a

declaration.
This is generally done via a symbol table. In most compiled
languages
it happens at compile time (in contrast, for example ,with LISP).
Environment and State
Environment maps source code names onto storage addresses (at

compile time),
while state maps storage addresses into values (at

runtime). Environment
relies on binding rules and is used in code generation; state
operations
are loads/stores into memory, as well as allocations and
deallocations.
Environment is concerned with scope rules, state is concerned
with things
like the lifetimes of variables.
Runtime Memory Regions
Operating systems vary in terms of how the organize program

memory
for runtime execution, but a typical scheme looks like this:
code
static data
stack (grows down)
heap (may grow up, from bottom of address space)
The code section may be read-only, and shared among multiple

instances
of a program. Dynamic loading may introduce multiple code
regions, which
may not be contiguous, and some of them may be shared by

different programs.
The static data area may consist of two sections, one for
"initialized data",
and one section for uninitialized (i.e. all zero's at the beginning).
Some OS'es place the heap at the very end of the address space,
with a big
hole so either the stack or the heap may grow arbitrarily
large. Other OS'es
fix the stack size and place the heap above the stack and grow it
down.
Questions to ask about a language, before writing its code
generator
1.
2.
3.
4.
5.
May procedures be recursive? (Duh, all modern

languages...)
What happens to locals when a procedure returns?
(Lazy deallocation rare)
May a procedure refer to non-local, non-global

names?
6.
(Pascal-style nested procedures, and object field
names)
7.
8.
How are parameters passed? (Many styles possible,

different
9.
declarations for each (Pascal), rules hardwired by
type (C)?)
10.
11.
May procedures be passed as parameters? (Not too
awful)
12.
13.
May procedures be return values? (Adds complexity
for non-local names)
14.
15.
May storage be allocated dynamically (Duh, all
modern languages...
16.
but some languages do it with syntax (new) others
with library (malloc))
17.
18.
Must storage by deallocated explicitly (garbage
collector?)
19.
Activation Records
Activation records organize the stack, one record per

method/function call.
return value
parameter
...
parameter
previous frame pointer (FP)
saved registers
...
FP--> saved PC
local
...
local
temporaries
SP--> ...
At any given instant, the live activation records form a chain and
follow a stack discipline. Over the lifetime of the program, this
information (if saved) would form a gigantic tree. If you
remember
prior execution up to a current point, you have a big tree in
which
its rightmost edge are live activation records, and the nonrightmost
tree nodes are an execution history of prior calls.
"Modern" Runtime Systems
The preceding discussion has been mainly about traditional

languages such as
C. Object-oriented programs might be much the same, only
every activation
record has an associated object instance; they need one extra
"register" in
the activation record. In practice, modern OO runtime systems
have many
more differences than this, and other more exotic language
features imply
substantial differences in runtime systems. Here are a few
examples of
features found in runtimes such as the Java Virtual Machine and
.Net CLR.
Garbage collection. Automatic storage

management plays a prominent role
in most modern languages; it is one of the single

most important features
that makes programming easier.
The Basic problem in garbage collection: given a

piece of memory, are there
any pointers to it? (And if so, where exactly are all

of them please).
Approaches:
o
o
o
o
reference counting
traversal of known pointers (marking)
o
copying (2 heaps approach)

compacting (mark and sweep)
generational
conservative collection
Reflection. Modern languages' values can often

describe themselves.
This plays a central role in Visual GUI builders

and Visual IDE's,
component architectures and other uses.
Just-in-time compilation. Modern languages often

have a virtual machine
model...and a compiler built-in to the VM that

converts VM instructions
to native code for frequently executed methods

or code blocks.
Security model. Modern languages may attempt

to guarantee certain
security properties, or prevent certain kinds of

attacks.
Goal-directed programs have an activation tree each instant, due

to
suspended activations that may be resumed for additional
results. The
lifetime view is a sort of multidimensional tree, with three types
of nodes.
Having Trouble Debugging?
To save yourself on the semester project in this class, you really

do have
to learn gdb and/or ddd as well as you can. Sometimes it can
help you
find your bug in seconds where you would have spent hours
without it. But
only if you take the time to read the manual and learn the
debugger.
To work on segmentation faults: recompile all .c files with -g

and run your
program inside gdb to the point of the segmentation fault. Type
the gdb
"where" command. Print the values of variables on the line
mentioned in the
debugger as the point of failure. If it is inside a C library
function, use
the "up" command until you are back in your own code, and
then print the
values of all variables mentioned on that line.
There is one more tool you should know about, which is useful
for certain
kinds of bugs, primarily subtle memory violations. It is called
electric
fence. To use electric fence you add
/home/uni1/jeffery/ef/ElectricFence-2.1/libefence.a
to the line in your makefile that links your object files together
to
form an executable.
Need Help with Type Checking?
Implement the C Type Representation given in

lecture #16.
Read the Book
What OPERATIONS (functions) do you need, in

order to check
whether types are correct? What parameters

will they take?
Intermediate Code Generation
Goal: list of machine-independent instructions for each

procedure/method
in the program. Basic data layout of all variables.
Can be formulated as syntax-directed translation
add new attributes where necessary, e.g. for

expression E we might have
E.place
the name that holds the value of E
E.code
the sequence of intermediate code statements
evaluating E.
new helper functions, e.g.

newtemp()
returns a new temporary variable each time it is
called
newlabel()
returns a new label each time it is called
actions that generate intermediate code formulated

as semantic rules
Production
Semantic Rules
S -> id ASN
S.code = E.code || gen(ASN, id.place, E.place)
E
E.place = newtemp();
E -> E1
E.code = E1.code || E2.code ||
PLUS E2
gen(PLUS,E.place,E1.place,E2.place);
E.place = newtemp();
E -> E1 MUL
E.code = E1.code || E2.code ||
E2
gen(MUL,E.place,E1.place,E2.place);
E -> MINUS E.place = newtemp();
E1
E.code = E1.code || gen(NEG,E.place,E1.place);
E -> LP E1 E.place = E1.place;
RP
E.code = E1.code;
E.place = id.place;
E -> IDENT
E.code = emptylist();
Three-Address Code
Basic idea: break down source language expressions into simple

pieces that:
translate easily into real machine code

form a linearized representation of a syntax tree
allow us to check our own work to this point
allow machine independent code optimizations to
be performed
increase the portability of the compiler
Instruction set:
mnemonic
ADD,
SUB,MUL,DIV
C equivalent
NEG
x := op y
ASN
ADDR
x := y
x := &y
LCONT
x := *y
x := y op z
description
store result of binary
operation on y and z to x
store result of unary
operation on y to x
store y to x
store address of y to x
store contents pointed to by
y to x
SCONT
*x := y
GOTO
PARM
goto L
if x rop y then
goto L
if x then goto
L
if !x then goto
L
param x
CALL
call p,n,x
RET
return x
BLESS,...
BIF
BNIF
store y to location pointed to

by x
unconditional jump to L
binary conditional jump to L
unary conditional jump to L
unary negative conditional
jump to L
store x as a parameter
call procedure p with n
parameters, store result in x
return from procedure, use x
as the result
Declarations (Pseudo instructions):

These declarations list size units as "bytes"; in a uniform-size
environment
offsets and counts could be given in units of "slots", where a slot
(4 bytes
on 32-bit machines) holds anything.
global
x,n1,n2
proc
x,n1,n2
declare a global named x at offset n1 having n2

bytes of space
declare a procedure named x with n1 bytes of
parameter space and n2 bytes of local variable space
declare a local named x at offset n from the

procedure frame
label Ln designate that label Ln refers to the next instruction
declare the end of the current procedure
end
local x,n
TAC Adaptations for Object Oriented Code

x := y
field z
class
x,n1,n2
field x,n
new x
lookup field named z within y, store address to x

declare a class named x with n1 bytes of class
variables and n2 bytes of class method pointers
declare a field named x at offset n in the class frame
create a new instance of class name x
Variable Allocation and Access Issues
Given a variable name, how do we compute its address?

globals
easy, symbol table lookup
locals
easy, symbol table gives offset in (current) activation
record
objects
easy, symbol table gives offset in object, activation record

has
pointer to object in a standard location
locals in some enclosing block/method/procedure
ugh. Pascal, Ada, and friends offer their own unique kind
of pain.
Q: does the current block support recursion? Example:
for procedures
the answer would be yes; for nested { { } } blocks in C
the answer
would be no.
if no recursion, just count back some

number of frame pointers based
on source code nesting
if recursion, you need an extra pointer

field in activation record
to keep track of the "static link", follow

static link back some
# of times to find a name defined in an

enclosing scope
Sizing up your Regions and Activation Records
Add a size field to every symbol table entry. Many

types are not required
for your C370 project but we might want to discuss

them anyhow.
The size of integers is 4 (for x86; varies by CPU).
o
o
The size of reals is... ? (for x86; varies by CPU).
o
o
The size of strings is... <= 256? You could

allocate static
o
256 character arrays in the global area, but better to
do them as a
o
descriptor consisting of a length and a pointer.
o
o
o
The size of arrays is (sizeof (struct descrip)) * the

number of elements? Do we know an array size?
o
o
o
o
o
o
Are arrays all int, or all real, or can they be mixed?

(in BASIC and other dynamic languages, they
can be mixed!)
Are there arrays of strings? -- yes
what about sizes of structs?
You do this sizing up once for each scope. The

size of each scope is the
sum of the sizes of symbols in its symbol table.
Run Time Type Information
Some languages would need the type information

around at runtime; for
example, dynamic object-oriented languages. Its
almost the case that one
just writes the type information, or symbol table
information that includes
type information, into the generated code in this
case, but perhaps one
wants to attach it to the actual values held at
runtime.
struct descrip {
short type;
short size;
union {
char *string;
int ival;
float rval;
struct descrip *array;
/* ... for other types */
} value;
};
Compute the Offset of Each Variable
Add an address field to every symbol table entry.

The address contains a region plus an offset in that region.
No two variables may occupy the same memory at the same
time.
Locals and Parameters are not Contiguous
For each function you need either to manage two separate

regions
for locals and for parameters, or else you need to track where
in that region the split between locals and parameters will be.
Basic Blocks
Basic blocks are defined to be sequence of 1+ instructions in

which
there are no jumps into or out of the middle. In the most
extreme
case, every instruction is a basic block. Start from that
perspective
and then lump adjacent instructions together if nothing can come
between
them.
What are the basic blocks in the following 3-address code?

("read" is a 3-address code to read in an integer.)
read x
t1 = x > 0
if t1 == 0 goto L1
fact = 1
label L2
t2 = fact * x
fact = t2
t3 = x - 1
x = t3
t4 = x == 0
if t4 == 0 goto L2
t5 = addr const:0
param t5
; "%d\n"
param fact
call p,2
label L1
halt
Basic blocks are often used in order to talk about

specific types of optimizations that rely on basic blocks. So if
they are
used for optimization, why did I introduce basic blocks? You
can view
every basic block as a hamburger; it will be a lot easier to eat if
you
sandwich it inside a pair of labels (first and follow)!
Intermediate Code for Control Flow
Code for control flow (if-then, switches, and loops) consists of

code to test conditions, and the use of goto instructions and
labels to route execution to the correct code. Each chunk of
code
that is executed together (no jumps into or out of it) is called
a basic block. The basic blocks are nodes in a control flow
graph,
where goto instructions, as well as falling through from one

basic block
to another, are edges connecting basic blocks.
Depending on your source language's semantic rules for things

like
"short-circuit" evaluation for boolean operators, the operators
like || and && might be similar to + and * (non-short-circuit) or
they might be more like if-then code.
A general technique for implementing control flow code is to

add
new attributes to tree nodes to hold labels that denote the
possible targets of jumps. The labels in question are sort of
analogous to FIRST and FOLLOW; for any given list of
instructions
corresponding to a given tree node,
we might want a .first attribute to hold the label for the
beginning
of the list, and a .follow attribute to hold the label for the next
instruction that comes after the list of instructions. The .first
attribute can be easily synthesized. The .follow attribute must
be
inherited from a sibling.
The labels have to actually be allocated and attached to
instructions
at appropriate nodes in the tree corresponding to grammar
production
rules that govern control flow. An instruction in the middle of a

basic block need neither a first nor a follow.
C code
Attribute Manipulations
E.true = newlabel();
E.false = S.follow;
S->if E then S1
S1.follow = S.follow;
S.code = E.code || gen(LABEL, E.true)||
S1.code
E.false = newlabel();
S1.follow = S.follow;
S->if E then S1 else S2 S2.follow = S.follow;
S.code = E.code || gen(LABEL, E.true)||
S1.code || gen(GOTO, S.follow) ||
gen(LABEL, E.false) || S2.code
Exercise: OK, so what does a while loop look like?
Announcement
Co-op positions available for fall 2006 at Los Alamos

National Laboratory-in the Computing, Telecommunications,
and
Networking Division.
LANL is seeking outstanding SOPHOMORE, JUNIOR AND

NONGRADUATING SENIOR LEVEL Computer Science majors to
work in
the areas of networking, desktop support, high performance
computing or software engineering. Positions are available
for the fall 2006 semester. MUST HAVE A GPA OF 3.0 OR
HIGHER.
To request a referral go to www.nmsu.edu/pment, click on "Coop Job Listings", Job #86 or call the co-op office at 6464115. LANL is requiring a cover letter to also be sent,
please send that via email at coop@nmsu.edu in the subject
line put attn: LANL cover letter.
Co-op Office
646-4115
More on Generating Code for Boolean Expressions
Last time we started to look at code generation for control

structures
such as if's and while's. Of course, before we can see the big
picture on these we have to understand how to generate code for
the
boolean expressions that control these constructs.
Comparing Regular and Short Circuit Control Flow
Different languages have different semantics for booleans; for

example
Pascal treats them as identical to arithmetic operators, while the
C family of languages (and many ) others specify "short-circuit"
evaluation in which operands are not evaluated once the answer
to
the boolean result is known. Some ("kitchen-sink" design)
languages
have two sets of boolean operators: short circuit and non-shortcircuit.
(Does anyone know a language that has both?)
Implementation techniques for these alternatives include:
1.
treat boolean operators same as arithmetic operators,

evaluate
2.
each and every one into temporary variable
locations.
3.
4.
add extra attributes to keep track of code locations
that are
5.
targets of jumps. The attributes store link lists of
those instructions
6.
that are targets to backpatch once a destination
label is known.
7.
Boolean expressions' results evaluate to jump
instructions and program
8.
counter values (where you get to in the code
implies what the boolean
9.
expression results were).
10.
11.
one could change the machine execution model so it
implicity routes
12.
control from expression failure to the appropriate
location. In
13.
order to do this one would
14.
15.
mark boundaries of code in which failure

propagates
maintain a stack of such marked

"expression frames"
Non-short Circuit Example
a<b || c<d && e<f

translates into
100:
if a<b goto 103

t1 = 0
goto 104
103: t1 = 1
104: if c<d goto 107
t2 = 0
goto 108
107: t2 = 1
108: if e<f goto 111
t3 = 0
goto 112
111: t3 = 1
112: t4 = t2 AND t3
t5 = t1 OR t4
Short-Circuit Example
a<b || c<d && e<f

translates into
L2:
L3:
L1:
L4:
if a<b goto L1
if c<d goto L2
goto L3
if e<f goto L1
t=0
goto L4
t=1
...
Note: L3 might instead be the target E.false; L1 might instead be

E.true;
no computation of a 0 or 1 into t might be needed at all.
While Loops
So, a while loop, like an if-then, would have attributes similar

to:
C code
Attribute Manipulations
E.false = S.follow;
S1.follow = E.first;
S->while E do S1 S.code = gen(LABEL, E.first) ||
E.code || gen(LABEL, E.true)||
S1.code ||
gen(GOTO, E.first)
C for-loops are trivially transformed into while loops, so they

pose no new
code generation issues.
Intermediate Code Generation Examples
Consider the following small program. It would be fair game as

input
to your compiler project. In order to show blow-by-blow what
the code
generation process looks like, we need to construct the syntax

tree and
do the semantic analysis steps.
void main()
{
int i;
i = 0;
while (i < 20)
i = i * i + 1;
print(i);
}
This code has the following syntax tree
Intermediate Code Generation Example (cont'd)
Here is an example C progrma to compile:

i = 0;
if (i >= 20) goto L50;
i = i * i + 1;
goto 20;
print(i);
This program corresponds to the following syntax tree, which a

successful homework #5 would build. Note that it has a height
of
approximately 10, and a maximum arity of approximately
4. Also: your
exact tree might have more nodes, or slightly fewer; as long as
the
information and general shape is there, such variations are not a
problem.
A syntax tree, with attributes obtained from lexical and

semantic
analysis, needs to be shown here.
During semantic analysis, it is discovered that "print" has not
been
defined, so let it be:
void print(int i) { }
The code for the boolean conditional expression controlling the

while
loop is a list of length 1, containing the instruction t0 = i < 20,
or more formally
opcode dest src1 src2
LT
t0 i
20
The actual C representation of addresses dest, src1, and src2 is

probably as a
region
offset
pair, so the
picture of this intermediate code instruction really looks

something like this:
opcode
dest
local
src1 src2
local const
LT
t0.offset i.offset 20
Regions are expressed with a simple integer encoding like:

global=1, local=2, const=3.
Note that address values in all regions are offsets from the start
of the
region, except for region "const", which
stores the actual value of a single integer as its offset.
opcode
dest
local
src1 src2
local local
MUL
t1.offset i.offset i.offset

Comments on Trees and Attributes
The main problem in semantic analysis and intermediate code

generation is to
Move Information Around the Tree. Moving information up the
tree is kind of
easy and follows the pattern we used to build the tree in the first
place.
To move the information down the tree, needed for HW4, you
write tree
traversal functions. The tree traversal is NOT a "blind" traversal
that does
the same thing at each node. It has a switch statement on what
grammar rule
was used to build each node, and often does different work
depending on what
nonterminal and what grammar rule a given node represents.
Traversal code example
The following code sample illustrates a code generation tree

traversal.
Note the gigantic switch statement. In class a student asked the
question
of whether the link lists might grow longish, and if one is
usually appending
instructions on to the end, wouldn't a naive link list do a terrible
O(n2) job. To which the answer was: yes, and it would be good
to use a smarter data structure, such as one which stores both the
head
and the tail of each list.
void codegen(nodeptr t)
{
int i, j;
if (t==NULL) return;
/*
* this is a post-order traversal, so visit children first
*/
for(i=0;i<t->nkids;i++)
codegen(t->child[i]);
/*
* back from children, consider what we have to do with
* this node. The main thing we have to do, one way or
* another, is assign t->code
*/
switch (t->label) {
case PLUS: {
t->code = concat(t->child[0].code, t->child[1].code);
g = gen(PLUS, t->address,
t->child[0].address, t->child[1].address);
t->code = concat(t->code, g);
break;
}
/* ... really, we need a bazillion cases, perhaps one for each
* production rule (in the worst case)
*/
default:
/* default is: concatenate our children's code */
t->code = NULL;
for(i=0;i<t->nkids;i++)
t->code = concat(t->code, t->child[i].code);
}
}
Code generation examples
Let us build one operator at a time. You should implement your

code generation the same way, simplest expressions first.
Zero operators.
if (x) S
translates into
if x != 0 goto L1
goto L2
label L1
...code for S
label L2
or if you are being fancy
if x == b goto L1
...code for S
label L1
I may do this without comment in later examples, to keep them
short.
One relational operator.
if (a < b) S
translates into
if i >= b goto L1
...code for S
label L1
One boolean operator.
if (a < b && c > d) S

translates into
if (a < b)
if (c > d)
...code for S
which if we expand it
if i >= b goto L1
if c <= d goto L2
...code for S
label L2
label L1
by mechanical means, we may wind up with lots of labels for

the same
target, this is OK.
if (a < b || c > d) S
translates into
if (a < b) ...code for S

if (c > d) ...code for S
but its unacceptable to duplicate the code for S! It might be
huge!
Generate labels for boolean-true-yes-we-do-this-thing, not just
for
boolean-false-we-skip-this-thing.
if a < b goto L1
if c > d goto L2
goto L3
label L2
label L1
...code for S
label L3
Array subscripting!
So far, we have only said, if we passed an array as a parameter

we'd have to
pass its address. 3-address instructions have an "implicit
dereferencing
semantics" which say all addresses' values are fetched / stored
by default.
So when you say t1 := x + y, t1 gets values at addresses x and y,
not the
addresses. Once we recognize arrays are basically a pointer
type, we need
3-address instructions to deal with pointers.
now, what about arrays? reading an array value: x = a[i]. Draw

the
picture. Consider the machine uses byte-addressing, not wordaddressing.
t0 := addr a
t1 := i * 4
t2 := plus t0 t1
t3 := deref t2
x := t3
What about writing an array value?
Debugging Miscellany
Prior experience suggests if you are having trouble debugging,

check:
makefile .h dependencies!
if you do not list makefile dependencies for important .h
files,
you may get coredumps!
traversing multiple times by accident?
at least in my version, I found it easy to accidentally retraverse
portions of the tree. this usually had a bad effect.
bad grammar?
our sample grammar was adapted from good sources, but
don't assume its
impossible that it could have a flaw or that you might
have messed it up.
Remind me to come back to HW #6 before the end of

today's lecture.
Final Code
Goal: execute the program we have been translating, somehow.
Alternatives:
interpret the source code
we could have build an interpreter instead of a compiler, in
which the
source code was kept in string or token form, and reparsed every
execution. Early BASIC's did this, but it is Really Slow.
interpret the parse tree
we could have written an interpreter that executes the
program
by walking around on the tree doing traversals of
various subtrees.
This is still slow, but successfully used by many

"scripting languages".
interpret the 3-address code
we could interpret the link-list or a more compact binary
representation
of the intermediate code
translate into VM instructions
popular virtual machines such as JVM or .Net allow
execution from an
instruction set that is often higher level than hardware,
may be
independent of the underlying hardware, and may be
oriented toward
supporting the specific language features of our source
language.
For example, there are various BASIC virtual machines
out there.
translate into "native" instructions
"native" generally means hardware instructions.
In mainstream compilers,
final code generation takes a linear sequence of 3-address
intermediate
code instructions, and translates each 3-address instruction into
one or
more native instructions.
The big issues in code generation are (a) instruction selection,
and (b)
register allocation and assignment.
Collecting Information Necessary for Final Code

Generation
a top-down approach to learning your native target

code is to
o
study a reference work supplied by the chip
manufacturer, such
o
as the Intel 80386 Programmer's Reference
Manual
o
o
a bottom-up approach to learning your native
target code is to
o
study an existing compiler's native code. For
example, running
o
"gcc -S foo.c" will compile foo.c into a humanreadable native
o
assembler code equivalent foo.s file which you
can examine.
o
By systematically studying .s files for various
toy C programs
o
you can learn native instructions corresponding
to each C construct,
o
including ones equivalent to the various 3address instructions.
Instruction Selection
The hardware may have many difference sequences of

instructions to
accomplish a given task. Instruction selection must choose a
particular
sequence. At issue: how many registers to use, whether a
special
case instruction is available, and what addressing mode(s) to
use. Given
a choice among equivalent/alternaive sequences, the decision on
which
sequence of instructions to use is based on estimates or
measurements of
which sequence executes the fastest. This is usually determined
by the
number of memory references incurred during execution,
including the
memory references for the instructions themselves. Simply
picking the
shortest sequence of instructions is often a good approximation
of the
optimal result, since fewer instructions usually translates into
fewer
memory references.
Register Allocation and Assignment
Accessing values in registers is much much faster than accessing

main memory.
Register allocation denotes the selection of which variables will
go
into registers. Register assignment is the determination of
exactly
which register to place a given variable. The goal of these
operations
is generally to minimize the total number of memory accesses
required
by the program.
In the Old Days, there were Load-Store hardware architectures

in which
only one (accumulator) register was present. On such an
architecture,
register allocation and assignment is not needed; the compiler
has few
options about how it uses the accumulator register. Traditional
x86
16-bit architecture was only a little better than a load-store
architecture,
with 4 registers instead of 1. At the other extreme, Recent
History has
included CPU's with 32 or more general purpose registers. On
such systems,
high quality compiler register allocation and assignment makes a
huge
difference in program execution speed. Unfortunately, optimal

register
allocation and assignment is NP-complete, so compilers must
settle for
doing a "good" job.
Discussion of Tree Traversals that perform Semantic Tests.
Suppose we have a grammar rule

AssignStmt : Var EQU Expr.
We might extend the C semantic action for that rule with
extra code after building our parse tree node:
AssignStmt : Var EQU Expr { $$ = alctree(..., $1, $2, $3);
lvalue($1);
rvalue($3);
}
lvalue() and rvalue() are mini-tree traversals for the lefthand side
and righthand side of an assignment statement. Their missions
are to
propagate information from the parent, namely, inherited
attributes
that tell nodes whether their values are being assigned to

(initialized)
or being read from.
void lvalue(struct tree *t)

{
if (t->label == IDENT) {
struct symtabentry *ste = lookup(t->u.token.name);
ste->lvalue = 1;
}
for (i=0; inkids; i++) {
lvalue(t->child[i]);
}
}
void rvalue(struct tree *t)
{
if (t->label == IDENT) {
struct symtabentry *ste = lookup(t->u.token.name);
if (ste->lvalue == 0) warn("possible use before
assignment");
}
for (i=0; inkids; i++) {
lvalue(t->child[i]);
}
}
What is different about real life as opposed to this toy

example
In real life, you should build a flow graph, and propagate these
variable definition and use attributes using the flow graph
instead
of the syntax tree. For example, if the program starts by calling
a subroutine at the bottom of code which initializes all the
variables, the flow graph will not be fooled into generating
warnings
like you would if you just started at the top of the code and
checked
whether for each variable, assignments appear earlier in the
source
code than the uses of that variable.
Runtime Systems
Every compiler (including yours) needs a runtime system. A

runtime system
is the set of library functions and possibly global variables
maintained by
the language on behalf of a running program. You use one all

the time; in C
it functions like printf(), plus perhaps internal compilergenerated calls
to do things the processor doesn't do in hardware.
So you need a runtime system; potentially, this might be as big

or bigger a
job than writing the compiler. Languages vary from assembler
(no runtime
system) and C (small runtime system, mostly C with some
assembler) on up to
Java (large runtime system, mostly Java with some C) and in
even higher level
languages the compiler may evaporate and the runtime system
become gigantic.
The Unicon language has a relatively trivial compiler and
gigantic virtual
machine and runtime system. Other scripting languages might
have no compiler
at all, doing everything (even lexing and parsing) in the runtime
system.
For your project: whether you generate C or X86 or Java, you'll

need a plan
for what to do about a runtime system. And, in principle, I am
not opposed
to helping with this part. But the compiler and runtime system
have to fit
together; if I write part of the BASIC runtime system for you, or

we write
it together, we have to agree on things such as: what the types of
parameters and return values must look like.
So, what belongs in a Color BASIC runtime system? Anything

not covered
by a three address instruction. Looking at cocogram.y:
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
INPUT/PRINT
READ/DATA
CLEAR
CLOAD/CSAVE/SKIPF
CLS
SET/RESET
SOUND
CHR$, LEFT$, MID$, RIGHT$, INKEY$
ASC, INT, JOYSTK, LEN, PEEK, RND, VAL
DIM
string +, string compares
What would a runtime system function look like? It would take

in and
pass out BASIC values, represented as C structs. You would
then link
this code in to your generated C or assembler code (if you
generated
Java code, you would have to deal with the Java Native Interface
or
else write these functions in Java).
void PRINT(struct descrip *d)
{
switch (d->type) {
case INTEGER: printf("%d",d->value.ival); break;
case REAL: printf("%f",d->value.rval); break;
case STRING: printf("%*s",d->size,d->value.string); break;
case ARRAY: printf("cannot print arrays"); break; /* can't get
here */
default: printf("PRINT: internal error, type %d\n", d->type);
}
}
Now, let's look at the "whole" runtime system:
libb.c
More on Memory Management in the BASIC Runtime

System
Arrays are interesting. They can be used without being declared

or DIM'ed.
They can only be DIM'ed once. If you use them before they are
DIM'ed, they
are implicitly DIM'ed to size 11 and can't be re-DIM'ed.
What do variables A, A(), A$, and A$() look like in
memory? How does our
runtime system make it so?
Let's take a look at DIM, in libc.c. This DIM is for arrays of
numbers.
How would you handle arrays of strings?
Can you implement STRCAT for your BASIC runtime system?
What other BASIC statements, operators, or functions allocate
memory?
How would we avoid memory "leaks"?
Final Project comments
We have a week of classes more, plus a couple

weekends, before your
o
final project (HW#7) is due.
o
o
Really, really, test your turnin, by "turning your
.tar in to
o
yourself": unpacking in a separate directly, and
verifying that it builds
o
and runs correctly in a separate
subdirectory. Turnins that do not
o
compile and run due to missing files, etc. may
receive a 0.
o
o
You are invited and encouraged, but not required,
to make an appointment
o
to demo your compiler with me during Finals
week. The purpose is to
o
make sure you get credit for those parts which
you can show me are
o
working (as opposed to me testing your program
independently and
o
somehow missing the parts that work).
o
o
Student assignments which have "impossible
similarity" to each other
o
may result in a 0 for the assignment or an "F"
for the course.
o
Impossible similarity means: beyond a
reasonable doubt, substantial
code other than sample code fragments provided

in lecture or lab notes,
o
has been shared or copied.
STRCAT
So, what does your STRCAT look like? Here's one.

GOSUBs
Our 3-address instruction set has call and return instructions, but
basic
is less structured than regular procedural languages; you can
GOSUB to any
line number you want. You can't use a variable to GOSUB to
line number X,
but in principle every line number could be the target of a
procedure call.
If you use the "call" (3-address) instruction to do GOSUB, your
native code
will have to make a clear distinction between BASIC call's and
calls to
runtime system (built-in) functions. Perhaps it is best to

implement BASIC
GOSUB by pushing a "param" (the next instruction following
the GOSUB) and
a "goto". The BASIC RETURN is then a "pop" followed by a
"goto". What,
we don't have a "pop" 3-address instruction? We do now... the
name of
"param" should probably be "push" anyhow.
Come to think of it, we've been talking about doing a call to a

built-in
function such as PRINT, but that PRINT function we wrote is C
code; it
doesn't do a 3-address "ret" instruction, hmmm. How are we
going to
generate the native code for the 3-address "call" instruction?
It may include an assembler call instruction, but it may also
involve
instructions to handle the interface between BASIC and C.

Register Allocation and Assignment (cont'd)
When the number of variables in use at a given time exceeds the

number
of registers available (the common case), some variables may be
used
directly from memory if the instruction set supports memorybased operations.
When an instruction set does not support memory-based
operations, all
variables must be loaded into a register in order to perform
arithmetic
or logic using them.
Even if an instruction set does support memory-based

operations, most
compilers will want to load load a value into a register while it is
being used, and then spill it back out to main memory when the
register
is needed for another purpose. The task of minimizing memory
accesses
becomes the task of minimizing register loads and spills.
Some Code Generation Examples
Reusing a Register
Consider the statement:

a = a+b+c+d+e+f+g+a+c+e;
Our naive three address code generator would generate a
lot of temporary variables here, when really one big number is
being added.
How many registers does the expression need? Some variables
are referenced once, some twice. GCC generates:
movl
addl
addl
addl
addl
addl
addl
addl
addl
addl
movl
b, %eax
a, %eax
c, %eax
d, %eax
e, %eax
f, %eax
g, %eax
a, %eax
c, %eax
e, %eax
%eax, a
Now consider
a = (a+b)*(c+d)*(e+f)*(g+a)*(c+e);
How many registers are needed here?
movl
movl
b, %eax
a, %edx
addl
movl
addl
imull
movl
addl
imull
movl
addl
imull
movl
addl
imull
movl
%eax, %edx
d, %eax
c, %eax
%eax, %edx
f, %eax
e, %eax
%eax, %edx
a, %eax
g, %eax
%eax, %edx
e, %eax
c, %eax
%edx, %eax
%eax, a
And now this:

a = ((a+b)*(c+d))+((e+f)*(g+a))+(c*e);
which compiles to
movl
movl
addl
movl
addl
movl
imull
movl
movl
b, %eax
a, %edx
%eax, %edx
d, %eax
c, %eax
%edx, %ecx
%eax, %ecx
f, %eax
e, %edx
addl
movl
addl
imull
leal
movl
imull
leal
movl
%eax, %edx
a, %eax
g, %eax
%edx, %eax
(%eax,%ecx), %edx
c, %eax
e, %eax
(%eax,%edx), %eax
%eax, a
Lastly (for now) consider:

a = ((a+b)*(c+d))+(((e+f)*(g+a))/(c*e));
The division instruction adds new wrinkles. It operates on an
implicit
register accumulator which is twice as many bits as the number
you divide
by, meaning 64 bits (two registers) to divide by a 32-bit
number. Note
in this code that gcc would rather spill than use %ebx. %ebx is
either
being used implicitly or is reserved by the compiler for some
(probably
good) reason. %edi and %esi are similarly ignored.
movl b, %eax
movl a, %edx
addl %eax, %edx
movl d, %eax
addl c, %eax
movl
imull
movl
movl
addl
movl
addl
imull
movl
imull
movl
movl
cltd
idivl
movl
movl
leal
movl
%edx, %ecx
%eax, %ecx
f, %eax
e, %edx
%eax, %edx
a, %eax
g, %eax
%eax, %edx
c, %eax
e, %eax
%eax, -4(%ebp)
%edx, %eax
-4(%ebp)
%eax, -4(%ebp)
-4(%ebp), %edx
(%edx,%ecx), %eax
%eax, a
Code Generation for Virtual Machines
A virtual machine architecture such as the JVM changes the

"final" code
generation somewhat. We have seen several changes, some of
which
simplify final code generation and some of which complicate
things.
no registers, simplified addressing

a virtual machine may omit a register model and avoid
complex
addressing modes for different types of variables
uni-size or descriptor-based values
if all variables are "the same size", some of the details of
memory management are simplified. In Java most
values occupy
a standard "slot" size, although some values occupy two
slots.
In Icon and Unicon, all values are stored using a samesize descriptor.
runtime type system
requiring type information at runtime may complicate the
code generation task since type information must be
present
in generated code. For example in Java method
invocation and
field access instructions must encode class information.
Just for fun, let's compare the generated code for java with that
X86
native code we were just looking at:
iload_1
iload_2
iadd
iload_3
iload 4
iadd
imul
iload 5
iload 6
iadd
iload 7
iload_1
iadd
imul
iload_3
iload 5
imul
idiv
iadd
istore_1
A Shallow Introduction to Code Optimization
There are major classes of optimization that can significantly

speedup
a compiler's generated code. Usually you speed up code by

doing the
work with fewer instructions and by avoiding unnecessary
memory reads
and writes. You can also speed up code by rewriting it with
fewer gotos.
Peephole Optimization
Peephole optimizations look at the native code through a small,

moving
window for specific patterns that can be simplified. These are
some of the
easiest optimizations because they potentially don't require any
analysis
of other parts of the program in order to tell when they may be
applied.
Although some of these are stupid and you wouldn't think they'd
come up,
the simple code generation algorithm we presented earlier is
quite stupid
and does all sorts of obvious bad things that we can avoid.
name
redundant load or store
sample
MOVE R0,a
MOVE a,R0
optimized as
MOVE R0,a
dead code
control flow
simplification
algebraic simplification
strength reduction
#define debug 0
...
if (debug)
printf("ugh");
if a < b goto L1
...
L1: goto L2
if a < b goto
L2
...
L1: goto L2
x = x * 1;
x = y * 16;
x = y << 4;
Constant Folding
Constant folding is performing arithmetic at compile-time when

the values
are known. This includes simple expressions such as 2+3, but
with more
analysis
some variables' values may be known constants for some of
their uses.
x = 7;
...
y = x+5;
Common Subexpression Elimination
Code that redundantly computes the same value occurs fairly

frequently,
both explicitly because programmers wrote the code that way,
and implicitly
in the implementation of certain language features.
Explicit:
(a+b)*i + (a+b)/j;
The (a+b) is a common subexpression that you should not have

to compute twice.
Implicit:
x = a[i]; a[i] = a[j]; a[j] = x;
Every array subscript requires an addition operation to compute

the memory
address; but do we have to compute the location for a[i] and a[j]
twice in
this code?
Loop Unrolling
Gotos are expensive (do you know why?). If you know a loop
will
execute at least (or exactly) 3 times, it may be faster to copy
the
loop body those three times than to do a goto. Removing
gotos
simplifies code, allowing other optimizations.
x += 0 *
0;
y += x *
y += x * x;
x;
x += 1;
for(i=0; i<3; i++) { x += i * i; y += x
x += 1 * y += x *
* x; } >
1;
x;
y += x * x += 4;
x;
y += x *
x += 2 * x;
2;
y += x *
x;
Hoisting Loop Invariants
t_0 = strlen(s);
for (i=0; i<strlen(s); i++)
for (i=0; i<t_0; i++)
s[i] = tolower(s[i]);
s[i] = tolower(s[i]);
Interprocedural Optimization
Considering memory references across procedure call

boundaries;
for example, one might pass a parameter in a register if both
the caller and callee generated code knows about it.
argument culling
when the value of a specific parameter is a constant, a custom
version
of a called procedure can be generated, in which the
parameter is
eliminated, and the constant is used directly (may allow

additional
constant folding).
f(x,r,s,1);
f_1(x,r,s);
int f(int x, float y, char *z, int n)

int f_1(int x, float y, char *z)
{
{
switch (n) {
do_A;
case 1:
}
do_A; break;
int f_2(int x, float y, char *z)
case 2:
{
do_B; break;
do_B;
...
}
}
...
}
Final Exam Review
The final exam is comprehensive, but with a strong emphasis on

"back end"
compiler issues: symbol tables, semantic analysis, and code
generation.
o
o
o
o
o
Review your lexical analysis, regular expressions,

and finite automata.
Review your syntax analysis, CFG's, and parsing.
If a parser discovers a syntax error, how can it

report what line
o
number that error occurs on? If semantic
analysis discovers a
o
semantic error (or probable semantic error),
how can it report what
o
line number that error occurs on?
o
o
What are symbol tables?
o
o
What are symbol tables used for? What
information is stored there?
o
o
How does information get into a symbol table?
o
o
How many symbol tables does a compiler need?
o
o
What is "semantic analysis"?
o
o
What does "semantic analysis" accomplish? What
are its side effects?
o
o
What are the primary activities of a compiler's
semantic analyzer?
o
o
o
o
o
o
o
o
o
o
What are memory regions, and why does a

compiler care?
What memory regions are there, and how do they
affect code generation?
What does code generation do, anyhow?
What kinds of code generation are there?
Why do (almost all) compilers use an

"intermediate code"? What does
o
intermediate code look like? How is it different
from final code?

Compiler Notes - Ullman

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Compiler Notes - Ullman

Uploaded by

Copyright:

Available Formats

Compiler Construction Lecture Notes

lecture #1 began here

interactions between algorithms will help you out on your

implementation language, and have to think about and

Intermediate Code Generation:

Optimization of the intermediate code allows the four

A lexical analyzer, also called a scanner, typically has the

In compilers, a "token" is:

copy of each unique string and name needs to be allocated

The following example is from Kernighan & Ritchie's "The C

execute whenever foo.o should be rebuilt, i.e. when it is missing

besides regular expressions). Regular expressions are a

For regular expression r,

lecture #3 began here

8. yylex() does not skip multiple spaces, mishandles spaces at

another = for example.

FILE *yyin; /* set this variable prior to calling yylex() */

The body consists of of a sequence of regular expressions for

C float and double constants have to have at least one digit,

actually... flexible. The first several examples use flex as a filter

sneaky string removal tool:

int num_lines = 0, num_chars = 0;

/* scanner for a toy Pascal-like language */

printf( "An identifier: %s\n",

"+"|"-"|"*"|"/" printf( "An operator: %s\n",

On the use of character sets (square brackets) in lex and

instructions, there is just a hardwired short loop that executes the

draw a circle for each state s in S; put a label inside the

The Automaton Game

If I give you a transition diagram of a finite automaton, you can

Nondeterministic Finite Automata (NFA's)

multiple transitions on the same symbol handle common

factoring may optimize the number of states. Is this picture

C Pointers, malloc, and your future

struct tokenlist *current, *head;

Regular expression examples

3. For regular expressions r and s, draw r | s by adding a new

4. For regular expressions r and s, draw rs by adding

5. For regular expression r, draw r* by adding new start and

6. For parenthesized regular expression (r) you can use the

NFA to DFA Algorithm:

...did you get:

OK, how about this one:

lecture #7 began here

I have a collection of compiler textbooks in my office,

future assignments also, but not immediately, so as to allow

Lexical Analysis and the Literal Table

nodes contain grammar rules and pointers to tokens or

{id} { yylval.id = install_id(); return ID; }

you've stored it in a global variable, but there is not a way for

Context Free Grammar Examples

Within each procedure, a) match terminals at appropriate

Recursive Descent Parsing Example #1

tokens to the lexical analyzer, plus backtracking logic to E() or

Homework #3 minor extension

Removing Left Recursion

Left Recursion Versus Right Recursion: When does it

Automatic techniques for constructing parsers start with

sval = appendstring(sval, yytext[i]); /* instead of

char *appendstring(char *s, char c)

sval = realloc(sval, strlen(sval)+1);

lecture #11 began here

YACC ("yet another compiler compiler") is a popular tool

YACC files end in .y and take the form

FILE yyin; / set this variable prior to calling yylex() */

struct tokenlist current, head;

char appendstring(char s, char c)