1

INFORMATION THEORY AND CODING

Class: Communication
By Dr. Ahmed A. Hamad
Recommended Books:
1. B.P. Lathi, Modern Digital and Analog Communications, 3rd Ed.,
Oxford University Press, 1998.
2. H. P. Hsu, Theory and Problems of Analog and Digital
Communications, Schaums Outline Series, McGRAW-HILL, 1993.
3. T. M. Cover and J. A. Thomas, Elements of Information Theory,
JOHN WILEY & SONS, INC., 1991.
4. S. Lin, Error Control Coding: Fundamentals and Applications,
Prentice Hall., 1983.
5. C. B. Schlegel and L. C. Pe rez, Trellis and Turbo Coding, John Wiley
& Sons, Inc., Publication, 2004.
6. T. K. Moon, Error Correction Coding Mathematical Methods and
Algorithms, John Wiley & Sons, Inc., Publication, 2005.

Dr. Ahmed A. Hamad

INFORMATION THEORY AND CODING

Information theory provides a quantitative measure of the information contained
in message signals and allows us to determine the capacity of a communication
system to transfer this information from source to destination. Through the use
of coding, a major topic of information theory, redundancy can be reduced from
message signals to that channels can be used with improved efficiency. In
addition, systematic redundancy can be introduced to the transmitted signal so
that channels can be used with improved reliability.
1. DIGITAL COMMUNICATIONS MODEL
U

In the transfer of digital information, the following framework is often used:

Source

Source
coder

Channel
coder

Channel

Channel
decoder

Source
decoder

The source is an object that produces an event, the outcome of which is

selected at random according to a probability distribution. A practical source in
a communication system is a device that produces messages, and it can be
either analog or discrete. A discrete information source is a source that has only
a finite set of symbols as possible outputs. The set of source symbols is called
the source alphabet, and the elements of the set are called symbols or letters.
Information sources can be classified as having memory or being memoryless.
A source with memory is one for which a current symbol depends on the
previous symbols. A memoryless source is one for which each symbol
produces is independent of the previous symbols.
A discrete memoryless source (DMS) can be characterized by the list of the
symbols, the probability assignment to these symbols, and the specification of
the rate of generating these symbols by the source.
The source encoder serves the purpose of removing as much redundancy as
possible from the data. This is the data compression portion.
The channel coder puts a modest amount of redundancy back in order to do
error detection or correction.
The channel is what the data passes through, possibly becoming corrupted
along the way. There are a variety of channels of interest, including:
The magnetic recording channel
The telephone channel
Dr. Ahmed A. Hamad

Other bandlimited channels

The multi-user channel
Deep-space channels
Fading and/or jamming and/or interference channels
Any place where there is the possibility of corruption in the data
The channel decoder performs error correction or detection
The source decoder undoes what is necessary to get the data back.
There are also other possible blocks that could be inserted into this model like
encryption/decryption block.
In light of the model presented here, several questions arise of engineering
interest:
How can we measure the amount of information?
How much can we compress?
How do we compress?
How do we avoid errors from affecting the performance?
How fast can we send through the channel?
What if data rate exceeds the capacity of the channel?
2. INFORMATION CONTAIN OF A DISCRETE MEMORYLESS
SOURCE
U

The amount of information contained in an event is closely related to its

uncertainty. Messages containing knowledge of high probability of occurrence
convey relatively little information. We note that if an event is certain (that is,
the event occurs with probability 1), it conveys zero information. Thus a
mathematical measure of information should be a function of the probability of
the outcome and should satisfy the following axioms:
1. Information should be proportional to the uncertainty of an outcome.
2. Information contained in independent outcomes should add.
Consider a DMS, denoted by X, with alphabet {1 , 2 ,., }. The information
content of a symbol , denoted by ( ), is defined by
( ) =

1
= ( )
( )

(2.1)

Where ( ) is the probability of occurrence of symbol . Note that ( )

satisfies the following properties:
Dr. Ahmed A. Hamad

( ) = 0 ( ) = 1
( ) 0

( ) > ( ) ( ) < ( )

( ) = ( ) + ( ) if and are independent

The unit of ( ) is; bit if b=2, hartley if b=10, and nat (natural unit) if b=e. it is
standard to use b=2. The conversion of these units to other units can be achieved
by the following relationships.
2 =

=
2 2

Example 2.1: How much information you need if you want to know that number
2 appears in flipping a fair die?
U

Solution:
U

p(no. 2)=1/6

(fair die)
1

I(no. 2)= 2 =
6

6
2

= 2.5849 bits

Example 2.2: Find the amount of information contained in a (B/W) TV picture

U

assuming that it consists of 625(625 ) dots, each dots has 8 equiprobable

3

level of brightness?

Solution: prob. of each level of brightness=1/8

U

Information/dot= 2 =
8

8
2

= 3

Information/picture= 3 (625) (625 ) = 1.5625 M bits

3

H.W: solve above example for colored TV, each dot has 16 equiprobable colors
each color has 8 equiprobable level of brightness?
U

3. AVERAGE INFORMATION OR ENTROPY

U

In practical communication system, we usually transmit long sequences of

symbols from an information source. Thus we are more interested in the average
information that a source produces than the information content of a single
symbol.

Dr. Ahmed A. Hamad

The mean value of ( ) over the alphabet of source X with m different symbols
is given by

() = [( )] = ( )( )
=1

= ( )2 ( ) /
=1

(3.1)

The quantity () is called the entropy of source X. it is a measure of the

average information content per source symbol. The source entropy H(X) can be
considered as the average amount of uncertainty within source X that is resolved
by use of the alphabet.
Note that for a binary source X that generates independent symbols 0 and 1 with
equal probability, the source entropy H(X) is
1
1 1
1
() = 2 2 = 1 /
2
2 2
2

The source entropy H(X) satisfies the following relation:

(3.2)

0 () 2

where m is the size (number of symbols) of the alphabet of source X. The lower
bound corresponds to no uncertainty which occurs when one symbol has
probability ( )=1 while ( )=0 for j i, so X emits the same symbol xi all the
time. The upper bound corresponds to the maximum uncertainty which occurs
when ( )=1/m for all , that is, when all symbols are equally likely to be
emitted by X.
4. SOURCE ENTROPY RATE (INFORMATION RATE)
U

If the time at which source X emits symbols is r (symbol/sec), the information

rate R of the source is given by

Where

= 1/.

= () = ()/

b/sec

(4.1)

Example 4.1: A source produces dots and dashes, the probability of dote is
twice the probability of dash. The duration of the dot is 10 ms. If the duration of
the dash is three times the duration of the dot. Find the source entropy rate.
U

Dr. Ahmed A. Hamad