Information & Coding Theory

Channel
Information (errors) Information
Symbols signal Symbols Destin-
Encoding signal Decoding
Source
Source/Channel
 + noise Channel/Source ation
s1,…,sq s1,…,sq

Noise
Example: Morse Code
telegraph wire
transmitter receiver

A, …, Z Encoding
dots, dashes ∙ ─ _ Decoding A, …, Z
spaces
keyer recognizer
shortwave radio

Example: ASCII Code

seven-bit Telephone seven-bit terminal

Character keyboard modem modem character
blocks wire blocks screen
Information Source – the symbols are undefined, and the “meaning” of the
information being sent is not dealt with – only an abstract measure of the
“amount” or “quantity.
Examples
text of various forms – reports, papers, memos, books, scientific data
(numbers)
pictures of various forms – diagrams, art, photographic images, scientific
data (e.g. from satellites)
sound of various forms – music, speech, noises, recorded sound, radio
animation of various forms – moving pictures, film, video tape, video
camera, television
equations representing mathematical ideas or algorithms – two textual
representation systems with graphical output: Tex & Mathematica

continuous waveforms discrete sampled and

analog and shapes digital quantized
 Analog vs. Digital
Examples of (apparently) analog sources of information:

sound (amplitude versus time) picture (amplitude versus space)

pressure: + →  intensity: [0, 1]  [0, 1] → +

granularity – molecular granularity – crystalline

microphone → tape recorder → speaker lenses → film (negative) → print (positive)

Examples of (apparently) digital sources of information:

different languages have
Text different characters
(often the typeface or writing
a′ ? B sequences of characters style conveys additional
meaning or information)
Examples of composites (some digital and analog)

color printing (as opposed to photography)

spatially discrete (dots)
spectrum discrete (color separation)
intensity discrete (dot or no dot)
television intensity unquantized, temporally discrete
color spatially discrete

random signal contains largest amount of information