Professional Documents
Culture Documents
Outline
What are Neural Networks? Biological neurons Artificial neurons Activation functions Training Example Voice recognition Different nets models Applications
Biological inspirations
Some numbers
The human brain contains about 10 billion nerve cells (neurons) Each neuron is connected to the others through 10000 synapses Working speed of human brain is estimated on 1018 operations/s , while the fastest machine ETA10 does 1010 operations/s.
Pigeons were able to discriminate between Van Gogh and Chagall with 95% accuracy (when presented with pictures they had been trained on) Discrimination still 85% successful for previously unseen paintings of the artists
Pigeons do not simply memorise the pictures They can extract and recognise patterns (the style) They generalise from the already seen to make predictions This is what neural networks (biological and artificial) are good at (unlike conventional computer)
Biological neuron
The brain is a collection of about 10 billion interconnected neurons. Each neuron is a cell that uses biochemical reactions to receive, process and transmit information. Each terminal button is connected to other neurons across a small gap called a synapse. A neuron's dendritic tree is connected to a thousand neighbouring neurons. When one of those neurons fire, a positive or negative charge is received by one of the dendrites. The strengths of all the received charges are added together through the processes of spatial and temporal summation.
Artificial neuron
Inputs p1 p2 p3 Weights w1 w2 w3
f
Bias
a
Output
a f p1w1 p2 w2 p3 w3 b f pi wi b
Neural computing requires a number of neurons, to be connected together into a neural network. Neurons are arranged in layers. Each neuron within the network is usually a simple processing unit which takes one or more inputs and produces an output. At each neuron, every input has an associated weight which modifies the strength of each input. The neuron simply adds together all the inputs and calculates an output to be passed on.
Activation functions
The activation function is generally non-linear. Linear functions are limited because the output is simply proportional to the input.
Training methods
Supervised learning
In supervised training, both the inputs and the outputs are provided. The network then processes the inputs and compares its resulting outputs against the desired outputs. Errors are then propagated back through the system, causing the system to adjust the weights which control the network. This process occurs over and over as the weights are continually tweaked. The set of data which enables the training is called the training set. During the training of a network the same set of data is processed many times as the connection weights are ever refined.
Training methods
Unsupervised learning
In unsupervised training, the network is provided with inputs but not with desired outputs. The system itself must then decide what features it will use to group the input data. This is often referred to as selforganization or adaption.
Format
Frequency distribution (60 bins) Analogy: cochlea
Network architecture
Feed forward network
60 input (one for each frequency bin) 6 hidden 2 output (0-1 for A, 1-0 for B)
0.43 0.26
0.73 0.55
Calculate error
A
0.43 0 0.26 1
= 0.43 = 0.74
0.73 1 0.55 0
= 0.27 = 0.55
0.43 0 0.26 1
0.73 1 0.55 0
0.01 0.99
0.99 0.01
Feed-forward nets
Information flow is unidirectional
Data is presented to Input layer Passed on to Hidden Layer Passed on to Output layer
Information is distributed
Recurrent Networks
Feed forward networks:
Information only flows one way One input pattern produces one output No sense of time (or memory of previous state)
Recurrency
Nodes connect back to other nodes or themselves Information flow is multidirectional Sense of time and memory of previous state(s)
Biological nervous systems show high levels of recurrency (but feed-forward structures exists too)
Elman Nets
Elman nets are feed forward networks with partial recurrency
Unlike feed forward nets, Elman nets have a memory or sense of time
Hopfield Networks
Sub-type of recurrent neural nets
Fully recurrent Weights are symmetric Nodes can only be on or off Random updating
Applications
Electronic systems diagnosis Stock market prediction Sonar target recognition (oil exploration) Medical test analyis Optical character recognition (OCR) Engines management Artificial intelligence robots To scan luggage at the aiports for dangerous items Speech synthesis Face recognition
Recurrent networks
Multidirectional flow of information Memory / sense of time Complex temporal dynamics (e.g. CPGs) Various training methods (Hebbian, evolution) Often better biological models than FFNs