A Particle Swarm

Optimization (PSO) Primer

With
Applications

Brian Birge
Overview
p Introduction

p Theory

p Applications

p Computational Intelligence

p Summary
Introduction
p Subset of Evolutionary Computation
n Genetic Algorithms
n Evolutionary Programming
n Evolution Strategies
n Genetic Programming

p Behaviorally Inspired
n Developed out of attempts to model bird flocks and fish
schools

p Emergence
n Complex behavior from simple rules
Theory - Qualitative Details
p PSO Algorithm in a nutshell
p Searches Hyperspace of Problem for Optimum
n Define problem to search
p How many dimensions?
p Solution criteria?
n Initialize Population
p Random initial positions
p Random initial velocities
n Determine Global Best Position
n Determine Personal Best Position
n Update Velocity and Position Equations
Theory - Equations
p The Basic PSO algorithm consists of the
velocity:
vi ( k + 1) = vi ( k ) + γ 1i ( pi − xi ( k ) ) + γ 2i ( G − xi ( k ) )
p ..and position:
xi ( k + 1) = xi ( k ) + vi ( k + 1)
p i – particle index
p k – discrete time index
p v – velocity of ith particle
p x – position of ith particle
p p – best position found by ith particle (personal best)
p G – best position found by swarm (global best, best of personal
bests)
p γ1,2 – random numbers on the interval [0,1] applied to ith
particle
Theory - Equations ..
p The Common PSO Algorithm

vi ( k +1) = φ ( k ) vi ( k ) + α1 γ1i ( pi − xi ( k ) ) +α2 γ 2i ( G − xi ( k ) ) 

p φ - Inertia function
p α1,2 – Acceleration constants

p As training progresses using a decreasing linear

inertia function, the influence of past velocity
becomes smaller.
Applications
p PSOt – A Matlab Toolbox

p Function Optimization

p Neural Net Training

n Replacing Standard Backpropagation with PSO

p Engineering Examples
Applications - PSOt, A Matlab Toolbox
p Matlab: scientific computing language run
in interpreter mode on a wide variety of
operating systems.

p Toolbox: Suite of Matlab ‘plug-in’

programs developed by third parties.

p The PSOt includes:

n Standalone MISO function optimizer
n Neural Net Toolbox ‘plug-in’ for training
Applications 1

– Function Optimization
0.8

0.6

0.4

p Example: Schaffer’s ‘f6’ 0.2

function 0
20
10 20
0 10
0
>> pso('f6',2,[-100,100;-100,100],0) -10 -10
-20 -20

PSO: 1/2000 iterations, GBest =

0.499643. 20

PSO: 26/2000 iterations, GBest =

Pos dimension 2
0.0858499. 0
PSO: 51/2000 iterations, GBest =
0.00197516.
-20
PSO: 76/2000 iterations, GBest =
0.00197516.
PSO: 89/2000 iterations, GBest = -40
-10 -5 0 5 10 15 20 25
0.00197516. Pos dimension 1
0
PSO: 2 dimensional prob search, Gbestval=0.0019752
10
ans =
-4.400531404313024e-002 -1
Gbest value

10
4.527939759978006e-003
1.975158000277422e-003 -2
10

-3
10
0 10 20 30 40 50 60 70 80
epoch
Applications –
Artificial Neural Network Training
p Example of finding a neural net
approximation to XOR 1
mv=4 inertia wt=0.78092, 9 dimensions, GbestVal= 0.01641486174
10
TRAINPSO: 100/1000 epochs, gbest SSE =
0.428059141 0
10
mv = 4, iwt = 0.8537691795

gbest
.
-1
. 10
.
TRAINPSO: 225/1000 epochs, gbest SSE = -2
0.04639357585 10
0 50 100 150 200 250 300
mv = 4, iwt = 0.7953969313 epoch
TRAINPSO: 250/1000 epochs, gbest SSE =
0.04454761798 6
mv = 4, iwt = 0.7837224817
4

********* Reached Goal ******** 2

pos dim 9

TRAINPSO: 256/1000 epochs, gbest SSE =

0.01641486174 0
mv = 4, iwt = 0.7809206137 -2
********* end of training ********
-4
-10 -8 -6 -4 -2 0 2 4
pos dim 1
Applications –
Engineering Examples
p Human Tremor Analysis
n PSO used to evolve neural network weights
n Neural network distinguishes between
p Subjects with Parkinson’s Disease, Essential Tremor
p Normal Subjects
n 60 input, 12 hidden neurons, 2 outputs
n 100% success with generalization
n Very fast human tremor classification with very
small computational hit.
Applications –
Engineering Examples..
p Ingredient Mix Optimization
n Refers to a mixture of ingredients used to grow
production strains of micro-organisms that naturally
secrete or manufacture something of interest.
n PSO provided optimized ingredient mix much better than
then current state of the art
p Battery Pack State of Charge Estimation
n PSO used in conjunction with Backpropagation
n Important in development of electric and hybrid vehicle
technology
n PSO used to optimize training set for neural network in
addition to training.
Computational Intelligence
p Artificial Intelligence
n A name for a paradigm in which people attempt to elicit intelligence
from machines.

p Computational Intelligence
n Practical adaptation concepts, paradigms, algorithms, and
implementations that enable or facilitate appropriate actions
(intelligent behavior) in complex and changing environments.

p CI is different than AI
n Deals well with complex, dynamic, poorly defined problems that AI has
problems with.

p Examples
n Neuro-Fuzzy controllers
n ‘Alternatively’ trained neural nets
n Evolution of Fuzzy-Expert Systems using PSO
Summary
p PSO is inspired but not based on animal
behavior.
p A few simple rules result in complex
action, i.e. Emergence.
p PSO is simple to code and has small
computational cost.
p Successful in application to wide range of
engineering and computer science
problems.
References
p Birge, B., 2003, PSOt, A Particle Swarm Optimization Toolbox for Matlab,
IEEE Swarm Intelligence Symposium Proceedings, April 24-26

p Eberhart, R., Simpson, P., Dobbins, R., 1996, Computational Intelligence

PC Tools, Academic Press, Inc., pp. 212-223.

p Haykin, S., 1999, Neural Networks, a Comprehensive Foundation,

Second Edition, Prentice-Hall

p Houck, C., Joines, J., and Kay M., 1995, A Genetic Algorithm for
Function Optimization: A Matlab Implementation, ACM Transactions on
Mathematical Software, Submitted 1996

p Kennedy, J., Eberhart, R., 1995, Particle Swarm Optimization, from

Proc. IEEE Int’l. Conf. on Neural Networks (Perth, Australia), IEEE
Service Center, Piscataway, NJ, IV: 1942-1948

p Kennedy, J., Eberhart, R., Shi, Y., 2001, Swarm Intelligence, Academic
Press, Inc.