A Short Introduction To Particle Swarm Optimization

Business Intelligence:
Optimization for Decision Making

A short introduction to Particle Swarm Optimization
Michael G. Epitropakis
Computational Heuristics, Operational Research and Decision Support CHORDS,
School of Natural Sciences,
Computing Science and Mathematics,
University of Stirling, UK
mge@cs.stir.ac.uk
Stirling, 24 March 2015
Michael G. Epitropakis Swarm Intelligence for Decision Making 1

Outline
1 Business Intelligence
Motivation
Definition
Business Intelligence Architecture
2 Global Optimization Problem
3 Swarm Intelligence
Particle Swarm Optimization (PSO)
Background, Origins.
The Original PSO model
PSO: Geometric Illustration
4 Applications
5 References

Business Intelligence Motivation
Business Intelligence: Motivation
Amazon, Barclays, Facebook, Google, Lloyds, Microsoft,

Sainsburys, TESCO, ...
Data!
The answer to my problem is hidden in my data... but I cannot
dig it up!

Business Intelligence Definition
Business Intelligence
The enterprises that are capable of transforming data into

information and knowledge can use them to make quicker
and more effective decisions and thus to achieve a competitive
advantage.
Business Intelligence:
Business intelligence may be defined as a set of
mathematical models and analysis methodologies that
exploit the available data to generate information and
knowledge useful for complex decision-making processes.

Main purpose of business intelligence systems:

Is to provide decision makers with tools and methodologies that
allow them to make effective and timely decisions.
Effective: The application of rigorous analytical methods
allows decision makers to rely on information and
knowledge from data which are more dependable.
Make better decisions and devise action plans that allow
their objectives to be reached in a more effective way.
Timely: Enterprises operate in economic environments
characterized by growing levels of competition and high
dynamism.
Rapidly react to the actions of competitors and to new
market conditions is a critical factor in the success or even
the survival of a company.


Data, information and knowledge
Data Information Knowledge

Data: are collected on a daily basis
in the form of bits, numbers,
symbols, and "objects".
Information: is "organized data",
which are preprocessed, cleaned,
arranged into structures, and
stripped of redundancy.
Knowledge: is "integrated
information", which includes facts
and relationships that have been
perceived, discovered, or learned.

Business Intelligence Business Intelligence Architecture
Business Intelligence Architecture

Global Optimization Problem
Global Optimization
Single objective minimization problem:

Given a real-valued objective function f : Dn Rn R, the aim is to
find an x? = (x1? , x2? , . . . , xn? )> Dn that
x? = arg min f (x).

xDn
x? is a global minimizer and Dn is an n-dimensional scaled translation

of the unit hypercube.
An objective function (f )
A solution representation of x (here x Dn Rn )
A search strategy optimization algorithm.

Swarm Intelligence
Swarm Intelligence
Swarm intelligence (SI) is the collective behavior of

decentralized, self-organized systems, natural or artificial.
(Wikipedia)
The system has abilities that are not present in the
individuals (is more intelligent)
The whole is more than the sum of its parts
Cooperation, co-evolution, competition, self-organisation
and communication
Examples of systems can be found in nature: ant colonies,
bird flocking, animal herding, bacteria molding and fish
schooling
Beni, G., Wang, J. Swarm Intelligence in Cellular Robotic Systems, Proceed. NATO Advanced Workshop on Robots
and Biological Systems, Tuscany, Italy, June 26-30 (1989)

Swarm Intelligence
Swarm Intelligence

Swarm Intelligence
Swarm Intelligence Applications
Swarm-bots, an EU project led by Marco Dorigo, aimed to

study new approaches to the design and implementation
of self-organizing and self-assembling artifacts
(http://www.swarm-bots.org/).
Swarmanoid: Towards Humanoid Robotic Swarms, The
main scientific objective of this research project is the
design, implementation and control of a novel distributed
robotic system (http://www.swarmanoid.org)
Creation of complex interactive environments.

Disneys The Lion King was the first movie to make use
of swarm technology (the stampede of the bisons scene).
The movie Lord of the Rings has also made use of
similar technology during battle scenes

Swarm Intelligence Particle Swarm Optimization (PSO)
Particle Swarm Optimization

The inventors:
James Kennedy Russell C. Eberhart

(social psychologist) (electrical engineer)
J. Kennedy, and R. Eberhart, Particle swarm optimization, in

Proc. IEEE. Int. Conf. on Neural Networks, Piscataway, NJ, pp.
19421948, 1995.

Swarm Intelligence Particle Swarm Optimization (PSO)
What is PSO:
a simple, computationally efficient optimization method
population-based, stochastic search method
direct search method, i.e. gradient free
individuals follow very simple behaviors:
emulate the success of neighboring individuals,
but also bias towards on experience of success
emergent behavior: discovery of optimal regions within a
high dimensional search space
Particle swarm algorithm imitates human (or insects) social behavior. Individuals
interact with one another while learning from their own experience, and gradually the
population members move into better regions of the problem space Eberhart &
Kennedy

Swarm Intelligence Background, Origins.
Induce complex behavior from simple rules

Origins of PSO (precursors)
Reynolds (1987)s simulation Boids: a simple flocking model

consists of three simple local rules: http://www.red3d.com/cwr/boids/
Separation: Avoid Collision with neighboring

agents (steer to avoid crowding local
flockmates)
Alignment: Match the velocity of neighboring

agents (steer towards the average heading of
local flockmates)
Cohesion: Stay near neighboring agents

(steer to move toward the average position of
local flockmates)
The work of Heppner and Grenander on using a roost as

attractor of all birds in the flock [HG90] (Seek roost)

Towards a computational principle
Evaluate your current position

Compare it to your own experience (previous best) and to
the experience of your society (neighborhood best)
Imitate yourself and the others
Basic hypothesis:
There are two major sources of cognition:
own experience and
communication from others
Leon Festinger, 1954/1999, Social Communication and Cognition

The Original PSO model:

is a simplified social model of determining nearest neighbors
and velocity matching
Initial objective: to simulate the graceful, unpredictable
choreography of collision-proof birds in a flock
Randomly initializes positions of birds
At each iteration, each individual determines its nearest
neighbor and replaces its velocity with that of its neighbor
This resulted in synchronous movement of the flock, but
flock settled too quickly on the same, unchanging flying
direction

Random adjustments to velocities (referred to as

craziness) prevented individuals to settle too quickly on an
unchanging direction
To further expand the model, roosts were added as
attractors:
personal experience (personal best)
social experience (neighborhood best)
Introduction of the Particle Swarm Optimization method.

Swarm Intelligence The Original PSO model

Its main components:
What are the main components:

a swarm of particles (size usually fixed: NP)
each particle represents a candidate solution of the problem at
hand
the elements of a particle represent parameters to be optimized
The search process:

Position updates:
Xi (t + 1) = Xi (t) + Vi (t + 1), xi,j (0) U(LBj , UBj )
Velocity updates:
denotes the amount of change (step size)
drives the optimization process
reflects the cognitive experience of a particle and the
socially exchanged information between particles.
The general PSO Algorithm

1: Initialize particles in the swarm
2: for each time step t do
3: for each particle i in the swarm i {1, 2, . . . , NP} do
4: Update cognitive knowledge/experience
5: Update social knowledge/experience
6: end for
8: Update Velocity of particle i
9: Update Position of particle i
10: end for
11: end for


The global best (gbest) PSO
A simple PSO model: global best (gbest) PSO (Eberhart &

Kennedy, 1995)
It uses a full neighborhood topology (star social network).
Velocity update rule per dimension:

vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) xi,j (t) + c2 r2 (t) pbest,j (t) xi,j (t) ,
vi,j (0) = 0 (preferred)

c1 , c2 are positive acceleration coefficients
r1 (t), r2 (t) U(0, 1)



Kennedy, 1995)

vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) Xi,j (t) + c2 r2 (t) pbest,j (t) xi,j (t) ,
| {z }
momentum
momentum:
inertia component
previous velocity term to carry the particle in the direction it
has traveled so far
prevents particle from drastically changing direction



Kennedy, 1995)

Vi (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) xi,j (t) + c2 r2 (t) pbest,j (t) xi,j (t) ,
| {z }
cognitive component
cognitive component:
Pi (t): personal best position vector
quantifies performance relative to past performances
tendency to return to the best position visited so far
(memory)
nostalgia


Kennedy, 1995)

vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) xi,j (t) + c2 r2 (t) pbest,j (t) xi,j (t) ,
| {z }
social component
social component:
Pbest (t): neighborhood best position vector (here: global
best position)
quantifies performance relative to neighbors
tendency to be attracted towards the best position found in
its neighborhood.
envy

Update experience
Pi (t) is the personal best position calculated as (assuming

minimization):

Pi (t) if f (Xi (t + 1)) f (Pi (t))
Pi (t + 1) =
Xi (t + 1) if f (Xi (t + 1)) < f (Pi (t))
Pbest (t) is the global best position calculated as:
Pbest (t) = min{f (P0 (t)), f (P1 (t)), . . . , f (PNP (t))}
where NP is the number of particles in the swarm.

The gbest PSO Algorithm

1: Initialize particles in the swarm
2: for each time step t do
4: if f (Xi (t)) < f (Pi (t)) then
5: Pi (t) = Xi (t)
6: end if
7: if f (Pi (t)) < f (Pbest (t)) then
8: Pbest (t) = Pi (t)
9: end if
10: end for
11:

Vi (t + 1) = Vi (t) + c1 r1 (t) Pi (t) Xi (t) + c2 r2 (t) Pbest (t) Xi (t) ,
13: Xi (t + 1) = Xi (t) + Vi (t + 1)
14: end for
15: end for

Swarm Intelligence PSO: Geometric Illustration

Geometric Illustration
x2
Vi(t) Pbest(t)
Xi(t)
Pi(t)
x1

x2
Vi(t) Pbest(t)
Pbest(t) Xi(t)
Xi(t) Pi(t) Xi(t)
Pi(t)
x1

x2
Vi(t) Pbest(t)
Pbest(t) Xi(t)
Xi(t) Pi(t) Xi(t)
Pi(t)
x1

x2
c1r1(t)(Pbest(t) Xi(t))
Vi(t) Pbest(t)
Pbest(t) Xi(t)
Xi(t) Pi(t) Xi(t)
Pi(t)
x1

x2
c1r1(t)(Pi(t) Xi(t))
Vi(t) Pbest(t)
Pbest(t) Xi(t)
Xi(t) Pi(t) Xi(t)
Pi(t)
x1

x2
c1r1(t)(Pi(t) Xi(t))
Vi(t) Xi(t + 1)
Pbest(t)
Pbest(t) Xi(t)
Xi(t) Pi(t) Xi(t)
Pi(t)
x1

The local best (lbest) PSO
The local best (lbest) PSO uses a neighborhood topology (ring

social network).

vi,j (t + 1) = vi,j (t) + c1 r1 (t) pi,j (t) xi,j (t) + c2 r2 (t) pnbest,j (t) xi,j (t) ,
| {z }
social component
Pnbest (t): is the neighborhood best, defined as:

Pnbest (t + 1) {x Ni | min{f (x), x Ni }}
Ni = {pinNi , pinNi +1 , . . . , pi1 , pi , pi+1 , . . . , pi+nNi }
where nNi is the neighborhood size
neighborhoods are based on particle indices, not spatial
information
neighborhoods overlap to facilitate information exchange

Social Neighborhood topologies: Social Network Structures
Two most common models:

lbest: each particle is influenced only by particles in local
neighborhood
gbest: each particle is influenced by the best found from
the entire swarm
Ring Topology Star/full topology


Social Neighborhood topologies: Social Network Structures (2)
Von Neumann Four Clusters Topology Wheel Topology

Topology


Velocity Clamping
Problem:
The velocity has a tendency to
explode to large values.
Solution: Velocity Clamping

vi,j (t + 1) if |vi,j (t + 1)| < Vmax,j
vi,j (t + 1) =
sgn(vi,j (t + 1))Vmax,j if |vi,j (t + 1)| Vmax,j
controlling the global exploration of the particles
it is problem-dependent
does not necessarily prevent particles from leaving the search
space, nor to converge.
it confines the step sizes, therefore restricting particles from
further divergence

Velocity Clamping
x2
Velocity Update
Position Update
Xi(t)
x1

Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1)
Xi(t) vi(t + 1)
x1

Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1) Xi(t + 1)
Xi(t) vi(t + 1)
x1

Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1) Xi(t + 1)
v2(t + 1)
Xi(t) vi(t + 1)
x1

Velocity Clamping
x2
Velocity Update
Position Update
v2(t + 1) Xi(t + 1)
i(t + 1)
X
v2(t + 1)
Xi(t) vi(t + 1)
x1
Some PSO Variants
Tribes (Clerc, 2006) aims to adapt population size, so that it does not have to
be set by the users
FDR-PSO (Veeramachaneni, et al., 2003) using nearest neighbour interactions
Cooperative PSO (van den Bergh and Engelbrecht, 2005) a cooperative
approach
CLPSO (Liang, et al., 2006) incorporate learning from more previous best
particles.
FIPS Fully Informed PSO (Mendes, Kennedy, 2004) use several attractors in
the update rule
BBPSO Bare Bones PSO (Kennedy, 2003) uses normal distribution around
personal/global best
UPSO Unified PSO (Parsopoulos, Vrahatis, 2004) a unification of gbest and
lbest versions
PSODE (Epitropakis et al., 2012) aims to combine various state-of-the-art DE
and PSO variants
Standard PSO 2006, 2007, 2011 aims to define a standard PSO version for
comparisons
http://www.particleswarm.info/

Applications
Applications (1)
Optimization Problems:
Continuous, discrete, mixed search spaces, Optimization
Problems
Combinatorial Optimization Problems
Large Scale Optimization Problems
Multi-modal Optimization Problems
Multi-objective Optimization Problems
Problems in Dynamic and Uncertain environments
Constraint Optimization Problems

Applications
Applications (2)
Applications in:
Machine Learning (clustering, classification, parameter
tuning, feature selection)
Artificial Neural Networks (training, evolving structures)
Robotics (path planning, localization)
Bio-informatics and Medical Informatics (Medical diagnosis
and decision making)
Image processing (Image analysis, segmentation, pattern
recognition)
Industrial Applications (Job Scheduling, Vehicle Routing
Problem, Traveling Salesman Problem)

(-: Thank you very much for your attention :-)
Questions ???
Michael G. Epitropakis: mge@cs.stir.ac.uk

www.epitropakis.co.uk

References
References: Business Intelligence (Incomplete)
1 Carlo Vercellis, Business Intelligence: Data Mining and Optimization for Decision
Making, John Wiley & Sons, 2009
2 Z. Michalewicz, M. Schmidt, M. Michalewicz, Adaptive Business Intelligence,
Springer, 2006
3 Foster Provost, Tom Fawcett, Data Science for Business: What you need to
know about data mining and data-analytic thinking, OReilly Media, 2013
4 A. Brabazon, M. ONeill, I. Dempsey, An Introduction to Evolutionary
Computation in Finance, IEEE Computational Intelligence Magazine, 2008
5 W. Pedrycz, N. Ichalkaranje, G.P. Wren and L. Jain, Introduction to
Computational Intelligence for Decision Making, in Studies in Computational
Intelligence, 97, 79-96, Springer-Verlag, 2008

References
References: PSO (Incomplete)

1 Reynolds, C.W.: Flocks, herds and schools: a distributed behavioral model.
Computer Graphics, 21(4), p.25-34, 1987.
2 Heppner, F. and Grenander, U.: A stochastic nonlinear model for coordinated
bird flocks. In S.Krasner, Ed., The Ubiquity of Chaos. AAAS Publications,
Washington, DC, 1990.
3 Kennedy, J. and Eberhart, R.: Particle Swarm Optimization. In Proceedings of
the Fourth IEEE International Conference on Neural Networks, Perth, Australia.
IEEE Service Center(1995) 1942-1948.
4 Kennedy, J., Eberhart, R. C., and Shi, Y., Swarm intelligence, San Francisco:
Morgan Kaufmann Publishers, 2001.
5 J. Kennedy, Small Worlds and Mega-Minds: Effects of Neighborhood Topology
on Particle Swarm Performance, Proceedings of the IEEE Congress on
Evolutionary Computation, 1999, pp. 1931-1938.
6 Clerc, M.: Particle Swarm Optimization, ISTE Ltd, 2006.
7 Engelbrecht, A.P., Fundamentals of Computational Swarm Intelligence, Wiley,
2006.
8 Xiaodong Li, Advances in Particle Swarm Optimization, Tutorial, ACISS09,
Melbourne
9 Engelbrecht, A.P., Particle Swarm Optimization Tutorial, IEEE Congress on
Evolutionary Computation, 2013
10 KE Parsopoulos, MN Vrahatis, Particle swarm optimization and intelligence:
advances and applications, Information Science Reference, 2010

A Short Introduction To Particle Swarm Optimization

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Short Introduction To Particle Swarm Optimization

Uploaded by

Copyright:

Available Formats

Business Intelligence:

Optimization for Decision Making

Stirling, 24 March 2015

Michael G. Epitropakis Swarm Intelligence for Decision Making 1

Michael G. Epitropakis Swarm Intelligence for Decision Making 2

Business Intelligence: Motivation

Amazon, Barclays, Facebook, Google, Lloyds, Microsoft,

Michael G. Epitropakis Swarm Intelligence for Decision Making 3

The enterprises that are capable of transforming data into

Michael G. Epitropakis Swarm Intelligence for Decision Making 4

Main purpose of business intelligence systems:

Michael G. Epitropakis Swarm Intelligence for Decision Making 5

Michael G. Epitropakis Swarm Intelligence for Decision Making 6

Data, information and knowledge

Data Information Knowledge

Michael G. Epitropakis Swarm Intelligence for Decision Making 7

Business Intelligence Architecture

Michael G. Epitropakis Swarm Intelligence for Decision Making 8

Single objective minimization problem:

x? = arg min f (x).

x? is a global minimizer and Dn is an n-dimensional scaled translation

Michael G. Epitropakis Swarm Intelligence for Decision Making 9

Swarm intelligence (SI) is the collective behavior of

Michael G. Epitropakis Swarm Intelligence for Decision Making 10

Michael G. Epitropakis Swarm Intelligence for Decision Making 11

Swarm Intelligence Applications

Swarm-bots, an EU project led by Marco Dorigo, aimed to

Creation of complex interactive environments.

Michael G. Epitropakis Swarm Intelligence for Decision Making 12

Particle Swarm Optimization

James Kennedy Russell C. Eberhart

J. Kennedy, and R. Eberhart, Particle swarm optimization, in

Michael G. Epitropakis Swarm Intelligence for Decision Making 13

Particle Swarm Optimization (PSO)

Michael G. Epitropakis Swarm Intelligence for Decision Making 14

Induce complex behavior from simple rules

Reynolds (1987)s simulation Boids: a simple flocking model

Separation: Avoid Collision with neighboring

Alignment: Match the velocity of neighboring

Cohesion: Stay near neighboring agents

The work of Heppner and Grenander on using a roost as

Michael G. Epitropakis Swarm Intelligence for Decision Making 15

Towards a computational principle

Evaluate your current position

Leon Festinger, 1954/1999, Social Communication and Cognition

Michael G. Epitropakis Swarm Intelligence for Decision Making 16

The Original PSO model:

Michael G. Epitropakis Swarm Intelligence for Decision Making 17

Random adjustments to velocities (referred to as

Introduction of the Particle Swarm Optimization method.

Michael G. Epitropakis Swarm Intelligence for Decision Making 18

Particle Swarm Optimization (PSO)

What are the main components:

The search process:

Particle Swarm Optimization

The general PSO Algorithm

Michael G. Epitropakis Swarm Intelligence for Decision Making 20

Particle Swarm Optimization

A simple PSO model: global best (gbest) PSO (Eberhart &

Velocity update rule per dimension:

vi,j (0) = 0 (preferred)

Michael G. Epitropakis Swarm Intelligence for Decision Making 21

Particle Swarm Optimization