PVM and MPI

What is more preferable?

Comparative analysis of PVM and

MPI for the development of physical
applications on parallel clusters

Ekaterina Elts
Scientific adviser: Assoc. Prof. A.V. Komolkin

Introduction
Computational Grand
Challenge problems
Parallel processing the
method of having many
small tasks to solve one
large problem
Two major trends :
MPPs - (massively parallel
processors) but cost $$$ !
distributed computing

Introduction
The hottest trend today
is PC clusters running
Linux
Many Universities and
companies can afford
16 to 100 nodes.
PVM and MPI are the
most used tools for
parallel programming

Contents
Parallel Programming
A Parallel Machine Model
Cluster
A Parallel Programming Model
Message Passing Programming Paradigm

PVM and MPI

Background
Definition
A Comparison of Features

Conclusion

A Sequential Machine Model

A central processing
unit (CPU) executes a
program that performs
a sequence of read and
write operations on an
attached memory
The
von Neumann
computer
SISD Single Instruction Stream Single Data Stream

A Parallel Machine Model

Interconnect

The cluster
A node can communicate with
other nodes by sending and
receiving messages over an
interconnection network

The
von Neumann
computer

MIMD Multiple Instruction Stream Multiple Data Stream

A Parallel Programming
Model
input

output
Sequential (serial) algorithm

input

output
Parallel algorithm

Example: scalar product of

vectors

a, b

a, b

input
input

do i=1,N
S=s+aibi
enddo

output

do i=1,N/2
s1=s1+aibi
enddo

do i=N/2+1,N
s2=s2+aibi
enddo

S=s1+s2
output

print S
Sequential (serial) algorithm

print S
Parallel algorithm

A Parallel Programming
Model
Message Passing
4
1

3
2

Many small tasks solve one large problem

Instantaneous state of computation

detailed picture of
a single task

Message Passing Paradigm

Each processor in a message passing
program runs a separate process (subprogram, task)
written in a conventional sequential language
all variables are private
communicate via special subroutine calls

Messages
Messages are packets of data moving
between processes
The message passing system has to be
told the following information:
Sending process
Source location
Data type
Data length
Receiving process(es)
Destination location
Destination size

Message Passing
SPMD
Single Program Multiple Data

Same program runs

everywhere
Each process only
knows and operates
on a small part of
data

MPMD
Multiple Program Multiple Data

Each process
perform a different
function (input,
problem setup,
solution, output,
display)

What is Master/Slave principle?

The master has the control over the
running application, it controls all data and
it calls the slaves to do there work
PROGRAM
IF (process = master) THEN
master-code
ELSE
slave-code
ENDIF
END

Simple Example
SPMD&Master/Slave
For i from rank step size to N do
s=s+aibi
a1b1+a1+sizeb1+size+a1+2*sizeb1+2*size+
enddo

s2 ai bi
2

a, b

s1 ai bi
1

s3 ai bi
3

slave

S=s1+s2 master
slave

PVM and MPI

Background

PVM

The development of
PVM started in summer
1989 at Oak Ridge
National
Laboratory
(ORNL).
PVM was effort of a single
research group, allowing
it great flexibility in design
of this system
MPI-1

PVM-1
1989

90

PVM-2
94

The development of MPI

started in April 1992.
MPI was designed by the
MPI Forum (a diverse
collection of implementors,
library writers, and end
users) quite independently
of
any
specific
implementation
MPI-2

PVM-3
96

MPI

PVM-3.4
97

99

2000

PVM and MPI

PVM

Goals

MPI

A library for writing

application program, not
a distributed operating
system
portability
High Performance
Heterogeneity
Well-defined behavior
Note: implementation specification!

A distributed
operating system
Portability
Heterogeneity
Handling
communication
failures

MPI implementations: LAM, MPICH,

What is MPI ?
MPI - Message Passing Interface
A fixed set of processes is created at program
initialization, one process is created per
processor
mpirun np 5 program
Each process knows its personal number (rank)
Each process knows number of all processes
Each process can communicate with other
processes
Process cant create new processes (in MPI-1)

What is PVM ?
PVM - Parallel Virtual Machine
Is a software package that allows a
heterogeneous collection of workstations (host
pool) to function as a single high performance
parallel machine (virtual)
PVM, through its virtual machine provides a
simple yet useful distributed operating system
It has daemon running on all computers making
up the virtual machine

PVM Daemon (pvmd)

UNIX process which oversees the operation of
user processes within a PVM application and
coordinates inter-machine PVM communications
The pvmd serves as a message router and
controller
One pvmd runs on each host of a virtual
machine
The first pvmd (started by hand) is designated
the master, while the others (started by the
master) are called slaves
Only the master can start new slaves and add
them to configuration or delete slave hosts from
the machine

Executing user
computation

master

Executing
PVM system
routines

What is Not Different?

Portability source code written for one architecture

can be copied to a second architecture, compiled and

executed without modification (to some extent)
Support MPMD programs as well as SPMD
Interoperability the ability of different implementations
of the same specification to exchange messages
Heterogeneity (to some extent)
PVM & MPI are systems designed to provide users with
libraries for writing portable, heterogeneous, MPMD
programs

Heterogeneity

Architecture
Data format
Computational speed
Machine load
Network load

static

dynamic

Heterogeneity: MPI
Different datatypes can be encapsulated
in a single derived type, thereby allowing
communication of heterogeneous
messages. In addition, data can be sent
from one architecture to another with data
conversion in heterogeneous networks
(big-endian, little-endian).

Heterogeneity: PVM
The PVM system supports heterogeneity
in terms of machines, networks, and
applications. With regard to message
passing, PVM permits messages
containing more than one datatype to be
exchanged between machines having
different data representations.

Process control
- Ability to start and stop tasks, to find out which
tasks are running, and possibly where they are
running.
PVM contains all of these capabilities
it can spawn/kill tasks dynamically
MPI -1 has no defined method to start new task.
MPI -2 contain functions to start a group of tasks
and to send a kill signal to a group of tasks

Resource Control
PVM is inherently dynamic in nature, and
it has a rich set of resource control
functions. Hosts can be added or deleted
load balancing
task migration
fault tolerance
efficiency

MPI is specifically designed to be static in

nature to improve performance

Virtual topology
- only for MPI
Convenient process naming
Naming scheme to fit the communication pattern
Simplifies writing of code
Can allow MPI to optimize communications

Virtual topology example

A virtual topology of twelve processes - grid with a cyclic boundary

condition in one direction e.g. processes 0 and 9 are ``connected''.
The numbers represent the rank and the conceptual coordinates mapped
to the ranks

Message Passing operations

MPI : Rich message support
PVM: Simple message passing

Point-to-Point communications

A synchronous
communication does not
complete until the message
has been received.

An asynchronous
communication completes as
soon as the message is on
its way

Non-blocking operations

Non blocking communication allows useful work to be performed

while waiting for the communication to complete

Collective communications

Broadcast
A broadcast sends a
message to a number of
recipients

Barrier
A barrier operation
synchronises a
number of
processors.

Reduction
operations
Reduction operations
reduce data from a
number of processors
to a single item.

Fault Tolerance: MPI

MPI standard is based on a static model
If a member of a group failed for some
reason, the specification mandated that
rather than continuing which would lead to
unknown results in a doomed application,
the group is invalidated and the
application halted in a clean manner.
In simple if something fails, everything
does.

Fault Tolerance: MPI

Fault Tolerance: MPI

Failed Node

There is a failure and

Fault Tolerance: MPI

Failed Node

the application is shut down

Fault Tolerance: PVM

PVM supports a basic fault notification scheme:
it doesnt automatically recover an application
after a crash, but it does provide notification
primitives to allow fault-tolerant applications to
be built
The Virtual Machine is dynamically
reconfigurable
A pvmd can recover from the loss of any foreign
pvmd except the master. The master must
never crash

Fault Tolerance: PVM

Virtual Machine

Fault Tolerance: PVM

Failed Node

Virtual Machine

Fault Tolerance: PVM

Virtual Machine

Fast host delete or recovery from fault

Conclusion
Each API has its unique strengths

PVM

MPI

No such abstraction
Rich message support
Support logical communication
topologies
Some realizations do not
interoperate across architectural
boundaries
Portability over performance Performance over flexibility
Primarily concerned with
Resource and process
messaging
control
More susceptible to faults
Robust fault tolerance

Virtual machine concept

Simple message passing
Communication topology
unspecified
Interoperate across host
architecture boundaries

Conclusion
Each API has its unique strengths

PVM is better for: MPI is better for:

Heterogeneous cluster, Supercomputers (PVM is
resource and process
not supported)
control
Application for MPP
The size of cluster and
the time of programs
Max performance
execution are great
Application needs rich
message support

Acknowledgments
Scientific adviser
Assoc. Prof. A.V.Komolkin

???

Thank you for your attention!