CST STUDIO SUITE TM

2010
MPI Computing Guide

Contents
1 Introduction 2

2 Nomenclature 2

3 Terms 3

4 Technical Requirements 3
4.1 Interconnection Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.1.1 Network Technology . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.1.2 Name Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.1.3 Unique Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.2 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.3 Daemon/Service User Account . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.4 Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

5 Supported Solvers and Features 5

6 Installation of CST STUDIO SUITE on a Cluster System 5

6.1 Installation of MPI Frontend Node . . . . . . . . . . . . . . . . . . . . . . 5
6.1.1 Installation on Windows . . . . . . . . . . . . . . . . . . . . . . . . 6
6.1.2 Installation on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6.2 Installation of MPI Client Node . . . . . . . . . . . . . . . . . . . . . . . . 8
6.2.1 Installation on Windows . . . . . . . . . . . . . . . . . . . . . . . . 8
6.2.2 Installation on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . 9
6.3 Testing your Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

7 Starting MPI Simulations 9

7.1 Starting MPI Simulations in Interactive Mode . . . . . . . . . . . . . . . . 9
7.2 Starting MPI Simulations from Command Line . . . . . . . . . . . . . . . 10

8 Updating Your Cluster 10

8.1 MPI Frontend Node = Windows, MPI Client Nodes = Windows or Linux . 11
8.2 MPI Frontend Node = Linux, MPI Client Nodes = Linux . . . . . . . . . . 12

9 Licensing 12

10 Troubleshooting 13
10.1 No Connection to MPI Nodes . . . . . . . . . . . . . . . . . . . . . . . . . 13
10.2 No More TCP Connections Possible . . . . . . . . . . . . . . . . . . . . . . 13

January 6, 2010 1
1 Introduction
The term ”MPI Computing” describes a software parallelization approach for cluster sys-
tems, i.e. computers connected via a fast interconnection network and used as a single
”supercomputer”. A computational task, like an EM simulation, is divided into several
parts and those parts are handled by the machines in the cluster in parallel. Typically, the
parts are not completely independend and, thus, the different computers need to exchange
data via the interconnection network and, additionally, have to be synchronized at certain
points. For this purpose the so-called ”Message Passing Interface” (MPI), which describes
a programming interface, has become a de facto standard and because of its importance
this type of parallel computing is often refered to as ”MPI Computing”. This manual con-
tains an overview of the MPI Computing capabilities of CST STUDIO SUITETM as well
as detailed installation instructions which help you to properly install and set up parallel
simulations on your cluster system. Please do not hesitate to contact us in case of any
problems during the setup process or during the simulations itself. In such a case please
send detailed information about the problem you are facing to info@cst.com.

2 Nomenclature
The following section explains the nomenclature used in this document.

command Commands you have to enter on a command prompt (either cmd on

MS Windows R
or your favorite shell on Linux) typeset using type-
writer fonts.

<...> Within commands the sections you should replace according to your
environment are enclosed in ”<...>”. For example ”<CST DIR>”
should be replaced by the directory where you have installed CST
STUDIO SUITETM (e.g. ”c:\Program Files\CST STUDIO SUITE
2010”).

Those icons indicate that the following section of the text applies
only to a certain operating system:

= MS Windows
R
= Linux

January 6, 2010 2
3 Terms
The following section contains the definition of some important terms needed to under-
stand the rest of the manual.

Cluster Computers connected via a fast interconnection network

and used as a single ”supercomputer”.
Node An arbitrary computer in your cluster system.

MPI Frontend Node The cluster node on which the simulation is started using
the CST front end (CST DESIGN ENVIRONMENTTM ).
MPI Client Node A cluster node which takes part in an CST MPI simulation.
An MPI Frontend Node can be an MPI Client Node
at the same time.

4 Technical Requirements
This section provides you with information about the hardware as well as the software
requirements necessary to successfully perform CST simulations on your cluster system.
Please ensure that your system is configured according to the following points to avoid
problems during the setup or operation of CST STUDIO SUITETM on your system.

4.1 Interconnection Network

4.1.1 Network Technology
The nodes in your cluster must be connected via a TCP/IP network (Ethernet). Native
support for such high speed interconnection networks is planned for the near future. If
your cluster is equipped with such a high speed interconnection network make sure that
the nodes can also communicate via a TCP/IP link (either Ethernet or, e.g., IP over In-
finiband).

4.1.2 Name Resolution

The names of the cluster nodes must be correctly resolved to IPv4 addresses (not IPv6
addresses) either using a DNS or appropriate entries in the ”hosts” file of your nodes if
you use static IP addresses.

4.1.3 Unique Network

All nodes participating in an MPI simulation must be part of the same network, i.e. the
MPI Frontend Node must not be a machine outside the cluster which cannot directly
access the cluster nodes. A simple check if all the cluster nodes are in the same network

January 6, 2010 3
can be performed using the ping command. To check if node A can ”see” node B just
enter the following command on a terminal on node A:
ping <node B>
If node B answers correctly the two nodes are in the same network and can communicate
during an MPI simulation.

4.2 Hardware Requirements

There are no special hardware requirements for the cluster nodes. However, if you intend
to use GPU Computing together with MPI please refer to the GPU Computing Guide to
learn more about the special requirements for the GPU hardware.

4.3 Daemon/Service User Account

In order to start simulation processes on the nodes a CST daemon/service must be running
on all nodes which take part in the parallel simulation. This daemon/service runs under a
certain user account. Thus, please make sure that the user account you would like to run
the daemon with exists prior to the installation of the CST software. Because of security
reasons it is recommended to choose a user account without administrator/root privileges
for this purpose.

4.4 Operating System

The following operating systems are supported (32 bit and 64 bit versions).
• Windows XP
• Windows Vista
• Windows Server 2003 and Windows Server 2003 R2
• Windows Server 2008
• Red Hat Enterprise Linux 4 and 5
Additionally, MPI Client Nodes must be homogeneous in terms of the operating system
(either MS Windows (32/64 bit) or Linux (32/64 bit)). The following configurations are
currently supported:
MPI Frontend Node MPI Client Nodes

January 6, 2010 4
5 Supported Solvers and Features
Currently not all solvers and not all features are supported for MPI Computing. The fol-
lowing table gives you an overview of the solvers which currently support MPI Computing
and of the features currently missing for those solvers. Additionally, the case of combined
MPI Computing and GPU Computing is handled for the transient solver. The other solvers
do not support combined MPI Computing and GPU Computing.

MPI+GPU
CST Product Solver Supported? Unsupported Features

CST MICROWAVE STUDIO

R
T-Solver Yes • Periodic Boundaries
• Network Co-simulation
• Subgridding
• Farfield Probes in case of
combined MPI and GPU

CST MICROWAVE STUDIO

R
I-Solver No Only the direct solver is sup-
ported.

CST PARTICLE STUDIO

R
WAK-Solver No • Indirect wake integration
over interfaces
• The features not sup-
ported for the T-solver
are also not supported in
the WAK-solver.

6 Installation of CST STUDIO SUITE TM

on a Cluster
System
The following section provides you with step-by-step installation instructions for several
typical cluster setups. There are different install sets needed to install an MPI Frontend
Node and an MPI Client Node.

6.1 Installation of MPI Frontend Node

Please remember that an MPI Frontend Node is a machine in your

cluster which you want to be able to start CST MPI simulations from.
Thus, if you want to be able to start CST MPI simulations from an
arbitrary node in your cluster (e.g., because this is the requirement of an
external job queuing system you are using), then each of your cluster
nodes is an MPI Frontend Node and you need to perform the following
steps on each of your nodes.

January 6, 2010 5
6.1.1 Installation on Windows
An MPI Frontend Node needs a full installation of the CST STUDIO SUITETM program
files. Thus, install at least the feature set shown in figure 1 on each Windows node of your
cluster which should be able to start a CST MPI simulation. Note that the ”License
Server” as well as the ”Dongle Driver” option must only be selected if the node should
act as license server for your installation.1 To start MPI simulations on other Windows

Figure 1: Please install at least the following install set on an MPI Frontend Node
running on Windows.

machines you have to provide the credentials of an user account which will be used by
CST STUDIO SUITETM to log in to the MPI Client Nodes and start the simulation
processes. Make sure that the user account you provide is known by MPI Client Node
machines. This is typically ensured if the user account is a domain account. However, if
you are providing a local user account for this purpose please make sure that this account is
defined on all MPI Client Nodes identically and the password for the account is also the
same on all nodes. To register the user account please open a terminal (Start→Run→cmd)
and go to the installation directory of CST STUDIO SUITETM then enter the command:

mpiexec -register
1
It will not harm the setup if you select the other features as well.

January 6, 2010 6
You will then be asked for the user credentials and the information you provide will be
written to the registry (see fig. 2). Please note that this step is mandatory even if your
MPI Client Nodes are all running on Linux. In this case, the user credentials behave just
as a ”dummy” setting. Actually, a different authentication method based on a passphrase
is used to log in and start processes on MPI Client Nodes running a Linux operating
system.

Figure 2: Register the user credentials.

6.1.2 Installation on Linux

Please notice that the following workflow assumes that you own a DVD which contains
the complete Linux version of CST STUDIO SUITETM . The standard DVD containing
the Windows version does not contain all the files necessary for the Linux MPI Frontend
Node installation (it contains only the files needed to install an MPI Client Node).
Please contact your CST sales representative or send an email to info@cst.com if you need
the DVD containing the complete CST STUDIO SUITETM files for Linux systems.
An MPI Frontend Node needs a full installation of the CST STUDIO SUITETM program
files. Thus, install at least the feature set shown in figure 3 on each Linux node of your
cluster which should be able to start a CST MPI simulation. Note that the ”License
Server” as well as the ”Dongle Driver” option must only be selected if the node should act
as license server for your installation.2 If your MPI Frontend Node should also act as
an MPI Client Node please also select the ”MPI Client Node” option.
2
It will not harm the setup if you select the other features as well.

January 6, 2010 7
 Figure 3: Linux install set for MPI Frontend Node. If the node will also act as license
server please select additionally the option ”License Server” as well as ”Dongle Drivers”. If
the MPI Frontend Node should also be an MPI Client Node select additionally the
option ”MPI Client Node”.

6.2 Installation of MPI Client Node

6.2.1 Installation on Windows

To install a machine as MPI Client Node two steps are necessary. The first step is to
install the CST MPI service which allows the MPI Frontend Node to connect to the
client nodes using the user credentials you provided in section 6.1.1. For this purpose a
special installer executable is provided within the CST installation directory on your MPI
Frontend Node. This executable is called ”mpi install.exe”. Double clicking on the
executable will start the installation process. Please follow the instructions of the installer.
Please note that the CST MPI service will be started using the user account of the user
who installed it. For security reasons you may consider using a non-administrator account.
The second step in the installation process is to make the solver executables accessible. This
can be performed in two different ways. The former way is to share the CST installation
folder on your MPI Frontend Node such that this folder may be accessed from the MPI
Client Node (make sure that the user account you’ve registered in section 6.1.1 has at
least read access to the shared folder). The latter way is to perform an installation of the
CST STUDIO SUITETM executables on the MPI Client Node as well (the same install
set as for the MPI Frontend Node in section 6.1.1).

January 6, 2010 8
6.2.2 Installation on Linux
To install a machine as MPI Client Node, please execute the Linux installer which can
be found on both the Windows and Linux DVD. Select the install option ”MPI Client
Node”3 . This will install all Linux solver executables as well as the daemon necessary to
start CST MPI jobs. If the MPI Client Node should also act as MPI Frontend Node
please refer to section 6.1.2. Please note that the CST MPI service will be started using the
user account you provide during the installation. For security reasons you may consider
using a non-root account.

6.3 Testing your Installation

As soon as you have finished the installation, it is a good idea to test if all your nodes are cor-
rectly configured. Please open the CST STUDIO SUITETM frontend on your MPI Fron-
tend Node. Either create a small test model, e.g., using ”Macros→Construct→Online”
or load a model you would like to test. Open the ”MPI Computing” dialog of your solver
(e.g. ”Solve→Transient Solver...→Acceleration→MPI Computing”). Enter the names of
your MPI Client Nodes as well as the CST installation directory on each node4 , and a
temporary directory which will be used to store files on the MPI Client Nodes during
the simulation (Please note that this directory must already exist on the nodes as it will not
be created automatically!). You can either enter this information for each node manually
or just enter global settings and apply them to all nodes stored in the list by right-clicking
on the head of each column and selecting ”Inherit Default”. Then click on ”Test active
nodes” to test all nodes in your list or right-click on the name of a certain node and select
”Test Node” in the context menu to test a special node, respectively. This test checks the
configuration of each active node in terms of the MPI simulation requirements, i.e. the
given hostname, installation folder, and temporary folder. Depending on the number of
nodes the test may last for some seconds. If the test was successful you can start your first
MPI simulation.

7 Starting MPI Simulations

7.1 Starting MPI Simulations in Interactive Mode
To activate MPI Computing the MPI option has to be checked in the ”Acceleration” set-
tings of the solver. In addition to this the solver nodes have to be specified in the ”MPI
Properties”. For each node the CST installation folder, a temporary folder, and an archi-
tecture has to be specified.5 The cluster settings can be saved to a file with the ”Export”
3
If your MPI Frontend Node is also running on Linux please select the ”GUI Frontend” option as
well, as this is necessary to be able to update your installation later on with the service packs provided by
CST.
4
In case of a shared folder (see section 6.2.1) enter the UNC path (network path) to this shared folder.
5
Currently only homogeneous architecture settings for the MPI Client Nodes are supported.

January 6, 2010 9
button. The settings can then be loaded afterwards using the ”Load File...” button. This
exported file corresponds to the CST machine file format described in section 7.2.

7.2 Starting MPI Simulations from Command Line

If you start your simulation from the command line you can either use the MPI settings
previously made in the ”MPI Properties” dialog of your model as described in section 7.1
or provide the MPI settings using a machinefile. A machinefile can be created by the
”Export” function in the MPI dialog but also a machinefile created by an external job
queuing system can be used.6 To start your simulation from the command line using a
machinefile you can specify the option
-withmpi="<FULL PATH TO MACHINEFILE>;CST"
which will load the MPI settings from the specified machinefile (CST format).7 The quotes
are mandatory. The format of the CST machinefile is as follows:

# CSTMachineFile 20080509
# Template for machine configuration in internal CST format.
# The very first line should be as shown, with version numbering.
# It is possible to set common default settings, to be applied to
# all nodes in the cluster, unless differently specified.
# All lines beginning with "#" are comments
#
# The default settings are specified here:
#
# DefaultInstallationFolder <PATH_TO_INSTALLATION_FOLDER>
# DefaultTempFolder <PATH_TO_TEMP_FOLDER>
# DefaultMachineArchitecture <ARCHITECTURE>
#
# Then each machine node is specified as follows:
# HostName # <PATH_TO_INSTALLATION_FOLDER> ; <PATH_TO_TEMP_FOLDER> ; <ARCHITECTURE> ; <ActiveFlag>
#
# Architecture can have following values:
# Windows IA32, Windows AMD64, Linux IA32, Linux AMD64
#
# The <ActiveFlag> can have the following values: Active, Inactive
#
# The following list provides an example for a valid list of machines
# machine1 uses its own settings
machine1 # "C:\Program Files (x86)\CST STUDIO SUITE 2010" ; "C:\temp" ; Windows AMD64 ; Active
# machine2 uses the default settings
machine2 # ; ; ; Active

8 Updating Your Cluster

The software version of your CST installation must be identical on all nodes of the cluster.
The way how you install the service packs you get from CST depends on the configuration
of your cluster system, i.e. the operating system installed on the nodes.
6
Please refer to the Application Note ”Interaction of CST STUDIO SUITE 2010TM with External
Job Queuing Systems” to learn more about the correct configuration of your queuing system for CST
simulations.
7
Please note that the settings in the machinefile have precedence over the settings made in the ”MPI
Computing” dialog.

January 6, 2010 10
8.1 MPI Frontend Node , MPI Client Nodes
In case you have set up your cluster such that the CST STUDIO SUITETM files are accessed
by the MPI Client Nodes using a shared folder you only need to update this single shared
installation. This may be performed automatically via the internet or manually using the
CST Update Manager.
In case you have performed a local installation of CST STUDIO SUITETM on each node
you can install the service pack on the MPI Frontend Node and then send the update to
the MPI Client Nodes using the following procedure. First update your MPI Frontend
Node using the automatic update or the CST Update Manager. This updated installation
will now act as a reference installation for your MPI Client Nodes. The cluster update
uses the Distributed Computing (DC) system. Thus, you need to start the CST DC Main
Controller on your MPI Frontend Node. For this purpose start the ”Main Controller
Administration” found in the Windows Start Menu folder of your CST installation. In the
DC Main Controller enter the path to your updated CST STUDIO SUITETM installation
in the ”Reference Installation” field and a temporary folder. Then start the DC Main
Controller by clicking on ”Start Server”. Figure 4 shows you how the dialog looks like.
After you have started the DC Main Controller open the ”MPI Computing” dialog and
press the ”Select Main Controller” button. Enter the name of your MPI Frontend
Node where you have started the DC Main Controller before and press OK. Then press
the ”Update Active Nodes” button to send the updated executables to the active MPI
Client Nodes.

Figure 4: The DC Main Controller GUI must contain the path to your reference installation
as well as a temporary folder.

January 6, 2010 11
8.2 MPI Frontend Node , MPI Client Nodes
Please note that the update of your Linux installation can only be performed either by
root or by the ”Main User”.8 The update of your Linux installation can be performed
using a script called ”update with supfile” which can be found in your CST installation
folder. Download the service pack from the CST fileserver and make the file accessible to
all your cluster nodes. Log in as root or as ”Main User” to all the cluster nodes and enter
the command

./update with supfile "<FULL PATH TO SUPFILE>"

Please note that the full path to the file is required and that the quotes are mandatory.

9 Licensing
The number of acceleration tokens in your license determines the maximum number of
nodes which you can use in a single simulation. The number of tokens required to perform
a simulation is listed in the ”Acceleration” dialog of your solver. Table 1 shows you the
maximum number of MPI nodes with a certain number of tokens. Simulations on clusters
with more than 32 nodes are currently not supported.

Number of Tokens Max. Number of MPI Nodes

0 1
1 2
2 4
3 8
4 16
5 32

Table 1: The table shows the maximum number of MPI nodes with a certain number of
tokens.

8
This is the user account you specified during the installation process.

January 6, 2010 12
10 Troubleshooting
10.1 No Connection to MPI Nodes
If you cannot connect to the MPI Client Nodes make sure that the firewall on your
nodes is configured properly. The following programs need access to the network:

• cstsmpd

• ParaControl

• mpiexec

This applies to the 32 bit versions as well as to the 64 bit versions of the programs found in
the AMD64 subfolder of your installation (on Windows) or in the LinuxAMD64 subfolder
(on Linux). Additionally, make sure that the hostnames are resolved properly to IPv4
addresses. If even the command ”ping <remote host>” fails this is an indication that
your network is not correctly configured.

10.2 No More TCP Connections Possible

If your simulation seems to start and aborts after a while with a timeout error this might
be due to a limitation of your operating system. Windows XP is quite restrictive regarding
the number of TCP connections which can be established at a time. Thus, increasing the
number of TCP connections allowed at a time probably solves this problem. Please ask
your system adminstrator to increase this limit.

January 6, 2010 13