ALESSIA:

EP 28189 Application of Large Eddy Simulation to the

Solution of Industrial problems:
BEST PRACTICE GUIDE: LES AND ACOUSTICS

Issue 1
 ALESSIA/AEA/Best Practice Guide Issue 1

ALESSIA:

EP 28189 Application of Large Eddy Simulation to the

Solution of Industrial problems:
BEST PRACTICE GUIDE: LES AND ACOUSTICS

Authors

Christiane Montavon*, Ian Jones*, Dick ten Bosch+, Stefan Szepessy †, Roland Henriksson†,
Hans Moberg†, Roberto Tregnago ‡, Zoubida El Hachemi #, Massimiliano Piccirillo#, Michel
Tournour#, Sylvie Dequand#, Frederic Tremblay !, Rainer Friedrich!

* AEA Technology Engineering Software

+ Shell International Oil Products
‡ Centro Ricerche Fiat
# LMS Numerical Technologies
† Alfa Laval Separation
! Technical University, München

Issue 1

Internal Version 1.4

Date 18/6/2002

AEA Technology Engineering Software ii

 ALESSIA/AEA/Best Practice Guide Issue 1

SUMMARY
Time dependent flow phenomena play an important role in many industrial processes,
including aerodynamic noise generation. In many cases experimental and theoretical analyses
of the problems are not feasible, either due to their high expense or to extreme operating
conditions. The primary aim of the Alessia Project has been to develop validated and
supported software tools for the prediction of industrially important fluctuating flow
problems. The tools are based around the technique of Large Eddy Simulation (LES), which
solves for the large-scale fluctuating flows, the Large Eddies, and uses ‘sub-grid’ scale
turbulence models for the small-scale motion. The leading European software packages, CFX
for Computational Fluid Dynamics (CFD), and SYSNOISE for the acoustics, have been
coupled together in this project to perform the prediction of industrially relevant flows, and
the consequent noise field. The main limitation to the solution of these problem has been the
computational cost of computer predictions. HPCN has been an enabling technique that has
provided the solutions for these multi-physics and multi-scale problems.

This report is a subset of one of the deliverables of the project and summarises some of the
experiences learnt from the project, in the form of a Best Practice Guide. It also contains some
considerations for the choice of computing system for LES computations.

AEA Technology Engineering Software iii

 ALESSIA/AEA/Best Practice Guide Issue 1

CONTENTS

1 INTRODUCTION 1

2 THE OVERALL STRATEGY 1

3 WHEN TO USE LES 2

4 RECOMMENDATIONS FOR LES PROBLEM SET UP 3

4.1 Boundary Layers 3

4.2 Inlets 3

4.3 Outlets 3

4.4 Geometrical Domain 4

4.5 Meshes 4

4.6 Numerical Considerations 4

4.7 Advection scheme 4

4.8 Algorithmic Considerations, Time stepping 4

4.9 Accelerating Convergence 6

4.10 Choice of LES model 6

4.11 Storage Of Results And Back Up 6

5 AEROACOUSTIC CALCULATIONS 7

5.1 SYSNOISE modules 7

5.2 Harmonic Acoustic FEM 7

5.3 Coupled Harmonic FEM 8

5.4 Acoustic I-FEM 8

5.5 Harmonic Acoustic BEM 8

5.6 Coupled Harmonic BEM 9

5.7 Numerical Consideration 9

5.7.1 Mesh coarsening 9

AEA Technology Engineering Software iv

 ALESSIA/AEA/Best Practice Guide Issue 1

5.7.2 Free edges 9

5.7.3 Junctions 9

6 SYSTEM REQUIREMENTS & CHOICE OF PLATFORMS 10

6.1 Operating environment 11

6.2 STORAGE OF RESULTS AND BACK UP 11

7 REFERENCES 12

AEA Technology Engineering Software v

 ALESSIA/AEA/Best Practice Guide Issue 1

1 Introduction
Time dependent flow phenomena play an important role in many industrial processes,
including aerodynamic noise generation. In many cases experimental and theoretical analyses
of the problems are not feasible, either due to their high expense or to extreme operating
conditions. The primary aim of the Alessia Project [ 1] has been to develop validated and
supported software tools for the prediction of industrially important fluctuating flow
problems. The tools are based around the technique of Large Eddy Simulation (LES), which
solves for the large-scale fluctuating flows, the Large Eddies, and uses ‘sub-grid’ scale
turbulence models for the small-scale motion. The leading European software packages, CFX
for Computational Fluid Dynamics (CFD), and SYSNOISE for the acoustics, have been
coupled together in this project to perform the prediction of industrially relevant flows, and
the consequent noise field. The main limitation to the solution of these problems has been the
computational cost of computer predictions. High Performance Computing and Networking
(HPCN) has been an enabling technique that has provided the solutions for these multi-
physics and multi-scale problems.

A summary of Validation results for this project are to be found in [ 2]. The partners in the
project have combined their experiences on these problems to produce a Best Practice Guide
and Recommendations. This focuses on:

• When to consider the use of LES, instead of conventional Reynolds Averaged modelling.
• How to use CFD software for large-scale transient simulations in an HPC environment.
This will include the recommendations on solver parameters, for efficiency.
• Recommendations for size of cluster, and cluster configurations, speed of interconnection,
memory requirements for different size problems and applications.
• Description of the parallel algorithms, and the specific issues on the running of large
dynamic transient cases

2 The Overall Strategy

Turbulent flows are notoriously difficult to predict, and they are one of the Grand Challenge
problems of science and engineering. The usual approach is to use the ‘Reynolds Averaged
Navier –Stokes’ equations, (RANS), which solve for time averaged quantities. However, there
are some situations where the approach is not adequate, and the alternative approaches of
Large Eddy Simulation (LES) or Direct Numerical Simulation (DNS) can be adopted. With
these methods, time dependent equations are solved for the chaotic motion with either no
approximations and all relevant scales resolved (DNS) or the equations are filtered in some
way to remove very fine time and length scales (LES). These approaches require fine grids
and can give very good results. In addition, they can give details on the structure of the
turbulent flows, such as pressure fluctuations and Lighthill stresses, which cannot be obtained
in other ways. This has always required the use of tuned special purpose software, using the
largest supercomputers available. This type of software is also usually very restrictive in the
kind of geometries that can be handled, with simple regular grids used to resolve the
geometries. This has restricted the problems that can be typically solved to cylinders and
boxes, with ‘immersed boundary methods’ (blocking off cells) to deal with complex

AEA Technology Engineering Software 1

 ALESSIA/AEA/Best Practice Guide Issue 1

geometries. This has wasted significant memory because of the blocking off. This project has
attempted to use the latest generation of commercial CFD software, with powerful methods
for dealing with complex geometries in conjunction with LES. Because of the difficulty of
validating the methods for solving problems which are inherently chaotic, the strategy has
been to use very accurate solutions from the MGLET software, from the Technical University
of Munich, to provide benchmark quality data for testing and evaluating the industrial
software. This has inevitably required the use of the largest supercomputers available in
Europe, for the benchmarking, along with lower cost ‘industrial strength’ systems, based on
commodity workstations, for the industrial computations.

3 When to use LES

This section outlines the situations where the underlying LES methodology can be used.

For Low Reynolds numbers, < 5000, consider DNS if you have a large computer available,
and it is important to be able to resolve the scales, for example for transition to turbulent flow.

For Higher Reynolds numbers, LES might be a suitable option for cases where:
• the flow is likely to be unstable, with large scale flapping of a shear layer or vortex
shedding.
• the flow is likely to be unsteady with coherent structures (cyclone, flasher)
• the flow is buoyant, with large unstable regions created by heating from below, or by
lighter fluid below heavier fluid (multi-phase flows in inclined pipelines).
• conventional RANS approach are known to fail (for example due to highly anisotropic
turbulence)
• a good representation of the turbulent structure is required for small-scale processes
such as micro-mixing or chemical reaction.
• the noise from the flow is to be calculated, and especially when the broadband
contribution is significant.
• other fluctuating information is required (e.g. fluctuating forces, gusts of winds)
• the user can afford to wait for up to a week for results, using an 8-16 processor
system.

Note that if the user has a simple geometry, requires good quality results, and has access to
large computers, he should consider the use of a special purpose code such as MGLET, rather
than a less accurate commercial package.

For more complex geometries however, a commercial package with LES should be more
suitable.

AEA Technology Engineering Software 2

 ALESSIA/AEA/Best Practice Guide Issue 1

4 Recommendations for LES problem set up

4.1 BOUNDARY LAYERS
Is the boundary layer structure important: If so, resolve it, with at least 10 points across the
boundary layer, and with the first grid point at a position of y+ around 1. The Technical
University of Munich’s experience is that for accurate LES calculations, they cannot gain
much over a Direct Numerical Simulation (DNS) and would recommend at least 30 points as
a minimum. In planes parallel to a wall, it is possible to use coarser grids, provided the flow
remains attached.

4.2 INLETS
The representation of the turbulent structures at inlets can be a difficult part of the setup. As
could be seen from the NTH case, the detailed properties of the incoming flow had a strong
effect on the development of the jet in the enclosure.

For the case of the cyclone however, the turbulent conditions at the inlet seemed to have
relatively little impact on the flow in the device, which was essentially determined by very
strong anisotropy effects in the cyclone body.

Cases for which the inlet turbulence plays a significant role are developing boundary layers
and turbulent jets.

If the inlet turbulence is felt to play an important role, the user should consider
a) Going upstream, with an appreciable region of the inlet resolved to use LES for the
flow upstream.
b) Use of pre-computed LES flows, storing only the values on a plane
c) Use of SNGR model and RANS models to generate inlet flows with the right turbulent
structures.
d) New methods such as Detached Eddy Simulation (DES) show promise, but they are
not fully proven yet. In particular, there could be upstream effects from the LES
portion, or the turbulent fluctuations in the upstream boundary layer could be
necessary to trigger off the unstable shedding of vortex sheets. This is a new area, and
more experience is required. Beware the claims of the originators of new methods,
since they may not report the bad experiences, and the difficulties encountered.

4.3 OUTLETS
With the transport of turbulent eddies outside of the computational domain, some
recirculation can occur at outlets. Our experience shows that if the code is allowed to bring
some fluid back into the domain at outlets, it can destabilise the solution. We therefore
recommend in CFX-5 to use OUTLET boundary conditions rather than OPENING when
performing a LES. OPENING type of boundary conditions allow the flow to come back in,
whereas with OUTLET boundary conditions, the code builds artificial barriers at the
boundary, when the flow tries to come back in. These barriers are later taken away when the
flow goes out again.

AEA Technology Engineering Software 3

 ALESSIA/AEA/Best Practice Guide Issue 1

The use of artificial walls at outlets might introduce some not very physical behaviour locally,
but it increases the robustness of the calculation.

4.4 GEOMETRICAL DOMAIN

Even though there might be symmetries in the geometry and flow, the geometrical model
should include the full region, since while the time averaged flows may be symmetric, the
instantaneous flows are not, and restricting the domain could constrain the turbulence. Two-
Dimensional Approximations are particularly bad.

4.5 MESHES
Remember that the mesh and time steps are an inherent part of the model. The LES models
make use of the grid scale for filtering out the turbulence. If the mesh is anisotropic to resolve
a jet, for example, the longer length scale in the flow direction may also have an undue effect
on the cross-stream turbulence. For this reason, consider the use of isotropic grids, maybe
using tetrahedral elements, rather than hexahedral meshes.

4.6 NUMERICAL CONSIDERATIONS

VLES Models, (Very Large Eddy Simulations) can be considered when small-scale
turbulence is not important, and large-scale structures are naturally unstable. An example of
this is for the vortex shedding from a cylinder.

4.7 ADVECTION SCHEME

• Where possible, methods should be at least second order accurate in space. Second
order CENTRAL DIFFERENCE methods have certainly given good answers within
the Alessia project, both in CFX-5 and MGLET.
• Second order ‘Higher Upwind’ Differencing Methods and the QUICK scheme, based
on backward difference methods appear to be a little too diffusing, and can damp out
turbulence. Similarly, almost second order accurate methods, with a flux limiter to
prevent overshoots also seem to be too diffusive. Some Japanese experience, however,
indicates that QUICK, in conjunction with no turbulence model, appears to give good
results in some cases. This is not recommended by the results from the Alessia project.

4.8 ALGORITHMIC CONSIDERATIONS, TIME STEPPING

• Tuned LES solvers conventionally use explicit methods, with central differencing in
space, with second or fourth order methods both in space and time, with regular
rectangular grids. Very simple, Smagorinsky based models are typically used. These
are best for highly accurate solutions, which challenge computing limits. With these
methods there is a reluctance to use methods that transport turbulence quantities, for
issues such as memory usage. This limits the method to length scales such that the
transport of sub-grid scale turbulent kinetic energy is not important.

AEA Technology Engineering Software 4

 ALESSIA/AEA/Best Practice Guide Issue 1

• For tuned LES explicit time-stepping programs, the dominant cost is that of solving a
Poisson equation at each time step, for the pressure field. This can still be very time
consuming.
• CFX-5 has the advantages for complex geometries, where the use of a body-fitted or
unstructured grid is necessary, either to resolve the geometry, or to resolve the flow (at
steps, near separation points, in boundary layers). It will also be able to make use of
new models being developed, eg for combustion or multi-phase flows, and provide
commercial support and infrastructure.
• Spectral and pseudo-spectral methods can give the most accurate results, but they are
very limited in scope, and when there is insufficient grid resolution, the results can
have large errors.
• Use second order time stepping. 2nd order backward differencing seems to be
relatively robust, and gives the right behaviour, despite the fact it is not an ‘A-stable’
method. That is, it can have different stability properties to that of the underlying
differential equation.
• Second order centred time stepping, like Crank-Nicolson, leapfrog, are more difficult
to run, and are prone to having different solutions on alternate time steps.
• First order fully implicit methods in time usually are too diffusive, and the turbulence
is damped out. For highly unstable problems, such as the cyclone, lower order
methods may work, but the results will be very damped, unless very small time steps
are used.
• For Explicit methods, keep CFL condition below 0.25. For a non-uniform grid, this
can restrict the time step to be constrained by the smallest grid size in the problem,
even if it is not in an important region of the flow. Refining the mesh also means that
much smaller time steps have to be taken, to maintain the CFL condition.
• Implicit methods can have a higher CFL condition, especially in regions that are not
particularly important, and the grid size is small for geometric considerations.
However, for accuracy, CFL numbers of Order 0.5 – 1 should be used. Larger values
can give stable results, but the results may be damped.
• For smaller time steps, with an implicit coupled solver as in CFX-5, one iteration
(Coefficient loop) can be carried out at each time step, to give a semi-implicit method.
This method may not be as stable as using two or three iterations. When the flow
properties have settled down, then the method may be better1.
• A large proportion of computing time is spent getting to the stage when it is possible
to start taking statistics. If this time can be shortened, faster results can be obtained.
This is discussed in more detail later.
• 1,000 – 10,000 time steps are typically required for getting converged statistics. More
steps are required for second order quantities, (variances) than for means. Check the
convergence of the statistics. For a vortex-shedding problem, several cycles of the
vortex shedding are required. This effect has also been seen in experimental results,
where the evidence of the results of TUM indicate that measurements should have
been taken for a longer time scale.

1
When more complex cases are considered with e.g. mixing of multicomponent flows, it is advised to perform
more coefficient loops per time step (typically 6-8). For these cases, unless the mass fraction equations are solved in
a fully coupled way together with the momentum and continuity equation, the mass conservation at the
component level will not be guaranteed with only one coefficient loop per time step.

AEA Technology Engineering Software 5

 ALESSIA/AEA/Best Practice Guide Issue 1

4.9 ACCELERATING CONVERGENCE

As indicated above, it can take a lot of time steps to get to the stage where it is possible to
accurately compute statistical quantities. It is necessary for the flow to have all the right
length scales for the turbulent flows, and this involves the cascade of turbulence from large
scales to small scales, and maybe associated back-scatter, from small scales to larger scales.

Consider the following ways of accelerating the development of statistically steady flows:
• Use the results from a similar case to start the flow
• Use less accurate methods with larger time steps to allow the large scale structures to
develop
• Use results from a RANS calculation to give the right mean fields.
• Use the SNGR model to give the flow the correct turbulent length scales.
• Get convergence on a coarse mesh, and interpolate the results onto a fine mesh. This
should retain the larger turbulent scales, and the smaller scales will develop more
quickly.

4.10 CHOICE OF LES MODEL

• The Smagorinsky model contains one free parameter, c0 that has a default value of 0.1.
The standard value has been giving good results in most applications, although it
appears that for boundary layer flows and for jets, a lower value can improve
agreement with measurements (e.g. CRF test case, NTH test case). It has however
been observed that it is easy to blame the model constant, when some other
shortcomings like a bad representation of the upstream conditions can affect the flow
just as much as a change in the constant (e.g. NTH case).
• To use dynamic models such as Germano, which compute the model coefficient
incorporating explicitly filtered resolved fields, the resolution needs to be sufficiently
good to give better results than Smagorinsky. Any zero order turbulence model will
have the same considerations.
• It is sometimes argued that it is important for the models to allow for backscatter
(energy cascading from the smaller to the larger scales close to walls) to improve the
simulation of boundary layers. Models such as INC & DINC developed by TUM have
shown encouraging results during a priori testing. These models however when used
in a simulation still require arbitrary clipping of the viscosity in order to be stabilised.
These models still lack the required robustness to be implemented in a commercial
package.

4.11 STORAGE OF RESULTS AND BACK UP

During the course of a calculation, we typically recommend to do a full backup of the results
every 50 time steps. At the end of the computations, there will then be two files, a full results
file at the end of the run, and a separate one created earlier in the run, at the last backup.

AEA Technology Engineering Software 6

 ALESSIA/AEA/Best Practice Guide Issue 1

The system should definitely have a read-write CD on one of the systems, and possibly a
large dat tape system. DVD writers are just becoming available commercially, (600 Euros)
and may also be a viable alternative.

Once it is becomes necessary to create an animation, or results for acoustic calculations, it is

necessary to dump these results out at more frequent intervals, possible every time step. To
minimise file space, the user ought to exercise care as to what and when data is required.
Ideally the system should have a very large scratch disk available for the storage of these
results.

Note that because the acoustic calculations are typically carried out in the frequency domain,
the whole time information is required, in order to perform the FFT for the frequency domain.
If the grid for the acoustic calculations is known a-priori, it is possible to create a CFX
definition file for this grid, and interpolate fine grids onto the coarser mesh, for acoustics post-
processing.

5 AEROACOUSTIC CALCULATIONS
The Alessia project has used an approach of decoupling the flow and acoustics calculations,
using the Lighthill analogy. This is good for situations where the acoustics is required in the
far field, and the flow domain can be much smaller. For near field calculations, the fluctuating
pressure field the ‘pseudo-pressure’ can give a good indication of the local noise field, but not
if reflections and diffraction effects are important.

Some authors advocate the use of highly accurate methods, to couple the flow and acoustics
in a single framework, using very high order methods. This may be appropriate for some
simple flow cases, but does not make full advantage of the facilities and models available in
commercial software packages.

If the flow is strongly compressible, it may be necessary to couple the flow and acoustics
more closely, for example by carrying out a full compressible flow calculation, within the
CFD solver. This is particularly good when there are acoustic resonances, and the acoustics
affects the flow.

In the approach chosen in the project (relevant for incompressible, low Mach number flows),
the aeroacoustic calculations are performed using SYSNOISE. The next two sections give
guidelines on which module to use in SYSNOISE together with some numerical
consideration.

5.1 SYSNOISE MODULES

5.2 HARMONIC ACOUSTIC FEM

Acoustic FEM can be used to model closed (‘interior’) fluid regions. 1-d (line/pipe) 2-d (plane
or axisymmetric) and 3-d (volume) models are possible. The benefit of FEM is that different
elements can have different sound speeds and/or densities, to model element-by-element
variations in acoustic properties (eg, due to temperature change, or – using special volume

AEA Technology Engineering Software 7

 ALESSIA/AEA/Best Practice Guide Issue 1

absorbent elements – to model thick absorbent regions, such as foam or wool fill). Transfer
across gaps between element free faces can be modelled using special transfer impedance
relationships, for example to model a perforated membrane. Boundary conditions can include
surface vibrations (assumed, imported from a structural model, or from tests) surface
impedance (absorption layer) and acoustic sources. Solutions can be direct acoustic response
to the excitation, or modal analysis (resonance of the enclosed region) followed by acoustic
response by modal superposition.

5.3 COUPLED HARMONIC FEM

Coupled Acoustic FEM enables the full two-way fluid-structure interaction to be taken into
account in an acoustic FE model. The module includes a structural element library (mainly
shells and plates) and also enables the import of structural models from standard FE codes, as
modal models. Structural modal damping can then be applied.

Boundary conditions can then include structural constraints (fixations, hinges) and forces and
moments (mechanical excitation).

5.4 ACOUSTIC I-FEM

Infinite elements extend finite element acoustic models into infinite or partly-infinite
domains. The acoustic results (pressure, intensity, radiated power) can be determined in the
infinite elements as well as in the near field (modelled with acoustic finite elements). The
advantage of FEM plus I-FEM over BEM, is that the near field region can have any of the
standard FEM features (inhomogeneous fluid, volume absorber, …). Two-way fluid-structure
coupling can also be applied, as in the Coupled FEM module.

The disadvantage of FEM plus I-FEM is the need to create a significant near-field volume
mesh for the FEM part. As well as a geometry-modelling/meshing task, this can also result in
a larger numerical problem size in some cases (hence, longer calculation times). The balance
between FEM/I-FEM and BEM is often very application-dependent.

5.5 HARMONIC ACOUSTIC BEM

Acoustic BEM can be used to model both closed (interior) and infinite (exterior) domains, and
mixed situations where the interior is open to the exterior through certain holes etc.
SYSNOISE has both Direct BEM (which requires a single, closed mesh) and Indirect BEM
(which allows quite arbitrary topologies, including free edges and t- and x-junctions). The
Indirect method in particular has the benefit of enabling efficient models of thin structures
having (the same) fluid on both sides, for example flexible casings or loudspeaker
membranes. The mesh generation for a BE model is much simpler than for FEM, since only a
‘shell-like’ mesh on the surfaces is required. However, solution times for BE models are
typically longer, especially if many frequencies are to be analysed, due to the frequency-
dependence of the solution parameters and because only direct solutions (no modal
approaches) are possible.

2-d (plane and axisymmetric) and 3-d (volume) domains can be modelled.

AEA Technology Engineering Software 8

 ALESSIA/AEA/Best Practice Guide Issue 1

Boundary conditions include surface vibration (as in acoustic FEM) surface impedance
(absorption layer) and acoustic sources. In the Indirect BE method, surface-related boundary
conditions can be applied on one side of the element only, if required. Transfer through
selected elements can be modelled using special transfer impedance relationships, for
example to model a perforated membrane. Fluid properties have to be homogeneous
throughout the domain. Acoustic results at any location can be found by using appropriate
field point grids.

5.6 COUPLED HARMONIC BEM

Coupled Acoustic BEM enables the full two-way fluid-structure interaction to be taken into
account in an acoustic BE model. The module includes a structural element library (mainly
shells and plates) and also enables the import of structural models from standard FE codes, as
modal models. Structural modal damping can then be applied.

Boundary conditions can then include structural constraints (fixations, hinges) and forces and
moments (mechanical excitation).

5.7 NUMERICAL CONSIDERATION

5.7.1 Mesh coarsening

An important step in the preparation of a boundary element or finite element model is to
coarsen as much as possible the original CFD grid. This step is important because it reduces
the solution time of the acoustic problem that is proportional to N3 where N is the number of
unknowns.

The six-element-per-wavelength rule for predicting the maximum frequency of an acoustic

mesh is recommended. Nevertheless, this rule ensures the convergence of the matrix operators
but does not consider the distribution of the sources. Therefore, this discretisation of the
acoustic model must be such that at least six elements per wavelength are used and that the
sources distribution is correctly represented.

5.7.2 Free edges

When working with the indirect approach of the boundary element method, the primary
unknowns are the single and double layer potentials. The layer potentials correspond to the
velocity jump and pressure jump through the boundary surface.

Along free edges, zero jump of pressure must be applied. This condition represents the fact
that there can’t be any pressure discontinuity through the surface along free edges because the
two sides are in contact with each other.

5.7.3 Junctions
In indirect boundary element approach, it is possible to model a structure composed of
subparts that are joined together at a given edge. For example, a T- or X-junction can be
modelled. The line along which the different surfaces join is called a junction and requires
some special treatment.

AEA Technology Engineering Software 9

 ALESSIA/AEA/Best Practice Guide Issue 1

A node located along a junction needs to be duplicated (eventually several times) to correctly
represent the different pressure discontinuity through the surface. A compatibility equation
needs to be defined to account for the fact that the total pressure discontinuity is equal to zero.

It is recommended to use the default automatic junction algorithm that has been tested on a
wide range of problem and proved to accurately detect junctions.

6 System requirements & Choice of platforms

Details on parallel performance of CFX-5 and SYSNOISE for the various supported
platforms are presented in [ 1]. The conclusions about system requirements and choice of
platforms are repeated here.

LES computations are very expensive in term of CPU time, and really require HPCN systems
for such simulation to be performed on industrial applications.

From a price performance viewpoint, the testing done in the project on various platforms
shows that there is no point to using dedicated supercomputers unless:
• Your organisation has one, or it is very cheap.
• The software you are using can extract high efficiency from the vector and parallel
system. This usually means special purpose programs. However, they can gain an
order of magnitude in performance.
• You need ultimate performance,(eg real time) as price performance is much better
from the commodity workstation systems.

For larger problems, the Unix workstations have 64 bits operating systems, and efficient
execution of double precision calculations. They also can address more memory. Consider
Unix systems with shared memory paradigm. (SGI, Compaq Ultra Server, SUN, IBM, HP)
for these problems. IBM SP systems seem to be more difficult to administer. If you regularly
run large problems in a multi-user Unix environment, then the large memory will enable mesh
generation and visualisation on large problems, whilst still running LES calculations.

For low cost systems based on Intel / AMD systems, then we found that:
• On similar systems, Linux shows a similar efficiency to Windows NT or 2000.
• Efficiency of low-level parallel libraries, (PVM / MPI) for NT varies considerably.
• Dual processor systems provide good performance but not as good as single processor
system. Both on Linux and Windows there can be a performance hit using dual
processors.
• On Windows NT/2000, for more than two processors, CFX-5 does not work as
efficiently in a split shared-memory and networked manner. Reasons for this are not
understood, but the conclusions could change with newer versions of operating
systems or software.
• Linux and distributed memory using PVM most cost effective, for problems up to 2 M
cells.
• PVM is more robust than MPI, for released versions of CFX-5, especially on clustered
systems.

AEA Technology Engineering Software 10

 ALESSIA/AEA/Best Practice Guide Issue 1

• Ethernet using 100 Base T connections seems to be adequate. Consider partitioning

the network to minimize network traffic, or using a dedicated network for ultimate
performance.
• It is perfectly feasible to use under-used systems for providing additional parallel
systems. CFX use its training network for this purpose, for example.
• 1 Gbytes per processor is recommended, (previously 500 Mbytes recommended). For
these systems, it is often not possible to run the larger problems on a small number of
processors, because of the memory limitations of these systems.
• As large disks as possible, at least 120 G Bytes.
• Compilers are free for Linux.
• Buy for now. Whatever you buy will be outdated in a year.

Results obtained on Unix / Linux can still be processed on Windows NT/2000.

6.1 OPERATING ENVIRONMENT

One of the major issues to consider when using HPC is the overall system management,
especially the allocation of resources to users, and the scheduling of jobs. This can impose a
significant cost to the management of these systems, as a departmental system can often be
run in a more informal manner.

However, consideration should be given to

• For more than one user, consider using a system such as LSF to dynamically manage
systems and queues. There are other systems available, some public domain. We have
no experience of these.
• There are also other system utilities, such as Perfmon.exe , which can be used to check
on whether a system is being utilised.

6.2 STORAGE OF RESULTS AND BACK UP

The system should definitely have a read-write CD on one of the systems, and possibly a
large dat tape system. DVD writers are just becoming available commercially, (600 Euros)
and may also be a viable alternative.

Once it is becomes necessary to create an animation, or results for acoustic calculations, it is

necessary to dump these results out at more frequent intervals, possible every time step.
Careful though about what data is required, and when that data is required is very important.
Ideally the system should have a very large scratch disk available for the storage of these
results.

Note that because the acoustic calculations are typically carried out in the frequency domain,
the whole time information is required, in order to perform the FFT for the frequency domain.
If the grid for the acoustic calculations is know a-priori, it is possible to create a CFX
definition file for this grid, and interpolate fine grids onto the coarser mesh, for acoustics post-
processing.

AEA Technology Engineering Software 11

 ALESSIA/AEA/Best Practice Guide Issue 1

7 References
[ 1] Jones, I P and Montavon, C A, The Alessia Project, available from
http://www.software.aeat.com/cfx/european_projects/alessia/alessia.htm

[ 2] Jones, I P and Montavon, C A, The Alessia Project, Validation Report, available from
http://www.software.aeat.com/cfx/european_projects/alessia/alessia.htm

AEA Technology Engineering Software 12

ALESSIA/AEA/Best Practice Guide Issue 1