CFD 101

Jeff Adie Principal Systems Engineer

jeffadie@sgi.com

Agenda
Introduction

What is CFD?
Fluid Characteristics
Flow Modeling
Turbulence Modeling
CFD Workflow

CFD Optimisation
CFD Compute challenges
MPI & CFD
MDO

Example case

CFD 101
The study of dynamic fluid flows using numerical methods

CFD 101
Fluid flows encountered in everyday life include:

Ship design (hull, sails, appendages)

Aerodynamics of ground vehicles, aircraft, missiles
Meteorological phenomena (rain, wind, hurricanes, floods, fires)
Environmental hazards (air pollution, transport of contaminants)
Heating, ventilation and air conditioning of buildings, cars etc.
Combustion in automobile engines and other propulsion systems
Complex flows in furnaces, heat exchangers, chemical reactors etc.
Processes in human body (blood flow, breathing, drinking . . . )
Many more

CFD 101
Example:

Racing yachts need to be fast, light, & strong

Pick any two, but dont get it wrong!

Wind Tunnels

Basic Idea
Traditionally all fluid simulation based on laws of
conservation, specifically:
Conservation of momentum
Conservation of mass
Conservation energy

This is known as continuum mechanics. We treat the

fluid as a continuum instead of individual molecules
and average properties over the macroscopic scales.

CFD 101
Reference Frame: temporal or spatial
Temporal (Lagrangian)
Reference to Time t=0
Reference moves with time
Used for particle-based CFD and solid mechanics

Spatial (Eulerian)
Reference to Time t = t
Reference moves with space
Used for flow-based CFD

CFD 101
Generally, we use spatial coordinates to
determine fluid parcel properties at a
particular point, ie Eulerian system:

Eulerian fixed observer

Lagrangian moving observer

CFD 101
Classification:

Continuum

Particle

Scheme:

Navier-Stokes

Euler

SPH

LBM

Plus:

Easy to get rate of

change

Easy to get rate of

change

No Topology

Handle complex
boundaries

Guaranteed
conservation of
mass

Microscopic
Interactions

Pressure from
kernel

Handle Multiphase
boundaries

Easy to implement

Scales well

Need a lot of
particles (many will
never be used)

Cannot handle fast

flows

Minus:

Hard to get body

shape

Hard to get body

shape

Fluid Characteristics
Macroscopic Properties
Density
Viscosity
p Pressure
T Temperature
v Velocity

Also, less often:

H Enthalpy
S Entropy

Fluid Types
Newtonian or Non-Newtonian:
Newtonian rate of deformation (strain rate) is
proportional to shear stress
Ie water, air, petrol, beer

Non-Newtonian Types:

Bingham (eg; toothpaste)

Dilatant (eg; corn starch)
Plastic (eg; ketchup)
Pseudoplastic (eg; blood)
Rheopectic (eg; some lubricants)
Thixotropic (eg; inkjet ink)

Fluid Flow Classification

Compressible or incompressible (D/Dt = 0)?
Viscous or inviscid? ( = 0)?
Steady or Unsteady? (constant properties at a
particular point irrespective of time)
Laminar or turbulent? (flow is in layers)
Single phase or multiphase? (gas-liquid, liquidliquid)
Fluid Type? (constant )?

Viscosity
The resistance of a fluid to shear stress
Measured in Pascals, symbol
Technically, is the dynamic viscosity. For
convenience, we also sometimes use the
kinematic viscosity, v, which is dynamic
viscosity divided by density, ie:
v = /
Note that v is the greek letter nu, not the english
letter v

The Reynolds Number

Defines the ratio of inertial forces to viscous
forces in a flow
Dimensionless, symbol Re
Re = vL/, or Re = vL/v
= density, v = mean velocity, L = length, = dynamic viscosity, v = kinematic viscosity

Note that L varies depending on the surface, ie

for flow in a pipe it is the hydraulic diameter of
the pipe
Note: with viscosity as the denominator, flows
with high Re numbers are essentially inviscid

The Reynolds Number

Low Reynolds numbers = Laminar flow
High Reynolds numbers = Turbulent flow
What is high/low? Depends on the system.
Example: Pipe flow (experimentally observed)
Re < 2300 = laminar flow
2300 < Re < 4000 = transition flow
Re > 4000 = turbulent flow

Example: flow over a flat plate

Boundary layer instability occurs ~ Re = 5 x 105 (with L
defined as distance from the leading edge of the plate)

The Reynolds Number

Fluid Flow Equations

Four basic laws cover fluid dynamics:
Conservation of mass
Newtons Second law of motion
Conservation of energy (1st law of
thermodynamics)
2nd law of thermodynamics (Entropy)

From these we derive fluid flow equations for

conservation of momentum, continuity and
energy.

Fluid Flow Equations

Usually have 5 unknowns, vx, vy, vz plus two others
=> need 5 equations to solve

Equation of motion in x
Equation of motion in y
Equation of motion in z
Continuity Equation
Energy Equation

Often add 6th equation (equation of state) to make

the energy equation simpler . For example, the
ideal gas law, p = RT, relates pressure & density

Fluid Flow equations

Continuity equation Conservation of mass
Momentum equations Navier-Stokes
Energy equations Conservation of energy

Note: u = gradient, .v = divergence, x v = curl, 2 u = Laplacian

Deriving Momentum

Consider forces on a static fluid parcel:

Gravitational force is g (Density x Gravity)
Forces from other parcels are pressure p
Average force per unit volume is gradient of
pressure: p/ x + p/ y + p/ z = p
For stasis, forces must be equal, ie:
g = p or -p + g = 0

Deriving Momentum
If forces are not equal, then the fluid parcel accelerates
As well as the gravitational force and pressure gradient, we also
have three additional terms; acceleration (v/ t) , convection and
anisotropic surface stresses:
Convection (convective acceleration) is acceleration w.r.t space (eg; a
tapered nozzle). It is defined as:

(vx / x + vy / y + vz / z)v = vv
Surface stresses are more complicated. These depend on the fluid
properties. In the general case, we can define as T Where T is the
deviatoric stress tensor (see later)

Putting everything together we get the Navier-Stokes eqn:

(v/ t + vv ) = -p + + g

Stress
Stress is defined as the average force per unit
area.
Symbol:
Definition: = F/A
Measured in Pascals (Pa) 1 Pa = 1N/1m2

For CFD, we need to calculate all stresses on a

fluid parcel. There are two basic types:
Normal Stresses
Shear Stresses

Stress Modeling
Stress can be defined by 9 components:
3 orthogonal normal stresses ()
6 orthogonal shear stresses ()
Expressed as a stress tensor,
11 12 13 xx xy xz
x xy xz
= 21 22 23 = yx yy yz = yx y yz
31 32 33 zx zy zz
zx zy z
Normal stress = stretching ie for x: xx = v + 2(vx/x)
Shear stress = deformation ie for xy: xy = yx = (vy/x + vx/y)

N-S for Incompressible flows

For incompressible flows (constant ), the stress
tensor simplifies to: 2v, where is viscosity,
making our equation for incompressible flow:

(v/ t + vv ) = -p + 2v + g

Unsteady
Acceleration

Convective
Acceleration

Pressure
Gradient

Viscosity

Other
Forces

Turbulence Modeling

Turbulence Modeling
Turbulence Models
DNS
RANS-based

Boussineq: algebraic
Spalart-Allmaras: k-
Prandtl: Mixing length model
RSM

LES
DES (RANS + LES)
PDF (with particles)
Vortex Method (grid free,
uses FMM)

DNS
Direct Numerical Simulation Explicitly
solving Navier-Stokes
Must resolve all turbulence scales in the mesh
Mesh N3 must be >= Re9/4 for spatial resolution
Memory required grows as Re2.25
Timestep must be very fine for explicit integration
(CFL condition for convergence, Cmax = 1), so:
C = vxt/x + vyt/y + vzt/z < Cmax

FLOPS required grow as Re3

Turbulence Modeling
RANS Reynolds Averaged Navier-Stokes
Navier-Stokes provides instantaneous velocity
field, v
RANS provides time-averaged equations
Have an average value over a period of time, v , plus a
fluctuating value, v
Mean value determined by spatial or temporal
averaging
Fluctuating (turbulent) part is a stress Tensor, called the
Reynolds stress Tensor,
ij = vivj

Turbulence Modeling
The problem with RANS
The term vivj is non-linear, preventing closure
We have too many unknowns, not enough eqns

Boussineq proposed an Eddy Viscosity Model

(EVM):
-vivj = 2vtSij 2/3Kij where:
Vt = Turbulence eddy viscosity
Sij = Average rate of strain
K = Turbulence kinetic energy
ij = Kronecker delta (=1 if i=j, else =0)

Turbulence Modeling
Spalart-Allmaras proposed a one equation
model for the turbulent viscosity for
aerodynamic flows (see AIAA paper or CFD
Wiki)
Prandtl added a mixing length to handle eddy
viscosity varying with distance from bounding
wall (boundary layer)
K models turbulent kinetic energy (k) and
dissipation () separately in two eqns

Turbulence Modeling
RSM Reynolds Stress Model

Discards eddy viscosity approach

Calculates Reynolds stresses directly
Reynolds stress tensor ij = vivj
RSM uses differential transport equations to calculate
stress tensor:

/t(vivj) + Cij = DTij + DLij + Pij + Gij + ij ij + Fij + S

Cij = Convection, DTij = Turbulent Diffusion, DLij = Molecular Diffusion, Pij =
Stress Production, Gij = Bouyancy Production, ij = Pressure Strain, ij =
Dissipation, Fij = Production by Rotation, S = Source Term

LES
Large Eddy Simulation
Fundamentally, a low-pass filtering to remove small
scales
Solution is a filtered flow velocity field
Can be explicit or implicit
Implicit filter based on the discretisation scheme
Good: Use full mesh, no computational cost for subfilter term
Bad: Shape of LES filter depends on gird, hard to determine,
truncation errors

Explicit filter applied to the N-S equations

Good: well-defined filter shape, lower truncation error
Bad: requires finer grid, computational cost grows as (x)4

LES
Parameters:
Filter type (usually box or gaussian)
Cutoff Filter width (cutoff length), denoted
Cutoff time scale, denoted c

DNS: Domain = L3

LES: = L/32

LES: = L/16
Images User:Charlesreid1 / Wikimedia Commons / CC-BY-SA-3.0

The Future?

From Bakker, 2006

Discretisation

Converting a continuous domain into a series of

discrete elements
Solving a CFD problem requires discretisation in three
areas:
Spatial discretisation of the geometry

Meshing

Temporal discretisation

Time steps

Equation discretisation

Numerical methods

Spatial Discretisation

Temporal Discretisation
Set start/stop times (range)
Steady-state: rule of thumb 10x domain
Unsteady: as desired

Specify timestep (interval)

Steady-state: steps to convergence
Unsteady: resolution to capture sufficient detail

CFL Number

For convergence of the cases using explicit (timemarching) solvers, the Courant-Friedrichs-Levy
condition (Cmax = 1) should be satisfied:
C = |U|t/x < Cmax

C is called the Courant number

|U| is the magnitude of the velocity through the cell
t is the timestep
x is the length step (grid resolution in velocity direction)

CFL Number

Implications of the CFL condition

To ensure solution convergence we need to have either:

Large x
Small t
Both

As |U| increases, need finer resolution mesh or smaller

timesteps. Both add significantly to the time to solution,
although generally it is better (faster) to make t smaller
than to increase x
Note: implicit (matrix-based) solvers are less sensitive to
Cmax

CFL Number

Example:

Geometry 1mx1mx0.1m
Fixed mesh 20x20x1 => x = 0.05m
|U| 2.0 m/s
For 10x domain, set end time = 5s

Calculate t required:
t = Cx / |U|
= 1.0 x 0.05 / 2.0
= 0.0025s

No of steps = 5/0.0025 = 2000 iterations

Discretisation Methods
Finite Differences (differential form of PDE)
Approximation of nodal derivatives
Simple and effective, easy to derive
Limited to structured meshes

Finite Volumes (integral form of PDE)

Approximation of integrals
Conservative (flux in = flux out)
Suitable for arbitrary meshes

Finite Elements (weak form of PDE)

Generate piecewise polynomial basis functions
Remarkably flexible and general
Suitable for arbitrary meshes

Basic CFD Workflow

Select the solver

Create the mesh
If parallel, decompose the problem domain
Set Initial/Boundary conditions
Set runtime conditions
Run the case
Post process results

Computational Meshes
Three basic classifications:

Structured Meshes
Topologically equivalent to cartesian grid
Limited to simple domains
Can be periodic, nonperiodic, or periodic with
cusp

Computational Meshes
Block Structured Meshes
Multilevel subdivision of the domain
Special treatment required at block interfaces
Greater flexibility, blockwise local refinement

Unstructured Meshes
Suit arbitrary domains, amenable to AMR
techniques
Complex data stuctures, difficult to implement

Initial and Boundary Conditions

Initial Conditions:
Define density, velocity,
energy, etc. in, vin, ein

Boundary Conditions:
Inlet (in, vn < 0)

= in, v = vin, e = ein

Solid Wall (w, vn = 0)

v = 0 (no-slip), T = Tw (given temperature) or

(T/n) = -q/k (prescribed heat flux)

Outlet (out, vn > 0)

vn = vn, vs = vs (prescribed velocity) or

-p + nn = 0, s.n = 0 (vanishing stress)

Solver
We have a system of Coupled nonlinear
algebraic equations to solve iteratively:
Outer loop: update coefficients using previous
iteration
Inner loop: use iterative solvers (ie cg) as direct
solvers (eg Gaussian Elimination) are too
expensive

Typically v. large systems (millions of

unknowns) but v. sparse

Solving Techniques
Generally too large for direct solvers
Use iterative solvers, eg:

SOR
Lanczos
CG, BiCG
GAMG
GMRES
BiCGStab

Preconditioning often used to

speed up matrix solution, eg:
SOR
Jacobi,
ILU

Solving for p (incompressible flows)

For incompressible flows, pressure is not linked to
density, so there is no independent equation for p
Pressure is in all 3 momentum equations & velocity
field must satisfy continuity. So even without explicit
equation for p, we can derive implicitly using pressurevelocity coupling algorithms
Examples:
PISO (Pressure Implicit with Splitting of Operators)
SIMPLE (Semi-Implicit Method for Pressure Linked
Equations)
SIMPLER (SIMPLE Revised)
SIMPLEC (SIMPLE Consistent)

Post Processing
Important Variables:
Pressure -> Forces on objects
Velocity -> Flow Structures
Shear -> Erosion
Vorticity -> rotation, mixing
Turbulent K.E & Dissipation rate -> heat
transfer/mass transfer
Temperature -> evaporation, combustion
Radiation

Post Processing
Calculate derived quantities (ie vorticity)
Calculate integral parameters (ie lift)
Visualisation:
1D lines
2D Streamlines, contours, slices
3D cutlines, planes, isosurfaces, isovolumes
Arrow plots, particle trackers
Animations

Post Processing
Translation:
Velocity vectors/flowlines
streamlines, pathlines, streaklines

Deformation
Strain Rate

Rotation
Vorticity/Helicity

Post Processing
Force calculations:
Lift, Drag, Cf, etc
Integrate pressure field over surface area

CFD Optimisation

CFD Challenges
Direct Numerical Simulation of Navier-Stokes is well
beyond present HPC capacity for real-world problems.
Demands for improved CFD simulations, ie:

Higher resolution Meshes

Interactions: FSI, rotor-stator, etc
Transient simulations for unsteady effects
Advanced turbulence modeling (RANS, LES)
Multiphase flow
Non-equilibrium chemical reactions
MDO

Grid Generation
Model must be discretised into a mesh in Euclidian
space
Structured vs Unstructured meshes
Triangles, Quads, hexahedrons
Grid generators
Large mesh pre-processing
10M Cells needs ~ 100GB of RAM

AMR for interesting flow regions

DMP Workload
Load balance between nodes
Communication between nodes
Shifting balance (dynamic mesh, AMR,
animated models)
Rotate the model, or the frame of reference?

Shifting workload (ie combustor models)

Collective operations (transient analysis)
I/O per time step. Through Rank 0? Parallel I/O?

Weak Scaling
Works well for CFD models as domains are readily
decomposed
..but, diminishing returns as we add more cores
Communication between cores becomes dominant,
eg. At 512 cores, threads spend 1/3 time on compute,
2/3 on communications/wait
Interconnect bandwidth & Latency become critical
Inter-domain message size defined by size of domain
faces => Bandwidth
Multi-level solvers (ie multigrid) use lots of small
messages => latency

Scaling and MPI

MPI Communication grows as core count scales
Wheres the time going?

Compute
Allreduce
Waitall

262.1

53.1

Send

13.2 7.28.35.2 25.3

Recv
Barrier

Reduction is largest, but

looking deeper we find
b_allred() is the culprit.

Others

47.5

5.6

(b_allred() is the waiting time in the reduction call,

allred() is the reduction itself)

b_allred
allred

Scaling and MPI

Why is b_allred() taking so long?
The actual reduction allred() cannot occur until all nodes
have completed, so MPI_Allreduce() blocks in b_allred()

This particular example has almost ideal load balance

in cells per parallel domain.
The blocking is caused by micro-imbalances:
Variations in the model
Variations in the domains (internal vs boundary)
Fluctuations in network/HCA loads

Multi-Rail Networks
Using multiple network rails can provide
several advantages:
Separation of Communications and I/O traffic
Load sharing across rails
Splitting large messages across multiple rails
Redundancy in case of network port/cable failure

The main disadvantage:

Cost

HPC Requirements for MDO (1)

Balanced HPC Environment
Different Dicsiplines have different requirements:
Structures (CSM-I): Memory Bandwidth and I/O are critical
with CPU speed a secondary concern
Crash (CSM-E): Needs fast CPUs for element force
calculations plus high memory bandwidth for contact
resolution
CFD: Parallel scalability is key although fast CPU and
Memory are also important
CEM: Communication between nodes is critical with
memory also important

HPC Requirements for MDO (2)

High Throughput Efficiency
Mix of analysis codes including high fidelity analyses
Need fast turnaround time for multiple jobs
Concurrent runs for stochastic (Monte Carlo) simulation

MDO Example
Supersonic Business Jet
Modeled as a coupled system:

Structures
Aerodynamics
Propulsion
Aircraft Range

Optimisation goal:
Maximum Range

S. Kodiyalam and J. S. Sobieski, Alternate Sampling Methods for Multidisciplinary Design Optimization in a High Performance Computing environment,
Proceedings, ASME Design Technical Conferences, Pittsburg, PA, September 2001. ASME Paper Number: DETC2001/DAC-21081.

MDO Example

S. Kodiyalam and J. S. Sobieski, Alternate Sampling Methods for Multidisciplinary Design Optimization in a High Performance Computing environment,
Proceedings, ASME Design Technical Conferences, Pittsburg, PA, September 2001. ASME Paper Number: DETC2001/DAC-21081.

CFD Packages (partial list)

Commercial

Open Source

Solvers

Solvers

CFD++ (Metacomp)
CFD-ACE (ESI)
CFD-FASTRAN (ESI)
CFX (Ansys)
Fluent (Ansys)
PowerFLOW (Exa)
STAR-CD (CD-Adapco)
STAR-CCM+ (CD-Adapco)

Grid Generators
Gambit (Ansys)
HyperMesh (Altair)
ICEM CFD (Ansys)

Visualisation
CFD-VIEW (ESI)
Ensight (CEI)
HyperView (Altair)

CalculiX
ISAAC
OpenFOAM
OpenFVM
SU2

Grid Generators
GMSH
Gridgen
Netgen

Visualisation

Gnuplot
ParaView
Visit
vtk

Getting started with CFD

Demonstration of laminar airflow over a flat
plate:
Known as the Blasius problem
See: http://www.grc.nasa.gov/WWW/wind/valid/fplam/fplam.html

Table 1. Freestream conditions.

Mach

Pressure (psia)

Temperature (R)

Angle-of-Attack (deg)

Angle-of-Sideslip
(deg)

0.1

6.0

700.0

0.0

0.0

Getting started with CFD

Build the grid
37x46 2D domain with grading => ~ 1,700 cells

Define boundary/initial conditions

Inlet pressure/velocity given
6 psia => 41.368 kPa
0.1M => 34.029 m/s

Set transport properties

Need to calculate kinematic viscosity of air at 41.3kPa and 700R => 115.74C

Set up the solver

Steady, incompressible, 10,000 iterations

Go!....

Getting started with CFD

Calculating kinematic viscosity, , at 41.368kPa and 115.74C
=/
Using Sutherlands formula:
=0* (T0+C/T+C) * (T/T0)3/2 where:
T0 = 291.15K, 0=18.27, C=120 <= air at STP
=22.786 Pa.s
Using Ideal gas law:
=p/RST where:
RS=287.06 J/kg
=0.3706 kg/m3

= 6.149x10-4 m2/s

Getting started with CFD

Coming up next time