Notes on Computational Mathematics

James Brunner
August 19, 2013
This document is a summary of the notes from UW-Madisons Math 714, fal l 2012
(taught
by Prof. Shi Jin) and Math 715, spring 2013 (taught by P rof. Saverio Spagnoli).
This
summary was written by and for me, so a lot may be unclear to others. If you are
reading
this and are not me, email me with questions: jdbrunner@ math.wisc.edu. I probably
cant
answer them, but then again you probably arent going to ask any, because the
chances anyone
but me reads this are slim. If you are reading this, however, I welcome both
questions and
comments (like if you find an error!) because they are sure to improve this
document, and
more importantly my own understanding of the material. This document is a work in
progress
and wont be ful ly edited for while, so there may be typos, errors, etc. If you
find any, feel
free to bring them to my attention.
1
Contents
1 Numerical Differentiation 4
2 Solving ODEs 6
3 Finite Difference Metho ds for PDEs 9
3.1 Parab olic
Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Stability, Consistency, Convergence . . . . . . . . . . . . . . . . . . . . . .
. 10
3.3 Metho ds for Non linear Diffusion equations . . . . . . . . . . . . . . . . . .
. 12
3.4 The heat equation in Higher Dimensions . . . . . . . . . . . . . . . . . . . .
13
3.5 Finite Difference Metho ds for Hyp erb olic Equations . . . . . . . . . . . . .
. 15
3.6 Non Linear H yp erb olic
Equations . . . . . . . . . . . . . . . . . . . . . . . . 18
3.7 Numerical Solutions to Non Linear Hyp erb olic Equations - Conservative Schemes
21
3.7.1 Lax-Friedrich Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.7.2 Lex-Wendroff Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.7.3 The Go dunov Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.8 Non Linear
Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.9 High Resolution Sho ck Capturing Schemes . . . . . . . . . . . . . . . . . . .
34
3.9.1 Flux Limiter Metho ds . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.9.2 Slop e Limiter Metho ds . . . . . . . . . . . . . . . . . . . . . . . . . .
37
3.10 Central
Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.11 Relaxation
Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4 Front Propagation 40
5 Finite Volume M etho ds For PDEs 43
6 Sp ectral Metho ds For PDEs 45
6.1 Trigonometric Interp
olation . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1.1 The Fast Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . 48
6.2 Basics ab out the Fourier Transform . . . . . . . . . . . . . . . . . . . . . .
. 49
6.3 Sp ectral Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . 50
6.4 Smo othness & Sp ectral
Accuracy . . . . . . . . . . . . . . . . . . . . . . . . 52
6.4.1 Convolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.4.2 Sp ectral Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.5 Non Perio dic Problems - Chebyshev Points . . . . . . . . . . . . . . . . . . .
54
2
6.5.1 FFT for Chebyshev Sp ectral Metho ds . . . . . . . . . . . . . . . . . . 55
7 Monte Carlo Metho ds 57
7.1 Probability
Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.1.1 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
58
7.2 Convergence of Random Variables . . . . . . . . . . . . . . . . . . . . . . . .
60
7.3 Random Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
7.3.1 Discrete Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
61
7.4 Multivariate Random Variables . . . . . . . . . . . . . . . . . . . . . . . . .
61
7.5 Variance Reduction Metho ds . . . . . . . . . . . . . . . . . . . . . . . . . .
. 62
8 Numerical Linear Algebra 64
8.1 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . 64
8.1.1 I nduced Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.1.2 General Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.2 Singular Value Decomp osition . . . . . . . . . . . . . . . . . . . . . . . . .
. 65
8.3 Pro jection Op
erators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
8.4 Gram-Schmidt Orthogonalization . . . . . . . . . . . . . . . . . . . . . . . .
69
8.5 Householder Triangularization . . . . . . . . . . . . . . . . . . . . . . . . .
. 71
8.5.1 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
72
8.6 Conditioning and
Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
8.6.1 Floating Point Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . 74
8.6.2 Accuracy and Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 74
8.6.3 Backwards Stability of QR algorithms . . . . . . . . . . . . . . . . . 75
8.6.4 Stability of Back Substitution . . . . . . . . . . . . . . . . . . . . . . 77
8.7 Gaussian
Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.8 Cholesky Decomp osition . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. 82
8.9 Eigenvalues and
Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . . 84
8.9.1 Finding Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
8.9.2 Two Great Eigenvalue Theorems . . . . . . . . . . . . . . . . . . . . 92
8.10 Numerically Finding the
SVD . . . . . . . . . . . . . . . . . . . . . . . . . . 93
8.11 Iterative Metho ds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . 94
8.11.1 Classical Iterative Metho ds . . . . . . . . . . . . . . . . . . . . . . .
94
8.11.2 Arnoldi Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
96
8.11.3 The Conjugate Gradient Metho d . . . . . . . . . . . . . . . . . . . . 99
9 Finite Elements 104
9.1 Intro duction and Review of
PDEs . . . . . . . . . . . . . . . . . . . . . . . . 104
9.2 A First Lo ok at Finite
Elements . . . . . . . . . . . . . . . . . . . . . . . . . 107
9.3 Some Analysis - Hilb ert & Sob olev
Spaces . . . . . . . . . . . . . . . . . . . 112
3
Chapter 1
Numerical Differentiation
We can approximate a derivative with a numb er of simple finite-difference ideas:
centred difference f0(x0) f(x 0+h)2-hf(x 0-h)
one sided difference f0(x0) f(x 0+hh)-f(x 0)
Estimation for the error of these approximations can be made using Taylor
expansions.
Taylor expanding f(x) and using this to derive expressions for f(x0 + h) and f(x0 -
h)
and plugging these expressions into the approximations give an exact expression for
the
approximation which can be compared to f0(x0). For example, this procedure shows:
f(x0 + h) - f(x0 - h)
2h = f0(x0) + f000(6x0) h2 + o(h2)
This shows that the centred difference approximation is second order with respect
to h, and
also depends on the regularity of f(x), because any blow up of f(n)(x) will cause
the error
( f0006(x 0) h2 + o(h2)) to blow up (note in this error, the term o(h2) is only
true if f ? Cloc8 ).
Using this same strategy, we can determine the error of any scheme of the form
f0(x0) af(x - h) + f(x0) + ?f(x0 + h)
and also use Taylor polynomials to choose a, , and ?. For a higher order method,
we could
increase terms, using f(x0 + 2h), etc.
Alternatively, we may use interpolating polynomials of degree n, Pn(x), and take
f0(x)
P0
n(x). Note, we can instruct an interpolating polynomial using Lagrange
interpolation as
follows:
Pn
(x) =
n Xi
=0
f(xi)li(x)
li(x) =
n Yj6=i
x - xj
xi - xj
Accuracy of interpolation depends on regularity of f. We can also use piecewise
polynomial
interpolation for better accuracy.
If we have a discontinuity in f, higher order interpolation will generate
oscillations. The
4
b est interp olation is the one interp olated with smo oth data.
Divided difference b etween p oints is defined:
1st f[xi,xi+1] = f(xxi +1i +1)-fxi(xi)
2nd f[xi-1,xi,xi+1] = f[xi,xxii+1+1]-fxi[x-i1-1 ,xi]
nth f[x0,...,xn] = f[x1 ,...,xnx]-n-f[xx00,...,xn-1]
We want to choose the interpolation with the smaller (in absolute value) divided
difference.
We can also produce the interpolating polynomial using Newtons interpolation:
Pn
(x) = f(x0)+f[x0,x1](x -x0)+f[x0,x1,x2](x -x0)(x -x1)+...+f[x0,...,xx](x -x0)(x
-x1)...(x -xn-1)
5
Chapter 2
Solving ODEs
In solving ODEs (and PDEs, for that matter) we will need to ask whether our metho
d
of approximating a solution is stable - i.e. will small errors refrain from growing
into large
ones. However, for this question to b e answerable, the equations we are s olving
should also
b e stable - i.e. small p erturbations in initial data will not grow exp
onentially. So we have a
theorem:
Theorem 1. If f(y, t) is Lipschitz Continuous, ie ? L > 0 st |f(y, t) - f(y *, t)|
= L|y - y *|
?y, y *, t, then there exists a unique solution to the IVP dydt = f(y, t), y(0) =
y0 where y(t) is
continuously differentiable.
For stability of the solution, the Lipschitz condition is sufficient.
To solve the IVP
dy
dt = f(y, t)
y(0) = y0
there are a number of numerical methods, generally based on replacing the
derivative with
a numerical approximation to a derivative. The simplest is the forward Euler
method:
yn+1 - yn
?t = f(yn, tn)
Here the derivative has been replaced by the forward difference approximation, and
this
an explicit method. The backward Euler method is similar, but requires evaluation
of
f(yn+1, tn+1), and so is implicit and requires solving a non-linear equation to
complete (analytically or, for example, by Newtons method). More generally, a
Crank-Nicolson scheme
is one in which these two are combined in a weighted average:
yn+1 - yn
?t = ?f(yn, tn) + (1 - ?)f(yn+1, tn+1)
For ? = 1
2, this is called the midpoint rule and is second order.
Finally, we may use a centred difference approximation for the derivative to get a
second
order leapfrog scheme:
yn+1 - yn-1
2?t = f(yn, tn)
6
It is necessary to determine the stability as well as accuracy of these schemes. To
consider
stability, we can consider f(y, t) = ?y, so that we know y = y0e?t. Then, if ? < 0,
we
have | y(t)| = | y0| ? t > 0. The forward Euler method can in this case be written
as
yn = (1 + ??t) ny0. Then | yn| = | y0| is only true if |(1 + ??t)| < 1. This gives
a region of
stability for ??t. This is called the region of absolute stability. For the
backwards Euler
method, however, we see that here,
yn = #1 -1??t# n y0
This is then stable for |1 - ??t| > 1, which is true if <(?) < 0, and, of course,
if <(?) > 0,
the analytical solution would not be stable, so the numerical one shouldnt be
either. The
same is true for the midpoint method. We call methods like this A-Stable.
Interestingly,
there is a theorem stating there is no A-stable method of third or higher order.
There are
higher order methods, and they have regions of stability, and so are useful. For
example, we
have multi-step methods. The general method of an r step method to solve ut = f(u)
is:
r Xj
=0
an+j
u = ?t
r Xj
=0
jf(un+j)
An Adams Method is one of the type:
un+r = un+r -1 + ?t
r Xj
=0
jf(un+j)
If r is chosen to be 0, this is an explicit method, called Adams-Bashforth, and
the coefficients
can be chosen so that it is O(?tr). If not, it is an implicit, Adams-Mouton,
method, and
can be O(?tr+1). We can choose the coefficients by considering the truncation error
t(tn+r) = 1
?t "Xj=0r ajuj+n - ?t Xj=0r jf(un+j)#
Then we can Taylor expand u and f, and plug this in and choose coefficients to fit
the
properties we want.
There are also Multi-stage methods, such as the Runge-Kutta methods. These require
us to
compute intermediate values. For example, the 2nd order Runge Kutta method is (and
lets
let ?t = k)
u * = un + k
2 f(un)
un+1 = un + kf(u *)
and the 4th order Runge Kutta method is:
F0 = f(un, tn)
F1 = f(un + 12 kF0, tn + 12 k)
F2 = f(un + 12 kF1, tn + 12 k)
F3 = f(un + kF2, tn+1)
un+1 = un + k
6(F0 + 2F1 + 2F2 + F3)
7
These also require relatively small k for stability.
8
Chapter 3
Finite Difference Metho ds for PDEs
3.1 Parab olic Equations
Our goal is to numerically approximate solutions to equations such as ut = ?u with
given
initial and boundary conditions. The idea of a finite difference method is to
discretize time
and space and use numerical approximations to derivatives. This will give a system
of
equations (which must be solved for implicit methods). If we have a tridiagonal
matrix, the
following algorithm can be used to solve it:
say -AjNj+1 + BjNj - CjNj-1 = Mj
and N0 = Nm = 0
with A, B, C > 0
and B > A + C
let N
j = EjNj+1 + Fj
so E0 = F0 = 0
then -A
jNj+1 + BjNj - Cj(Ej-1Nj + Fj-1) = Mj
Nj
=
A
j
B
j-CjEj- 1 Nj + 1 + CBjjF-jC-j1E+jM- 1j
Ej
=
A
j
B
j-CjEj- 1 Fj = CBjjF-jC-j1E+jM- 1j
So this can be solved with 2 loops.
A simple finite difference method to solve ut = uxx would look like:
un+1
j - unj
k = ? # unj+1 - 2hu2nj + unj-1 # + (1 - ?) unj+1+1 - 2uhnj2+1 + unj-+11 !
Here h is the space grid SIZE(assumed to be constant - ie we have a uniform grid),
j the
index of the individual space grid point, k the time step size, and n the time
point index.
9
3.2 Stability, Consistency, Conve rgence
There are a few ways to think ab out the stability of these metho ds. For example,
the
diffusion equation (ab ove) has the prop erty that the L2 norm of the density
function (u(x, t)
above) should decrease. It is then reasonable to expect the same of our numerical
solution
(using the l2 norm for our discrete data). Remembering some analysis, Plancherels
theorem
says || f|| L 2 = || f|| L 2, so we can work with the Fourier transform if we
want. We can
approximate the Fourier transform of our exact solution with the discrete Fourier
transform
of our numerical solution:
u(?, t) = h
J Xj
=0
uj(t)ei?xj
and so
uj(t) = 1
h X
?
u(?, t)ei?xj
I should note here I am using xj = jh as the grid points (this is all 1 D in space
for ease of
typing, the principle extends to higher D). We can plug this second relation into
our scheme
and we get:
X?
u n+1 - u n
k ei?hj = ? X
?
u n ei?h - 2 + e-i?h
h2 ei?jh + (1 - ?) X
?
u n+1 ei?h - 2 + e-i?h
h2 ei?jh
so ? integer ?:
u n+1 - u n
k = ? ei?h - 2 +h2 e-i?h u n + (1 - ?) ei?h - 2 +h2 e-i?h u n+1
And this simplifies to:
u n+1 - u n = ? -4 sin( ?h2 )
h2 u n + (1 - ?) -4 sin(h2 ?h2 ) u n+1
and if we let ? = k
h 2 , we at long last see:
u n+1 = 1 - 4?? sin2( ?h2 )
1 + 4?(1 - ?) sin2( ?h2 ) u n
and so this method is stable (in the sense that it agrees that || u|| L 2 is
decreasing, as it should)
if
1 + 41 -?(14??-sin?) sin2( ?h22() ?h2 )
 = 1
Some straightforward computation reveals that this is true if:
?(2? - 1) = 1
2
10
This metho d of determining stability is called Von Neumann Analysis after John Von
Neumann, who, along with being one of the best mathematicians and scientists of the
past 100
years, was a member of the Manhattan Project. In general, Von Neumann analysis
involves:
assume un
j = gne-ij?h
where g = uun +1n(?()?)
and we have stability if |g| = 1
This method for determining stability works for finite difference methods applied
to linear
PDEs, which is something I guess. We can check out consistency (ie truncation
error
approaches 0 with gridsize) using Taylor expansions (and a lot of scratch paper).
It would
be great, then if we could say something about convergence of the method. Luckily,
we have:
Theorem 2. The Lax Equivalence Theorem A scheme converges iff it is consistent and
stable.
By convergent, we mean that if Ek(, t) is the difference between the analytic and
numerical solution to a problem, ||Ek|| ? 0 as k ? 0 in some norm || ||.
Formally, call the
numerical solution Uk and the analytic solution u, then
Ek(, t) = Uk(, t) - u(, t)
and local truncation error is:
Lk(x, t) = 1
k[u(x, t + k) - Hk(u(, t), x)]
Where Hk is the operator associated with the method (ie Hk(u(, t), x) is the
result of using
the method to find Un+1() when Un() = u(, tn) exactly). Then consistency is the
condition
that ||Lk|| ? 0 as k ? 0.
Continuing with some more formal definitions: A method is of order p if for all
sufficiently
smooth initial data with compact support, ? a constant Cp such that ||Lk|| = Cpkp ?
k < k0,
t < T.
Formally, a method is stable if for each time T, ? Cs > 0 such that ||(Hk)n|| =
Cs ? nk = T,
k < k0, noting that nk = tn.
So we see now that in Von Neumann analysis, we can have stability if |g| = 1 + ck.
We can also think about energy in the case of the diffusion equation to determine
stability.
The problem is:
ut = uxx
u(x, 0) = u0(x)
u(0, t) = u(1, t) = 0
and so a multiplication, integration, and integrating by parts gives:
12
?t Z0 1 u2dx = uux| 10 - Z0 1(ux) 2dx = 0
11
and so,
|| u(, t)|| L 2 = || u(, 0)|| L 2
This shows that energy in the system should decay (it is the diffusion equation
after all).
Now, because of the 0 Dirichlet boundary condition, this leads to the Maximum
Principle
that the maximum of u can only be obtained along x = 0, x = 1, or t = 0. We can
look
at L8 stability for a finite difference method to solve a linear PDE rather simply
by solving
for un+1
j and using some basic inequalities. L2 stability, on the other hand, is a very
long
computation. One helpful equality in this computation is:
J-1
Xj
=1
uj(vj+1 - 2vj + vj-1) = uJ(vJ - vJ-1) - u1(v1 - v0) -
J-1
Xj
=1
(uj+1 - uj)(vj+1 - vj)
The computation is left as an exercise (ha!).
3.3 Metho ds for Non linear Diffusion equations
Now we deal with a more difficult class of equations, non linear diffusion
equations, which
have the form:
ut = ?x(D(u, x)?xu)
We will usually just deal with 0 Dirichlet boundary conditions on the interval [0,
1], but of
course that is not necessary (just simpler). We note that this equation gives a
conservative
system, because if ?xu| x=0,1 = 0, then
?t Z0 1 udx = 0
so that with no flux, total mass is conserved. A scheme that preserves this
property is called
a conservative scheme (shockingly). A scheme that is not conservative is probably
bad. For
example, the obvious finite difference scheme to deal with this equation:
?tuj = Dj+1 - Dj-1
2h
uj+1 - uj-1
2h + Dj uj+1 - 2hu2j + uj-1
is not conservative. This isnt good. I can point out here that a good rule of
thumb for
ANY numerical approximation is to try to keep the analytic properties of the
system. A
conservative scheme for this system is:
?tuj = Dj + Dj+1
2h
uj+1 - uj
h2 -
D
j + Dj-1
2h
uj - uj-1
h2
which can be more compactly written as:
?tuj = Dj+1/2?xuj+1/2 - Dj-1/2?xuj-1/2
h
12
For stability, we again notice that
R uut = R u?x(D(u, x)?xu)
12
?t R01 u2dx = - R01 D(?xu) 2dx < 0
||u|| L 2[0,1] < ||u0|| L 2[0,1]
And so we will call a method stable if this property is kept. To determine
numerical stability,
Von Neumann analysis no longer works. We need to use the energy method. This is
quite a
lot of computation, I will not include it here.
3.4 The heat equation in Higher Dimensions
In two dimensions, the equation becomes
ut = ?u
for readability, we will denote dx2u = uj+1,k - 2uj,k + uj-1,k and dy2u similarly.
If we use a
square, uniform grid, ie ?x = ?y, I will use ? = ?t
(?x) 2 =
?t
(?y) 2. It turns out that for an
explicit scheme, the stability condition is:
?t
(?x) 2 +
?t
(?y) 2 =
12
We can use Von Neumann stability analysis, here the Fourier expansion gives:
un
j,k = gne-i(?j?x+?k?y)
An implicit scheme in 2D leads to a complicated linear algebra problem. The Crank
Nicolson
scheme is:
un+1
j,k - unj,k
?t =
1
2(?x) 2
dx2unj,k+1 + dy2unj,k+1 + dx2unj,k + dy2unj,k#
Alternatively, we have the Alternating Direction Method (ADI) (see what I did
there?):
un+1/2 - un = 1
2? dx2un+1/2 + dy2un#
un+1 - un+1/2 = 1
2? dx2un+1/2 + dy2un+1#
The advantage here is that in each half-step, only one derivative is implicit. This
gives two
tri-diagonal systems, which are easy to solve. This is a 2nd order unconditionally
stable
method. This is shown using Von Neumann analysis. It is second order because it can
be shown to be a fourth order correction to the Crank Nicolson scheme. The ADI can
be
extended to 3D and is still unconditionally stable:
u* - un = ?
2 #dx2(u* + un) + dy2(2un) + dz2(2un)#
u** - un = ?
2 #dx2(u* + un) + dy2(u** + un) + dz2(2un)#
un+1 - un = ?
2 #dx2(u* + un) + dy2(u** + un) + dz2(un+1 + un)#
13
Computations which eliminate u * and u ** show that this is second order as well.
In general, we can deal with an arbitrarily high number of dimensions with the
Fractional
Step Method. The problem is then:
ut = A A = A1 + A2 + ... + Aq
Ai = dxixi
We can attack this in q steps, dealing with only one dimension at a time with a
centred
difference operator Bi approximating Ai:
Bi = 1
(?x)2 dx2i
so that we have:
un+i/q - un+i- 1/q
?t = Bi un+i/q +2un+i- 1/q
This is a Crank-Nicolson scheme in each direction, and so is unconditionally
stable, and it
is also 2nd order. A more general problem is
ut = uxx + f(u)
Here we can use a fractional step method:
ut = uxx tn ? tn+1/2
ut = f(u) tn+1/2 ? tn+1
This is a consistent splitting, and the splitting error is first order, we whatever
method
we choose to solve each half-step, overall we will have a first order method. To
see that
it is consistent with first order error, we do the following computation with the
operators
Au = d
xxu and Bu = f(u)
ut = Au + Bu = (A + B)u
u(t) = e(A+B)tu0
un+1 = e(A+B)?tun
Un+1/2 = eA?tun
Un+1 = eB?tUn+1/2 = eB?teA?tun
Un+1 - un+1 = eB?teA?t - e(A+B)?t# un
now we Taylor expand the exponentials and simplify to see that:
Un+1 - un+1 = (?t)2
2
(BA - AB) + O((?t)3)
14
And so the overall error is first order. We can improve this to 2nd order splitting
error using
Strangs splitting, which uses one half step of A, one step of B, and one half step
of A. This
forms the exponential
e
? t/2A
e?tBe? t/2A
and
e
? t/2A
e?tBe? t/2A - e?t(A+B) = O((?t) 3)
so this is second order splitting. Also, in implementation this is n- 1 steps of
simple splitting
began and ended with a half step of A.
3.5 Finite Difference Metho ds for Hyp erb olic Equations
The simplest hyperbolic equation is:
ut + aux = 0
u(x, 0) = u0(x)
which can be solved analytically using the method of characteristics, and so will
be nice for
setting up some theory of these methods. The solution is u(x, t) = u0(x - at), so
the initial
data propagates at speed a. This advection equation can also be a system:
Ut + AUx = 0
with U ? Rd and A ? Rdd. The equation is called hyperbolic if A has d distinct,
real,
eigenvalues. This means that A = T?T-1 is diagonalizable (so ? is diagonal), so we
can
solve this system. In fact, if V = T-1U, we have:
Vt + ?Vx = 0
vi(x, t) = vi(x - ?it, 0)
U = T V
A first guess at a finite difference approach to these problems would be
un+1
j - unj
k + a
un
j+1 - unj-1
2h = 0
and indeed this is first order in time and second order in space. However, Von
Neumann
analysis shows:
g-1
k + a e- i? h2-h ei?h = 0
g = 1 + kha i sin(?h)
|g| 2 = 1 + kha # 2 sin2(?h) > 1
15
so this is unconditionally unstable. However, we can use
un+1
j -
un
j +1+unj- 1
2
k + a
un
j+1 - unj-1
2h = 0
Here (I will skip the calculation, but it is good practice of Von Neumann stability
analysis
if youve forgotten how to do it) we have
|g| 2 = 1 - 1 - a2 #hk # 2! sin2(?h)
and so ??, |g| 2 = 1 ? |a| hk = 1. We call ? = |a| hk the Courant number or CFL
number,
and this particular scheme is the Lax - Friedrichs Scheme, which we will have to
return
to in more complicated cases (after all right now we are talking about a problem
that is
easy to solve analytically!). The stability condition is 0 = ? = 1. We can
understand this
condition in a physical sense we remember that hyperbolic equations (our example is
the
advection equation and is simple, but a more important example is the wave
equation) allow
information to propagate at finite speeds (in contrast to parabolic equations). In
this case
that speed is a. This means that our solution should not see information from too
far
away. The area of influence on the solution is called the domain of dependence and
the
statement that 0 = ? = 1 is the statement that the domain of dependence is inside
the
numerical domain of dependence, ie the information used in the numerical method.
Another approach we could take is a one sided difference approximation, known as an
upwind
or downwind method:
downwind u
n +1
j -unj
k + a
un
j +1-unj
h = 0
upwind u
n +1
j -unj
k + a
un
j -unj- 1
h = 0
Von Neumann analysis reveals that the stability condition is the CFL condition,
ie ? = 1,
depending on propagation direction: if a > 0, the downwind scheme is
unconditionally
unstable, and if a < 0 the upwind scheme is unconditionally unstable. This makes
sense
physically, the downwind scheme should only work if information is propagating to
the left
(backwards or down wind), and upwind should only work if information is
propagating
to the right (forwards). This is because each scheme only uses information from the
left or
right, but not both.
So far all of these method have been first order. For a second order method we can
use the
Lax - Wendroff Scheme, which comes from the Taylor series (as all these kind of do)
and the
equation itself:
un+1
j = unj + k?tunj + k22 ?ttunj + O(k3)
un+1
j = unj - ak?xunj + a22k2 ?xxunj + O(k3)
un+1
j = unj - ak unj+12-hunj-1 + a22k2 unj+1-22uhnj +unj-1
This is a second order method because local truncation error is O(k3 + kh2), and
Von Neumann analysis (all these chances to practice that!) shows that this is
stable under the CFL
condition. (I would also like to point out that this derivation is a good clue as
to how we are
16
going to develop numerical metho ds for more interesting hyp erb olic equations).
One thing
that comes up with a second order scheme is a problem with discontinuities (here
they can
only come from initial data, later they will develop on their own). We see
oscillations at
discontinuities in second order schemes.
Discontinuities cause some problems in first order metho ds, as well. Considering
an upwind scheme, using Taylor series to find lo cal truncation error shows it to b
e a first order
approximation to ut + aux = 0, but a second order approximation to
?tu + a?xu = ah
2 #1 - akh # uxx
and the CFL condition implies a positive diffusion coefficient. So this method will
tend to
smooth out the solution, which will of course cause discontinuities to be
destroyed. This is
true of other methods, for example the Lax-Friedrichs scheme has a slightly larger
diffusion
coefficient. This phenomenon is known as numerical diffusion or numerical
dissipation.
Similar truncation error analysis of the Lax-Wendroff scheme reveals it to be a
third order
approximation to
ut + aux = -
16
ah2 1 - # akh # 2! uxxx
We say that this is a dispersive scheme.
There is a general Dispersion Relation for a differential equation (more on this in
my applied
math notes), using the assumption that
u ~ ei(?t -?x)
where ? is the frequency and ? is the wave number. This leads to (plugging in to
the equation
from the Lax-Wendroff truncation error analysis)
i? - ai? = -1
6
ah2 1 - # akh # 2! (i?3)
solving for ? gives the dispersion relation
?(?) = a? + 1
6
ah2 1 - # akh # 2! ?3
for the upwind scheme, the dispersion relation is
?(?) = a? + iah #1 - akh # ?2
and in general we have (in 1 D)
?(?)
? group velocity
?0(?) phase velocity
17
It is imp ortant to realize that the disp ersion relation is a prop erty of the
PDE, not the
scheme. We asso ciate a disp ersion relation with the scheme when the scheme will
act more
like some PDE than the one it was designed for (eg the upwind scheme b eing a
second
order approximation to the diffusive PDE). The disp ersion relation reveals damping
and
oscillations.
3.6 Non Linear Hyp erb olic Equations
We will rarely care ab out the advection equation, as it is simple and can
generally b e dealt
with analytically. A more interesting problem is a non-linear advection typ e
equation:
ut + f(u)x = 0
where f : Rn ? Rn is the flux. If the Jacobian matrix f0(u) has n real eigenvalues
and a
complete set of n linearly independent eigenvectors, this equation is hyperbolic.
This form
of the equation is called a conservation law, with u being the conserved quantity,
because
(assuming flux vanishes at the boundaries)
?t Z udx = 0
An example of a hyperbolic system that is non linear is the Euler equations for gas
dynamics:
?t? + ?x?u = 0 conservation of mass
?t?u + ?x(?u2 + p) = 0 conservation of momentum
?t 12?u2 + ?e# + ?x 12?u2 + ?e + p# u# = 0 conservation of energy
so that
U =
??
?
?
eu
12
?u2 + ?e
??
?
f(U) =
??
?
?u
?u2 + p
12?u2 + ?e + p# u
??
?
The Jacobian of the system has 3 eigenvalues u, u c where c is the speed of
sound, which
depends of ?, p and e. Another example (which is less unwieldy and so easier to use
for
demonstrative purposes) is the inviscid Burgers equation
ut + #u22 # x = 0
18
where u is wave speed. We can use the method of characteristics on these equations
as well,
and we see that the characteristics have equations
x(t) = x0 + tu0(x0)
and u(x, t) = u0(x0) for some x0. These characteristics run in straight lines.
However, each
characteristics slope is u0(x0), and so they may cross. When characteristics
cross, we lose
the uniqueness of the solution. This is called a shock. Also possible are
rarefactions in
which characteristics separate, leaving gaps in the solution. When
characteristics meet,
? u/?x ? 8, and because
?u
?x =
du0
dx
?x0
?x =
u 0
0(x0)
1 + tu 00(x0)
this implies that
1 + tb(u 00(x0)) = 0
where tb is the time the discontinuity occurs. If this time is positive, ie
tb =
-1
u 0
0(x0) > 0
then a shock will occur. So, if there is some x0 so that u 00(x0) < 0 there will be
a shock. We
can also calculation the shock time by
tb = min # u 00-(x10) #
At a shock (or a rarefaction) the strong solution fails to exist. However, we can
still discuss
a weak solution to the equation.
Consider
f(x, t) (ut + f(u) x) = 0
for a text function phi(x, y) ? C01, and the integral
ZR Z0 8 f (ut + f(u) x) dtdx = 0
We then integrate by parts, and say that u is a weak solution to ut + f(u) x = 0 if
for any
phi ? C08,
- ZR f(x, 0)u0(x)dx - ZR Z0 8(uft - f(u)fx)dtdx = 0
holds. If u is a strong solution, it must also be a weak solution, but the converse
is not true.
The following theorem is important in determining what happens at a shock:
Theorem 3. if u is a weak solution to ut + f(u) x = 0, then at a discontinuity
[f(u)] = s[u]
Here, [a] = limx?x +
s
a(x) - limx?x -
s
a(x). This theorem, the Rankine-Hugoniot jump
condition, tells us the speed of the shock, s:
s =
[f(u)]
[u]
19
This all follows from Greenes theorem.
So, considering the inviscid Burgers equation with Riemann initial data
ut + # u22 # x = 0
u(x, 0) =
??? uRL x <x > 0
we can look for a self similar solution, which has the form
u(x, t) = u #xt # = u(?)
Then the equation is reduced to
(u - ?)u? = 0
So the solution is either a constant or u = ?. If uL > uR, we see that a shock
forms with
speed
s =
u2
R/2 - u2L/2
uR - uL
=
12
(uR + uL)
and so the solution is simply
u(x, t) =
??? uRL x < stx > st
However, if uL < uR, the solution is
u(x, t) =
????? uxtRL uLt < x < ux > ux < uRLt Rt
This is one of infinitely many weak solutions, of course, but this one is the
solution derived
from the self similar equations. We can choose a different weak solution by
considering the
Burgers equation with viscosity:
ut = #u22 # x = #uxx
This parabolic equation has a unique solution u# which depends on epsilon. We can
then
take lim
#?0 u# to arrive upon the viscosity solution to the inviscid Burgers equation.
Yet another way to deal with the non uniqueness is through an entropy condition. If
?(u) is
an eigenvalue of f0(u), the Lax entropy condition is that, at a shock,
?(uL) > s > ?(uR)
20
This condition can indicate if there is a sho ck or rarefaction, and under this
condition, the
inviscid Burgers equation can only have a sho ck if uL > uR. Otherwise, it has a
rarefaction.
There are other entropy conditions. For example, if there exists F(u) which is
convex and
?(u) such that
?0(u) = F0(u)f0(u)
they are called convex entropy F and entropy flux ?, and they must satisfy
?tF(u) + ?x?(u) = 0
For the inviscid Burgers equation, the viscosity solution can be shown to satisfy
this condition.
In order to solve non linear hyperbolic equations numerically it is necessary to
consider
the concepts of bounded variation and total variation. We will not delve deeply
into these
concepts, but rather confine ourselves to their relevance in numerical methods.
Total variation can be defined
T V(u(, t)) =
d Xi
=1
lim sup
#?0
1#
ZRd |u(xi + #ei, t) - u(xi, t)|dx
The total variation can be thought of as a measurement of the oscillations of a
function u.
Bounded variation means
BV(Rd) = {f : T V(f) = C}
If we have the problem
ut + Pdi=1 ?xifi(u) = 0u(x, 0) = u0(x) ? L8(Rd)
then given F and ?, there is a unique u such that the entropy condition is
satisfied and
||u(, t)|| 8 = || u0|| 8 and if
u0 ? L8(Rd) n BV(Rd)
then
u(, t) ? BV(Rd)
T V(u(, t)) = T V(u0)
3.7 Numerical Solutions to Non Linear Hyp erb olic E quations - Conservative
Schemes
Again, we are trying to solve
ut + f(u)x = 0
The general form of a conservative scheme for hyperbolic equations is
?tuj + Fj+1/2 - Fj-1/2
?x = 0
These are the schemes we want. A theorem from Lax & Wendroff states
21
Theorem 4. A consistent, conservative scheme, if convergent, converges to a weak
solution
of the non linear conservation law
Clearly, it is only necessary to know the numerical flux Fj1/2 to know a
conservative
scheme. However, that is not always a great way to implement the scheme! In
general, we
will use the notation fj = f(uj) and
uj =
1 ?x
Zxxj+1/2
j-1/2
u(x)dx
or, for simplicity, uj = u(xj)
3.7.1 Lax-Friedrich Scheme
The Lax-Friedrich Scheme is
Fj
+1/2 =
fj + fj+1
2
-
?x
2?t (uj + 1 - uj)
The Local Lax-Friedrich Scheme is
Fj
+1/2 =
fj + fj+1
2
-
vaj+1/2
2
(uj + 1 - uj)
where va = max|f0(u)|, u ? (uj, uj+1), or the largest eigenvalue of f0(u) if we are
dealing
with a system of equations.
3.7.2 Lex-Wendroff Scheme
The Lax-Wendroff Scheme is
Fj
+1/2 =
f(uj+1) + f(uj)
2
- ?tf0(uj+1/2) f(uj+1) - f(uj)
2?x
where u
j+1/2 is the average of uj and uj+1. It is not obvious why this scheme is
consistent,
but the derivation of this scheme will make it clear. We can start, as usual, with
a Taylor
expansion:
un+1
j = unj + ?tunj ?t + ?ttunj ?t2
2
+ O(?t3)
we then use the PDE:
ut = -f0(u)ux = -f(u) x
utt = -(f(u) x) t = -f0(u) tx = -(f0(u)ut) x
= (f0(u)?xf(u)) x
22
For last equality, making the substitution ut = -f(u)x from the PDE. This gives us
expressions for utt that we can substitute into the Taylor expansion get rid of the
time derivatives
in favour of spacial derivatives:
un+1
j = unj - ?t?xf(unj ) + ?t2
2
(f0(unj )?xf(unj ))x + O(?t3)
Then we can use standard finite difference approximations for ?xf(unj ) and so on.
To keep
the scheme second order (as it can be seen to be so far from the Taylor expansion)
we will
need centred difference approximations. Making these substitutions and rearranging
terms
gives:
un+1
j - unj
?t +
f(unj+1) - f(unj-1)
2?x =
?t
2
??
f0(unj+1/2) f(unj+1?)-x f(unj )
?x -
f0(unj-1/2) f(unj )-?fx(unj-1)
?x
??
3.7.3 The Go dunov Scheme
The non linear upwind scheme is called the Godunov Scheme. The basic idea of this
scheme is
to divide the problem into small spacial regions in which the problem is a Riemann
problem.
Then the solution to the Riemann problem (either analytic or making use of a
Riemann
Solver) is used. We will define the solution of the local Riemann problem as un(x,
*). To
create a set of local Riemann problems, we need piecewise constant data, so we
approximate
u(x):
un
j =
1 ?x
Zxxj+1/2
j-1/2
u(x)dx
where u(x) is the exact solution and un(x, *) := unj for t = tn, xj-1/2 < x <
xj+1/2. This
solution is valid for tn < t < tn+1. Then the solution from our method at each time
step tn
is
Un
j =
1 ?x
Zxxj+1/2
j-1/2
un(x, tn)dx
and likewise,
Un+1
j =
1 ?x
Zxxj+1/2
j-1/2
un(x, tn+1)dx
meaning we take our solution and approximate it as piecewise constant again so that
we
have another set of local Riemann problems.
Step by step this means we
1. Approximate the data as piecewise constant (Ujn))
2. Evolve the data as a Riemann problem (un(x, *))
3. Approximate this solution as piecewise constant (Ujn+1)
23
For the problem of non linear advection, we have
?tun + ?xf(un) = 0
for tn < t < tn+1. Integrating,
1 ?
t Ztntn +1 Zxxj +1/2
j-1/2
(?tun + ?xf(un))dxdt = 0
and so
1 ?x
Rxxj+1/2
j-1/2
un(x,tn+1)dx - ?1x Rxxj+1/2
j-1/2
un(x,tn)dx
?t
+
Rttnn+1 f(un(xj+1/2,t))dt - Rttnn+1 f(un(xj-1/2,t))dt
?x = 0
or, more simply,
un+1
j - unj
?t +
Fn
j+1/2 - Fjn-1/2
?x = 0
where
Fn
j+1/2 =
1 ?
t Ztntn+1 f(un(xj+1/2,t)dt
This is an upwind scheme, and its construction makes it clear that it will only be
valid as
long as the local Riemann problems do not interact, that is the local parts of the
piecewise
data do not collide. Because the data is moving at a speed less than max |f0(u)|,
this method
will be valid as long as
?t < ?x
max |f0(u)|
ie, before neighbouring waves arrive. This is the CFL condition again. To see this,
we can
examine the solution of the Riemann problems. The solution is self similar, so,
locally,
un(x,t) = un #x - xt j+1/2 #
so at x = x
j+1/2, the solution does not depend on time t. we therefore have
Fn
j+1/2 = f(un(xj+1/2)) = f(R(0; unj ; unj+1))
where R xt ; Ujn; Ujn+1# solves
ut + f(u) x = 0
u(x, 0) =
??? uunj+1nj x <x > 0
24
This solution is then only valid b efore waves b egin to intersect. We may also use
this notation
to say
Fj
+1/2 = R(0; uj; uj+1)
We can either use the analytic solution to the Riemann problem, or use a Riemann
Solver.
This is a first order accurate scheme due to the approximation at each step of the
data as
piecewise constant (ie averaging over small intervals). The Godunov scheme gives
better
resolution than the Lax-Friedrichs scheme due to less numerical dispersion. The
viscosity
modified equation is again
?tu + ?xf(u) = C?xuxx
The difference is in the constant C, and Cupwind < CLax -F riedrich.
The Godunov scheme satisfies the entropy condition as long as f(u) is convex. If
f(u)
is convex,
R(0; ul; ur) =
????? f(u0rl) urlu> u< ul < u00 0&< us >s <r 0
f0(u0) = 0
s =
[f]
[u]
Taking convex F(u), we need ?(u) so that ?0(u) = F0(u)f0(u) and ?tF(u) + ?x?(u) =
0. If
?tF(un) + ?x?(un) = 0
for tn = t = tn+1, then
1
?x?t Ztntn+1 Zxxj+1/2
j-1/2
?tF(un) + ?x?(un)dxdt =
1 ?x
Rxxj+1/2
j-1/2
F(un(x, tn+1))dx - ?1x Rxxj+1/2
j-1/2
F(un(x, tn))dx
?t
+
1 ?
t Rttnn+1 ?(un(xj+1/2, t)dt - ?1t Rttnn+1 ?(un(xj-1/2, t)dt
?x = 0
and so
F(unj +1) + F(unj )
?t +
?(R(0; unj ; unj+1)) - ?(R(0; unj-1; unj ))
?x = 0
because we have
1 ?x
Zxxj+1/2
j-1/2
F(un(x, tn+1))dx = F ?1x Zxjx-j+1/12/2 un(x, tn+1)dx!
and by definition
F ?1x Zxjx-j+1/12/2 un(x, tn+1)dx! = F(unj +1)
25
This scheme thus satisfies the discrete entropy inequality
It may b e necessary to use an approximate Riemann solver if the Riemann problem
cannot
b e solved. Roes approximate Riemann solver linearizes the flux of the system so
that for
constant matrix A,
f = Au
Local linearization gives
u t + A(ul, ur)ux = 0
A must have n real eigenvalues and a complete set of eigenvectors (? i, ?i), and
should approximate the Jacobian. The requirements are, in general,
A(ul, ur)(ur - ul) = f(ul) - f(ur)
Diagonalizable with real eigenvalues
A(ul, ur) ? f0(u) as ul, ur ? u
At a shock, f(ur) - f(ul) = s(ur - ul), so ? ? i such that A(ur - ul) = ? i(ur -
ul) with s = ? i
(shock speed). For a scalar equation,
a = f(ur) - f(ul)
u
r - ul
To analyse the scheme, we note that a full set of eigenvectors gives a basis for
the space we
are working in, so
u
r - ul =
n Xp
=1
a
p?p
and so
R(?; ul; ur) = u(?) = ul + X
?
p<?
a
p?p = ur - X
?
p>?
a
p?p
This can of course be recognized as a characteristic decomposition of the
linearized problem.
As a result, the Roe solver gives
FRoe = f(ul) + X
?
p<0
a
p
?
p?p = f(ur) - X
?
p>0
a
p
?
p?p
so
Fj
+1/2 = FRoe(uj, uj+1) = A(uj, uj+1)?pj+1/2 = ? pj+1/2?pj+1/2
(You have to love those subscripts on subscripts)
We now have to determine, once and for all, how to choose A. Consider ? ? [0, 1]
and
q(?) = ui-1 + (ui - ui-1)?
f(ui) - f(ui-1) = R01 df(d?q(?)) d?
= R01 f0(q(?))q0(?)d?
= R01 f0(q(?))(ui - ui-1)d?
= (ui - ui-1) R01 f0(q(?))d?
26
so if we cho ose R01 f0(q(?))d? to be A, we will satisfy the first condition on
A. However,
this may not be an easy integral to evaluate. Roes solution to this question was
to use a
parametrization vector z(q). This must be invertible, so q = q(z). Thus, f(x) =
f(q(z))),
and we can integrate.
z(?) = zi-1 + (zi - zi-1)?
zi = z(ui)
z0(?) = zi - zi-1
then
f(ui) - f(ui-1) = Z0 1 df(d?z(?)) d? = (zi - zi-1) Z0 1 f0(z(?))d? = C (zi - zi-1)
and
ui - ui-1 = Z0 1 dq(d?z(?)) d? = (zi - zi-1) Z0 1 q0(z(?))d? = B (zi - zi-1)
So if we can find z such that B and C are invertible and easy to evaluate, then
A = C B -1.
It is nice to illustrate the scheme with the problem of isothermal flow:
?t? + ?x?u = 0
?t?u + ?x(?u2 + a?2) = 0
and with m = ?u,
U = ?? m? ??
and
f(U) =
??
?u
?u2 + a2?
??
=
??
m
m 2
?
+ a2?
??
so
f0(U) =
??
0 1
a2 - u2 2u
??
We need to choose A, so we need z. Let
z = ?- 1/2
??
?
?u
??
=
??
z1
z2
??
=
??
?
1/2
?
1/2
u
??
then
q(z) = U = z1z
f(U) =
??
z1z2
a2z2
1 + z2
??
27
and we can write U and f(U) in quadratic form:
z =
12
(zl + zr) =
??
z1
z2
??
=
12
??
?
1/2
l + ?
1/2
r
ml
?
1/2
l
+ mr
?
1/2
r
??
and
Ul - Ur =
??
2z1 0
z2 z1
??
(zl - zr)
and
f(Ul) - f(Ur) =
??
z2 z1
2a2z2 2z2
??
(zl - zr)
so
B = ?? 2zz21 z01 ?? C = ?? 2az22z2 2zz12 ??
So, finally, we have
A = C B -1 = ?? a2 -0 1u2 2u ??
where
u =
z2
z1
=
v?lul + v?rur
v?l + v?r
is called the Roe Average.
3.8 Non Linear Stability
In order to determine the stability of methods used to solve non linear equations,
some
tougher analysis is required. The first concept needed I have already mentioned:
total
variation. Total variation may be thought of as a way to measure the oscillatory
nature
of a function:
T V(v) = lim sup
#?0
1#
ZR | v(x) - v(x - #)| dx
Also, if v is differentiable, then
T V(v) = ZR | v0(x)| dx
We can also consider total variation in more dimensions:
T VT(u(x, t)) = lim sup
#?0
1#
#Z0 T ZR | u(x + #, t) - u(x, t)| dxdt + Z0 T ZR | u(x, t + #) - u(x, t)| dxdt#
28
We often consider the 1 norm of our functions:
|| v || 1,T = Z0 T k v(, t)k 1dt = Z0 T ZR | v(x, t)| dxdt
and the space of functions that are finite under that norm:
L1,T = {v|k vk 1,T < 8}
Consider
K = {u ? L1,T| T VT(u) = R, supp(u(, t)) ? [-M, M]?t ? [0, 1]}
K is a compact set.
We can also define the total variation of a discrete solution un
j :
T VT(u) =
8 X
j=-8
N Xn
=0
?t| unj+1 - unj | + ?x| unj +1 - unj |# =
N Xn
=0
?tk unk T V +
N Xn
=0
k un+1 - unk 1
We will call a numerical solution with ?t = k uk.
A numerical method is total variation stable (TV stable) if all approximate
solutions uk
for k < k0 lie in some fixed set K (where R and M depend on the initial data u0 and
the
flux function f(u) but not on the time step k).
Theorem 5. Consider a conservative method with a Lipschitz continuous numerical
flux
F(u;j). Suppose that for each initial data u0, ? some k0 > 0 such that T V(un) =
R ?n, k,
k < k0, nk = T. Then the method is TV stable.
This theorem means that it is sufficient to check TV stability in space under the
right
conditions. We also have the lemma:
If the conditions of the theorem are met, ?a such that k un+1- unk = ak ?n, k k <
k0, nk < T
Proof. (of the Theorem)
T VT(u) = PNn+0 kR + PNn=0 ak = (a + R) P k = (a + R)N K = (a + R)T
Proof. (of the lemma)
The scheme is conservative, so
un+1
j - unj = ??xt [F(un; j) - F(un; j - 1)]
k un+1 - unk 1 = ?t P8j=-8 | F(un; j) - F(un; j - 1)|
F(un; j) depends on a finite number of spacial points, unj-p to unj+q. F is
Lipschitz continuous
in u, so
| F(un; j) - F(un; j - 1)| = C max
-p<i<q
| unj+i - unj+i-1| =
q X
i=-p
| unj+i - unj+i-1|
29
so
k un+1 - unk 1 = C?t
8 X
j=-8
q X
i=-p
|unj+i - unj+i-1| = C?t
q X
i=-p
T V(un) = C?tR(q + p - 1)
and so a = CR(p + q - 1)
So, under the conditions of the theorem, it is sufficient that
T V(un) =
8 X
j=-8
|unj+1 - unj |
is bounded for the method to be TV stable. Furthermore,
Theorem 6. Suppose uk is generated by a numerical method in conservation form with
a
Lipschitz continuous numerical flux consistent with some scalar conservation law.
If the
method is TV stable, then the method is convergent (converges to a weak solution).
A stronger condition for a method is Total Variation Diminishing (TVD), meaning
T V(un+1) = T V(un)
Unsurprisingly, TVD ? TV stable (as a trivial consequence of theorem 5), and so a
TVD
method that is consistent and conservative converges to a weak solution.
Another desirable property in a numerical method the absence of numerical
oscillations. A
method is Monotonicity Preserving if
u0
j = u0j+1?j ? unj = unj+1?j, n
Theorem 7. If a method for a hyperbolic problem is TVD, it is monotonicity
preserving
Proof. T V(un) = P8j=-8 |unj+1 - unj | = T V(u0) = P8j=-8 |u0j+1 - u0j| because the
scheme is
TVD. Monotonicity of the initial data implies
T V(u0) =
8 X
j=-8
|u0j+1 - u0j| = |u0-8 - u08|
and because of the finite wave speed of a hyperbolic problem, un-8 = u0-8 and un8 =
u08.
This means that
T V(u0) = u0-8 - u08 =
8 X
j=-8
un
j+1 - unj = T V(un)
by the triangle inequality. The scheme is TVD, so equality must hold. This implies
that
T V(un) =
8 X
j=-8
un
j+1 - unj = un-8 - un8
which can only be true if unj = unj+1, so the scheme is monotonicity preserving.
30
For analytic weak solutions of hyp erb olic problems,
k u(, t2)k 1 = k u(, t1)k 1?t2 = t1
and i u and v are both entropy solutions with different initial data,
k u(, t2) - v(, t2)k 1 = k u(, t1) - v(, t1)k 1?t2 = t1
We would like our numerical methods to in some sense preserve this, using the norm
k unk 1 = X
j
|unj |
A numerical method is l1 contracting if
k un+1 - vn+1k 1 = k un - vnk 1
? un, vn numerical solutions with different initial data.
Theorem 8. l1 contracting ? TVD
Proof. Let u, v be solutions with different initial data.
k un+1 - vn+1k 1 = k un - vnk
T V(un+1) = Pj |unj+1+1 - unj +1|
let vn
j = unj+1, ? vjn is a numeric solution.
? T V(un+1) = X
j
|vjn+1 - unj +1| = X
j
|vjn - unj | = T V(unj )
Consider for a non linear advection equation ut + f(u)x = 0 the scheme
un+1
j - unj
?t +
f(unj ) - f(unj-1
?x = 0
so that
un+1
j = unj - ??xt f(unj ) - f(unj-1)#
vn+1
j = vjn - ??xt f(vjn) - f(vjn-1)#
then let w = u - v. We can show that this method is l1 contracting by showing that
k wjn+1k = k wjnk.
wn+1
j = wjn - ?t
?x
#f(unj ) - f(vjn) - f(unj-1) + f(vjn-1)#
31
and by the mean value theorem, ?? so that
wn+1
j = wjn - ?t
?x
#f0(?jn)wjn - f0(?jn-1)wjn-1# = #1 - ??xt f0(?jn)# wjn + ?t f0(?jn-1)wjn-1
?
|wjn+1| = |1 + ??xt f0(?jn)||wjn| + | ??xt f0(?jn-1)||wjn-1|
?x Pj |wjn+1| = ?x Pj |1 + ??xt f0(?jn)||wjn| + ?x Pj | ??xt f0(?jn-1)||wjn-1|
if the CFL condition is met, then 1 - ??xt f0(u) = 0 ?u, and ??xt |f0(u)| = 1 ?u,
which means
that
k wn+1k 1 = ?x X
j
#1 - ??xt f0(?jn)# |wjn|+?x X
j
# ??xt f0(?jn-1)# |wjn-1| = ?x X
j
|wjn| = k wnk 1
so this scheme is l1 contracting under the CFL condition.
If u(x, t) and v(x, t) are analytical solutions which satisfy the entropy
conditions, and
u0(x) = v0(x)?x, then u(x, t) = v(x, t)?x, t. Again, we want to preserve this
property
in our numerical method.
A method Un+1 = H(Un) is called a Monotone Method if
?
?un
j
H(un) = 0?j
then, if u0j = vj0?j,
un+1
j - vjn+1 = H(unj ) - H(vjn) = X
k
?H
?uk wkn(unk - vkn) = 0
and so (inductively) we see that the property is preserved.
Theorem 9. Any monotone method is l1 contracting.
Proof. We need to show that k un+1 - vn+1k 1 = k un - vnk 1.
Suppose
un+1
j = H(unj-k, unj-k+1, ..., unj+k)
let
un
j = (unj-k, ..., unj+k); snj = sgn(uj - vjn)
k un+1 - vn+1k 1 = ?x Pj |unj +1 - vjn+1| = ?x Pj |H(unj ) - H(vjn)|
= ?x Pj snj +1 H(unj ) - H(vjn)#
and
H(unj ) - H(vjn = R01 dd?H ?unj + (1 - ?)vjn# d?
= R01 P2l=1k+1 ?uHl (?jn(?))(unj-k+l-1 - vjn-k+l-1)d?
32
where ?jn(?) = ?unj + (1 - ?)vjn# . Let m = j - k + l - 1. Then
?x Pj snj +1 H(unj ) - H(vjn)# = ?x Pj snj +1 R01 P2l=1k+1 ?u Hl (?jn(?))(unm -
vmn )d?
= ?x Pm(unm - vmn ) R01 P2l=1k+1 sm+k-l+1 ?uHl (?mn +k-l+1(?))d?
= ?x Pm | unm - vmn | R01 P2l=1k+1 ?uHl (?mn +k-l+1(?))d?
= ?x Pm | unm - vmn |
R01 P2l=1k+1 ?uHl (?mn +k-l+1(?))d? = 1 because:
un+1
j = unj - ?t
?x
#F(unj-k+1, ..., unj+k) - F(unj-k, ..., unj+k-1)#
so
un+1
k = H(u1, .., u2k+1) = uk+1 - ??xt #F(un1 , ..., unj+k-1) - F(un2 , ..., unj+k)#
?H
?uk +1 = 1 - ??xt # ?u?Fk +1 (u1, ..., uj+k-1) - ?u?Fk (u2, .., uj+k)#
?H
?u 1 = -
?t
?x
?F
?u 1 (u1, ..., uj+k-1)
?H
?u
j +k
=
?t
?x
?F
?u
j +k
(u2, ..., uj+k)
and for all i 6= k + 1, 1, or j + k,
?H
?ui = -
?t
?x # ?u?Fi (u1, ..., uj+k-1) - ?u?Fi-1 (u2, ..., uj+k)#
so all of these cancel except for the leading 1 in ?u?kH+1 and so the sum is 1.
Finally, we present a couple of theorems without proof:
Theorem 10. (Due to Crandall & Majda)
The numerical solution of a monotone method converges to the entropy solution of
the scalar
conservation laws.
Theorem 11. (Due to Harten, Lax, & Hyman)
A monotone method is at best first order.
We have now developed the following hierarchy:
Monotone Method ? l1 Contracting ? TVD ? Monotonicity Preserving
33
3.9 High Resolution Sho ck Capturing Schemes
Monotone metho ds are at most first order, so if higher accuracy is desired,
numerical oscillations will o ccur. To deal with this, artificial viscosity can b e
intro duced to damp en artificial
oscillations. For example, if we want to improve the (second order) Lax-Wendroff
scheme
for a linear problem:
ut + aux = 0
un+1
j = unj - ?(unj+1 - unj-1) + 12?(unj+1 - 2unj + unj-1)
? = a ?t
?x is the Courant number
we need to add an artificial viscosity term:
kQ(unj+1 - 2unj + unj-1)
so the modified scheme is
un+1
j = unj - ?(unj+1 - unj-1) + 1
2
?(unj+1 - 2unj + unj-1) + kQ(unj+1 - 2unj + unj-1)
Inspecting the truncation error of this modified method:
L(x, t) = LLW - Q(unj+1 - 2unj + unj-1) = O(h2)
so the scheme is still second order if Q is chosen wisely. The last challenge is
then to choose
the parameter Q so that we do not loose second order convergence, but large enough
to
dampen the numerical oscillations. The way to do this is to make Q depend on u,
making
it larger near shocks, where these oscillations occur. If Q is constant, it cannot
be chosen to
dampen all the oscillations, which is consistent with the theorem by Godunov:
Theorem 12. A linear, monotonicity preserving method is at most first order
So, if we want to preserve monotonicity, we need Q ~ 1/h, and thus have a first
order
scheme. Clearly, a higher order, monotonicity preserving scheme must be non linear,
even
for this linear problem. So we will need to choose a function Q(u) which is large,
on the
order of 1/h, near the shock and very small away from the shock.
3.9.1 Flux Limiter Metho ds
We can manipulate the flux of a method, splitting it according to areas of high and
low flux:
FH(u; j) = FL(u; j) + (F(u; j) - FL(u; j))
and add a limiter
FH = FL + f(u; j)(FH - FL)
where
f(u; j)
??? 1 smooth region0 near discontinuity
34
F = FH + (1 - f)(FL - FH)
this the Flux Corrected Transport (FCT) Method. Applying it to our example, the
LaxWendroff method:
un+1
j = unj - ?(unj+1 - unj-1) + 12 ?(1 - ?)(unj+1 - 2unj + unj-1)
FH(u; j) = auj + 12 a(1 - ?)(uj+1 - uj)
Let F(u; j) = auj + 12 a(1 - ?)(uj+1 - uj)fj
and we use the smoothness indicator
?
j =
uj - uj-1
uj+1 - uj
which has the property that, in smooth regions, ?j ~ 1, and at discontinuities, |?
j| # 1. So
we can easily create fj = f(?j) with the properties we require of it.
Theorem 13. The flux limiter method F(u; j) given above is consistent with the
equation
ut + aux = 0 provided f(?) is a bounded function. It is second order accurate (on
smooth
solutions with u
x bounded away from 0) provided f(1) = 1 with f Lipschitz continuous at
? = 1.
so this scheme becomes:
un+1
j = unj - ?t
?x # a(unj - unj-1) + 12 a(1 - ?) #(unj+1 - unj )fj - (unj - unj-1)fj-1# #
= uj - #? - 12 ?(1 - ?)fj-1# (uj - uj-1) - 12 ?(1 - ?)fj(uj+1 - uj)
In general, suppose the scheme is
un+1
j = uj - Cj-1(uj - uj-1) + Dj(uj+1 - u - j)
Theorem 14. In order for the above scheme is be TVD, the following conditions on
the
coefficients are sufficient:
Cj
-1 = 0 Dj = 0 0 = Cj + Dj = 1 ?j
Proof.
un+1
j+1 - unj +1 = unj+1 - unj - Cj(unj+1 - unj )+ Cj-1(unj - unj-1)+ Dj+1(unj+2 -
unj+1) - Dj(unj+1 - unj )
?
un+1
j+1 - unj +1 = Cj-1(unj - unj-1) + (1 - Cj - Dj)(unj+1 - unj ) + Dj+1(unj+2 -
unj+1)
35
then
Pj | unj+1+1 - unj +1| = Pj Cj-1| uj - uj-1| + Pj(1 - Cj - Dj)| uj+1 - uj| + Pj
Dj+1| uj+2 - uj+1|
= Pj Cj| uj+1 - uj| + Pj(1 - Cj - Dj)| uj+1 - uj| + Pj Dj| uj+1 - uj|
= Pj | uj+1 - uj|
= T V(n)
? T V(n + 1) = T V(n)
In our scheme, we have
Cj
-1 = ? - 1
2 ?(1 - ?)fj-1
D
j = -
12
?(1 - ?)fj-1
Because ? ? (0, 1), f > 0, we cannot satisfy the theorem. We can rewrite the scheme
as
un+1
j = unj - #? + 12 ?(1 - nu)fj-1 - 12 ?(1 - ?)fj uj+1- -uj-u1j # (uj - uj-1)
so that
Cj
-1 = ? + 1
2 ?(1 - nu)fj-1 - 12 ?(1 - ?)fj uj +1-uj
j-uj- 1
D
j = 0
and
Cj
-1 = ? + 1
2 ?(1 - nu)fj-1 - 12 ?(1 - ?) f?j
j
= ? - 1
2 ?(1 - ?) #-fj-1 + f?jj #
= ? #1 - 12(1 - ?) # f?jj - fj-1##
and 0 = ? = 1 so
0 = Cj-1 = 1 ? | f(?j)
?
j
- f(?j-1)| = 2
so we can pick a function f, which need not be unique, and have a method which is
TVD.
Some possibilities for f are
Van Lear f(?) = 1+|?|+|??|
superbee f = max{0, min{1, 2?)}, min{?, 2}}
36
3.9.2 Slop e Limiter Metho ds
A more geometrical approach uses a scheme that lo oks like
uj+1/2 = uj +
?x
2
?x
uj
where a piecewise linear approximation to ?xu is used:
?x
uj
??? sj-+ = uj+1-?ux-j-u1j
When the solution is smooth, it doesnt matter which you choose. However, at a
discontinuity
a choice must be made. Choose
?x
uj =
??? sgn(sj+) min0 if{|sj-|, |sj+|} otherwisesj+, sj- = 0
or, using Van Lears criteria, choose
?x
uj =
????? min{|ssj-0 ifj0|, |sj+|} if |sj0| <otherwises2 minj+, sj-{|=sj-0|, |sj+|}
This can be written
?x
uj ~= f(?j) uj+1 - uj
?x
where f(?) is the Van Lear choice:
f(?) = |?| + ?
1 + ?
3.10 Central Scheme s
Recalling the Godunov scheme, we took the average of each space grid at time tn.
Now, we
will take
un+1
j+1/2 =
1 ?x
Zxxj+1
j
u(x, tn+1)dx
so that
1
?t?x Rxxj+1
j
Rttnn+1 un + f(un)xdtdx = 0
1 ?
x Rxxjj+1 un(x,tn+1)dx- ?1x Rxxjj+1 un(x,tn+1)dx
?t +
1 ?
x Rttnn+1 f(un(xj+1,t))dt- ?1x Rttnn+1 f(un(xj,t))dt
?t = 0
37
So, if we assume that each time step is small enough so that uj + 1n is constant,
un+1
j+1/2 -
un
j +unj+1
2
?t +
f(unj+1) - f(unj )
?x = 0
This method is much like the Lax-Friedrich method, but on a staggered grid. This
also puts
a different constraint on the time step size: it now must be half the CFL number:
|f0(u)| ?t
?x =
12
This is a first order method. However, if we construct the method with piecewise
linear
spacial approximations (rather than piecewise constant), we can come up with a
second
order scheme. We need to approximate
uj+1/2 - # 12(unj + unj+1) + ?8x(u0j - u0j+1)#
?t +
1 ?
t Rttnn+1 f(un(xj+1,t))dt - ?1t Rttnn+1 f(un(xj,t))dt
?x
We do that with the midpoint rule (to get a second order scheme)
Rttnn+1 f(un(xj,t))dt ?tf(un(xj,t))
u(xj,tn+1/2 u(xj,tn) + ?2t?tu(xj,tn) = unj - ?2t?xf(unj )
so the 2nd order scheme is
u
n+1/2
j = unj - ?2t?xf(unj )
un+1
j+1/2 =
12
(unj + unj+1) + h8 (u0j - u0j+1) + ??xt hf(unj+1+1/2) - f(unj +1/2)i
where u0
j and ?xf(unj ) can be defined using slope limiters.
3.11 Relaxation Scheme
For the problem
ut + f(u)x = 0
we can approximate the equation with a system
ut + vx = 0
vt + aux = 1
# (f(u) - v)
which can be written
??
uv
??
t
+
??
0 1
a 0
??
??
uv
??
x
=
??
0
1#
(f(u) - v)
??
38
we can show that the matrix
A = ?? a0 10 ??
has eigenvalues ?(A) = va, so this is a hyperbolic system. Note:
v = f(u) - #(vt + aux)
so as epsilon ? 0, v ? f(u). This gives
v = f(u) - #(f(u) t + aux) + O(#2)
and
f(u) t = f0(u)ut = -f0(u)vx = -f0(u)(f0(u)ux + O(#))
so
v = f(u) - #(-(f0(u)) 2 + aux) + O(#2)
then
ut + f(u) x = #((a - (f0(u)) 2)ux) x
so we need (f0(u)) 2 = a, or rather |f0(u)| = va. Note that va is the
characteristic speed of
the system we are using. We next diagonalize the system to get
#u v1av# t va #u v1a v# x = RHS
We then do an operator splitting. We will use an upwind scheme for the convection
term
va #u v1av# x:
#u + v1av# j+1/2 = #u + v1av# j
#u - v1av# j+1/2 = #u - v1av# j
which can be solved to find
vj+1/2 =
vj+vj+1
2 -
va
2 (uj+1 - uj)
?tuj + vj+1/2?-xvj-1/2 = 0
We then relax using a local Lax Friedrich scheme:
ut = 0
vt = 1
# (f(u) - v)
?
vj+1/2 =
f(uj) + f(uj+1)
2
-
va
2
(uj+1 - uj)
We can make this second order by using, instead of simple upwind,
#u + v1av# j+1/2 = #u + v1av# j + ?2x ?x #u + v1av# j
#u - v1av# j+1/2 = #u - v1av# j - ?2x ?x #u - v1av# j
using slope limiters.
39
Chapter 4
Front Propagation
Often of interes t is the propagation of fronts, i.e. the movement of curves
across, for
example, space. For instance, we may want to simulate the surface of a fireball
explo ding
out of a car after John McClane sho ots it with a grenade launcher. To do that, we
dont
need to keep track of what might b e happ ening b ehind the surface of the fireball
(which
would b e invisible to our audience anyway), so we only need to simulate the front
s urface.
The propagation sp eed F of the front G may depend on a variety of factors,
including
local geometry (normal curvature, etc)
physical properties (temperature, pressure, etc)
We will assume that F is known. The Lagrange Method is to parametrize G:
G(t) : ~x(s, t) 0 = s = S
dx
dt = F~n
x 0(s) = F~nx
y 0(s) = F~ny
~n =
??
?
y s
vx 2s +ys2
-xs
vx 2s +ys2
??
?
Perhaps F = F(?) where ? is mean curvature:
? =
yssxs - xssys
(x2s + ys2) 3/2
Simply keeping track of a section ?s of the curve corresponding to si = s = si+1
presents
problems. One is that, if the front expands, ?s may stretch. Another is the
possibility of
topological changes such as the merging of fronts.
The Eulerian Method, in contrast, does not rely on a parametrization of G, but
instead
40
on the use of level sets. This requires that we define a multivariable function
F(~t, t) such
that
G(~x, t) = {~x|F(~x, t) = 0}
and so
F(~x, t) = 0
F t + x 0(t) ?~xF = 0
Ft + F~n ?~xF = 0
and note
~n = ?~xF
|?~xF|
so the level set equation is
Ft + F|?F| = 0
We can then just solve this on a fixed grid. This method can handle topological
changes
- these are simply reflected in F(, t) - and this method does not lose accuracy as
a front
expands. To choose an initial condition F(~x, 0, we can use linear sloping sides:
F(~x, t) =
??? -dist(dist(x,x,G) ifG) if x 6? Ginterior
The level set equation
Ft + F|?F| = 0
is an example of a Hamilton - Jacobi equation:
st + H(?s) = 0
and H is called the Hamiltonian. In one dimension this is
st + H(sx) = 0
and if we let u = s
x, this becomes
ut + ?xH(u) = 0
and so this is a non linear advection type equation with flux H. We can solve it:
?tuj + Hj+1/2 - Hj-1/2
?x = 0
where we can use any method that is appropriate for the approximation of the flux
Hj+1/2.
We also need to approximate u = sx:
uj =
sj+1/2 - sj-1/2
?x
41
so we are dealing with
?t # sj +1/2?-xsj-1/2 # + Hj+1/2?-xHj-1/2 = 0
?tsj+1/2 + Hj+1/2 = 0
?tsj + Hj = 0
with H
j = H(uj-1/2,uj+1/2), or rather Hj = H sj-?sxj-1 , sj+1?x-sj # , so we have
?tsj + H #sj -?xsj-1 , sj+1?-x sj # = 0
42
Chapter 5
Finite Volume Metho ds For PDEs
Consider the equation in two dimensions:
?tu + ? f = 0
and consider a non uniform grid so
?xi = xi+1/2 - xi-1/2
?yj = yj+1/2 - yi-1/2
and denote
u
ij =
1
?xi?yj Zxxi+1/2
i-1/2
Zyj+1/2
yj-1/2
u(x,y)dxdy = 1
Aij Z Z?
ij
u(x,y)dxdy
where A
ij = ?xi?yj is the area of the rectangle ?ij. Then our problem is:
?tuij + A1
ij
R R?
ij
? fdxdy = 0
?tuij + A1
ij
R??
ij
f ~ndl = 0
and because we are using rectangular regions ?ij,
?tuij + 1
A
ij Z??i+1/2,j fx(xi+1/2,y)dy - Z??i-1/2,j fx(xi-1/2,y)dy
+ Z??
i,j+1/2
fy(x,yj+1/2)dx - Z??
i,j-1/2
fy(x,yj-1/2)dx! = 0
and we can denote
fxi+1/2 = ?1yj Zyjy-j+1/12/2 fx(xi+1/2,y)dy
and approximate this with the trapezoidal rule:
fxi+1/2 1
2
#fx(xi+1/2,yj+1/2) + fx(xi+1/2,yj-1/2)#
43
So we b egin with u and approximate in x to get:
ui+1/2,j =
?xi+1uij + ?xiui+1,j
?xi + ?xi+1
and then approximate in y to get
ui+1/2,j+1/2 =
?yj+1ui+1/2,j + ?yjui+1/2,j+1
?yj + ?yj+1
So at each point, we are using a weighted average of the 4 immediate neighbouring
points.
We can approximate the other integrals and ultimately arrive upon
?tuij +
?yj fxi+1/2 - fxi-1/2# + ?xi fyj+1/2 - fyj-1/2#
?xi?yi
44
Chapter 6
Sp ectral Metho ds For PDEs
Sp ectral metho ds use trigonometric interp olation to get exp onential accuracy at
a higher
computational cost. These metho ds are global metho ds, which leads to this high
cost.
The advantage is that, for a smo oth function, the accuracy is O(?xm) ?m = 1.
Finite
difference and volume methods are ultimately based on polynomial interpolation.
Spectral
methods take advantage of the improved accuracy of trigonometric interpolation,
especially
for periodic functions.
6.1 Trigonometric Interp olation
Suppose f is periodic with period t > 0, that is to say t is the smallest real
number such
that f(x + t) = f(x) ?x. Let t = 2p.
If
Pn
(t) = a0 +
n Xj
=1
aj cos(jt) + bj sin(jt)
and if | an| + | bn| = 0, we say that Pn(t) is a trigonometric polynomial of degree
n. The
interpolating points are
0 = t0 = t1 = ... = tN < 2p
and so f(tj) = Pn(tj) for 0 = j = N = 2n + 1. We can represent this in a number of
alternative ways:
Pn
(t) = a0 + Pnj=1 aj eijt+2e-ijt + bj eijt -2ei -ijt
= a0 + Pnj=1 12(aj - ibj)eijt + Pnj=1 12(aj + ibj)e-ijt
= Pnj=-n cjeijt
where c0 = a0, cj = 12(aj + ibj) and c-j = 12(aj - ibj) and i = v-1 is the
imaginary unit
(and not an index!). So if we let z = eit, then
Pn
=
n X
j=-n
cjzj
45
and so
znP
n(t) =
n X
j=-n
cjzj+n
or
znP
n(t) =
2n
Xj
=0
cj-nzj
and we need P
n to satisfy f(tj) = Pn(tj) 0 = j = N where tj = j 2n2+1p 0 = j = 2n. This is
a question, then, of solving for the coefficients of the series cj. This is
recognizable by this
point as a discrete Fourier series, and the process for finding the coefficients is
the familiar
one - i.e. make use of the orthogonality of the exponentials:
f(tk) = PNj=0-1 cjeijtk 0 = k = N - 1
PNk=0-1 e-iltkf(tk) = PNk=0-1 PNj=0-1 cjei(j-l)tk 0 = l = N - 1
= PNk=0-1 PNj=0-1 cjei(j-l)k 2Np
= PNk=0-1 cj PNj=0-1 ei(j-l)k 2Np
and we make use of the orthogonality of the exponentials. First, let
z = ei 2Np (j-l)
so
N-1
Xk
=0
e-iltkf(tk) =
N-1
Xk
=0
cj
N-1
Xj
=0
ei(j-l)k 2Np =
N-1
Xk
=0
cj
N-1
Xj
=0
zk
and
N-1
Xj
=0
zk =
??? 01 j 6= l
so we have
N-1
Xk
=0
e-iltkf(tk) = Ncl
and finally we have found the lth coefficient:
cl =
1N
N-1
Xk
=0
f(tk)e-iltk 0 = l = N - 1
For the sake of completeness (and because its pretty easy) lets prove the
orthogonality
relation we used. If j = l
z = ei 2Np (j-l) = e0 = 1
46
and so clearly
N- 1
Xj
=0
zk =
N- 1
Xj
=0
1 = N
If j 6= l, we need to first remember that, for any integer m,
ei2pm = 1
and, if x is NOT an integer:
ei2px 6= 1
and the series relation found in most calc 2 textbooks:
N-1
Xk
=0
ak = 1 - aN
1 - a
and so, because zN = ei2p(j-l),
N-1
Xk
=0
zk = 1 - zN
1 - z
= 0
As was mentioned, this is the same basically a discrete version of a Fourier
series, and so, unsurprisingly, the coefficients cj are called Discrete Fourier
Coefficients and the trigonometric
polynomial
Pn
(t) =
N-1
Xj
=0
cjeijt
is called the Discrete Fourier Transform. It is not very easy to compute the
coefficients cj,
as it is necessary to compute a large sum and many function evaluations. Luckily,
the Fast
Fourier Transform exists to compute these quickly.
It is interesting to look at the continuous Fourier transform and discrete
counterpart for a
function f(t) that is periodic with period 2p:
Continuous f(?) = 21p R02p f(t)e-i?tdt
Discrete cl = 1
N PNk=0-1 f(tk)e-iltk
and notice that the discrete transform is the trapezoidal rule approximation to the
continuous
transform. The same is true of the inverse transforms:
Continuous R02p f(?)ei?td?
Discrete PNk=0-1 cjeiltk
For periodic functions, the trapezoidal rule has exponential accuracy, so spectral
methods
(which are built after all on this transform) have exponential accuracy for
periodic functions.
47
6.1.1 The Fast Fourier Transform
As I mentioned, the computational cost of a discrete Fourier transform is large, in
fact the
op eration count is O(N2). That is very slow, so the Fast Fourier Transform (FFT)
was
developed to improve this. I will discuss the theory of the FFT, not
implementation.
For the FFT, we need N = 2m for some integer m. For each cl, if l is even, l = 2l0,
and we
can compute
cl = c2l 0 =
1 N
N/2-1
Xk
=0
f(tk)e-i2l 0k 2Np + 1
N
N-1
X
k=N/2
f(tk)e-i2l 0k 2Np
now let f(tk) = ak and write the second term in the transform as
1 N
N-1
X
k=N/2
f(tk)e-i2l 0k 2Np = 1
N
N/2-1
Xk
=0
ak+N/2e-i2l 0(k+N/2) 2Np
so that
cl = c2l 0 =
1 N
N/2-1
Xk
=0
(ak + ak+N/2e-il 0k N2/p2
Now we have a discrete Fourier transform on a0 + aN/2, ..., aN/2-1 + aN-1.
If l is odd, then l = 2l0 + 1.
cl = c2l 0+1 =
1 N
N-1
Xk
=0
f(tk)e-i(2l 0+1)k 2Np = 1
N
N-1
Xk
=0
#ake-ik 2Np # e-2l 0k 2Np
so if we say bk = ake-ik 2Np , we have the same problem as before, and can do the
same thing
to get the discrete Fourier transform on b0 + bN/2, ..., bN/2-1 + bN-1:
c2l 0+1 =
1 N
N/2-1
Xk
=0
(bk + bk+N/2)e-il 0k N2/p2
and
bk = ake-ik 2Np
bk+N/2 = ak+N/2e-i(k+ N2 ) 2Np = -ak+N/2e-ik 2Np
so
bk + bk+N/2 = (ak - ak+N/2)e-ik 2Np
We now have 2 Fourier transforms on N/2 data, rather than 1 on N data. We can
continue
this until we have N transforms on 1 piece of data, which is very trivial. The
operation
count on the FFT is O(N log2 N).
48
6.2 Basics ab out the Fourier Transf orm
Consider u(x) for x ? R. The Fourier Transform is:
u(k) = Z-88 e- ik xu(x)dx
and the inverse Fourier transform is
u(x) = 1
2p Z-88 eikxu(k)dk
where k ? R is called the wave number.
Now consider x = h Z = {xj = jh| - 8 < j < 8}. Now, k is bounded in a domain of
length
2p
n
because of aliasing of k:
eik 1 x 6= eik 2x, k1 6= k2, x ? R
If x is only defined on h Z then eik 1 x = eik 2x if k1 - k2 is an integer multiple
of 2p/h, so we
want
k ? [- p
h ,
ph
Let v(x) = vj and x = xj. The relevant transforms are
FT (v)(k) = h P8j=-8 e-ikxj vj
IFT v
j =
1
2p R-p/ph/h eikxj v(k)dk
Now, if we want an interpolant P(x) such that P(xj) = vj, we can use
P(x) = 1
2p Z-pp//hh eikxj v(k)dk
then
P(k) =
??? v(0 otherwisek) k ? [- ph , ph]
because
P(k) = Z-88 e-ikx 21p Z-pp//hh v(l)eilxdldx
=
1
2p Z-pp//hh dlv(l) Z-88 e-i(k-l)xdx = Z-pp//hh v(l)d(k - l)dl
One of the most useful facts about Fourier transforms is that for a function w(x),
w 0(k) = -ikw (k)
and in fact,

w(n)(k) = (-ik) nw (k)

This is not too hard to show using integration by parts. More on it can be found in
any
analysis text.
49
6.3 Sp ectral Differentiation
Having found an interp olant for our data, we need to use this to approximate the
derivative
of our data. The obvious thing to do is to set vj0 = P0(xj). Then write
vj =
8 X
m=-8
v
mdj-m
where d
j is the Kronecker delta at 0 (note dj(k) = h).
Pd = h
2p Z-pp//hh e-ikxdx =
??? sin(px1px/h/h) x 6= 0
let
sh =
??? sin(px1px/h/h) x 6= 0
This is called the sinc function. So,
Pv
=
8 X
m=-8
v
msh(x - xm)
so
v0
j , Pv0(xj) =
8 X
m=-8
v
ms0h(x - xm)
s0
h(xj) =
??? (-jh01)j j = 06= 0
and for a second derivative
s00
h(xj) =
??? 2(-j3-21)hhp2j2+1 j = 06= 0
This is not a very easy thing to deal with, and so we will find a better way to
come up with
an approximation to the derivative.
We want the derivative of P(x) = 21p R-p/ph/h v(k)eikxdk. However, we know
something about
P(k), in fact we know P (k) = v(k). For continuous data, the method we would use
would
then be, given vj, -8 < j < 8:
1. Take FT to find v(k)
2. let w(k) = (-ik) nv(k)
3. take IFT on w to find w(k)
50
4. set v(n)
j , w(k)
But we have discrete data. We can use the discrete transforms
Discrete FT vk = h
N Xj
=1
e-ikxjv
j k = N/2 + 1,..., N/2
Discrete IFT v
j =
1
2p
N/2
X
k=-N/2+1
eikxjvk j = 1,...,N
Our interpolating polynomial is
P(x) = 1
2p
N/2
X
k=-N/2+1
eikxvk
We notice that
ei N2 x = cos #N2 x# + i sin #N2 x#
and because x
j = jh = j 2np,
ei N2 xj = cos(jp)
and the derivative is 0, but
#ei N2 x# 0 = iN2 ei N2 x
is always complex. To fix this, we can define v-n/2 = vN/2. Then,
vk = h
N/2
X
k=-N/2
eikxjv
j
and
P(x) = 1
2p
N/2
X
k=-N/2
eikxvk
and v0
j = P0(xj). So, with the discrete data, the process is
1. Take DFT to find vk, k = -N/2 + 1,..., N/2
2. let wk = (-ik)nvk, wN/2 = 0
3. take DIFT on wk to find wk
4. set v(n)
j , wk
using the FFT for the transforms.
51
6.4 Smo othness & Sp ectral Accuracy
Smo oth functions have fast decay in their Fourier co efficients, which makes the
discrete
Fourier transform more accurate. Furthermore, b etter accuracy will b e found in
functions
that decay at infinity. Assume
u(x) ? Cm
u(k) = Z-88 eikxu(x)dxu(n)(x) ? 0 as x ? 8
for n = 0, 1, ..., m. Integrating by parts, we see
u(k) = - 1
ik Z-88 u0(x)eikxdx
Repeating m times:
u(k) = - 1
(ik)m Z-88 u(m)(x)eikxdx ~ #k1 # m
For large wave number, u decays. For large m, u decays very fast in Fourier
space. Recall
that error from discrete Fourier transforms is related to the fact that these give
bounded k,
so this decay is a good thing. In the discrete transform, k ? [-p/h, p/h] and the
error is due
to k 6? [-p/h, p/h]. These terms, for smooth functions, are tiny and so the error
is not very
big.
Theorem 15. Let u(x) ? L2( R), with Fourier transform u(k)
1. If u has p - 1 continuous derivatives in L2( R) for p = 0 and the pth derivative
is of
bounded variation then u(k) = O # |k|1p +1 # as k ? 8
2. If u has infinitely many derivatives in L2( R), then u(k) = O # |k1| n # ?n = 0
as k ? 8.
The converse also holds.
For example, consider
s(x) =
??
12
|x| < 1
0 |x| = 1
The Fourier transform is
s(k) = 1
2 Z-88 eikxs(x)dx = 12 Z-11 eikxdx = 21ikeikx| 1-1 = sin(kk)
and so this decays as k1 .
52
6.4.1 Convolutions
The op eration
u * v = Z-88 u(y)v(x - y)dy = Z-88 u(x - y)v(y)dy
is called a convolution. These have the property that
u[* v = uv
and so convolving a function can increase smoothness.
6.4.2 Sp ectral Approximation
Let u(x) ? L2( R) and suppose u0(x) is of bounded variation. Let v be the grid
function
defined on h Z by vj = u(xj) for j = 1, ..., N (so v is the set of discretized data
from u).
Then
v(k) =
8 X
j=-8
u #k + 2pjh # = u(k) + X
j6=0
u #k + 2pjh #
The error
Xj6
=0
u #k + 2pjh #
is called the Aliasing Error. This error is small if u decays fast, which we have
seen is the
case if u(x) is smooth. Furthermore,
Theorem 16. The following estimates hold uniformly for all k ? [- ph , ph ]
1. if u has p - 1 continuous derivatives in L2( R) for some p = 1 and a pth
derivative of
bounded variation, then |v(k) - u(k)| = O(hp+1) as h ? 0
2. if u has infinitely many continuous derivatives in L2( R), then |v(k) - u(k)|
= O(hm)
as h ? 0 ?m = 0 (exponential or spectral accuracy).
We also have Parsevals Identity: v2pk uk L 2 = k uk L 2, so if |v(k) - u(k)| is
small, then
|v(k) - u(k)| is also small.
Let the IFT of v be P(x), our interpolating polynomial.
Theorem 17. Accuracy of Fourier Spectral Differentiation
Let u ? L2( R) have a ?th derivative (? = 1) of bounded variation and let ? be
the ?th spectral
derivative of u on h Z. Then ?x ? h Z,
1. If u has p - 1 continuous derivatives in L2( R) for some p = ? + 1 and a pth
derivative
of bounded variation, then |?j - u(?)(xj)| = O(hp+1-?) as ? ? 0.
2. If u has infinitely many continuous derivatives in L2( R), then |?j - u(?)(xj)|
= O(hm)
as h ? 0 ?m.
It is important to notice that this accuracy is only achieved for periodic
problems. For non
periodic functions on bounded domains (which are generally necessary for a computer
to deal
with the problem), Gibbs Phenomenon occurs. This means that trigonometric
interpolation
introduces oscillations at discontinuities (which exist at the boundaries of the
problem) of
O(1) amplitude. These oscillations do not decay, even as h ? 0, N ? 8.
53
6.5 Non Perio dic Problems - Chebyshev Points
In p olynomial interp olation oscillations o ccur, esp ecially near the b oundary
of the interp olated region. On a uniform grid, error near the b oundaries is O(N),
which is, of course,
extremely bad. To improve the interpolation, we need to choose a grid with more
points
centered around the boundary.
Assume our domain is [-1, 1] (a simple transformation can get us to any domain we
need,
of course). Choose
xj = cos # jpN # j = 0, ..., N
This puts more points near the boundary of domain because the curve on which the
points
lie is flatter near x = -1 and x = 1. This is also suitable for problems with
boundary layer
issues because more points are near the boundaries. Polynomial interpolation using
this grid
and differentiations will give accuracy for f(?)(x) of O(hN+1-?). However the
differentiation
matrix:
??
?
v0
0
v0
1
.
v0
N
??
?
=
??
?
.
.
.
Differentiation
Matrix
.
.
.
??
?
??
?
v0
v1
.
vN
??
?
is a full matrix, so it becomes O(N2) computations, with is slow. There is a faster
way,
taking advantage of the structure of this matrix.
Theorem 18. The Chebyshev Differentiation matrix takes the form
D = (Dij)(N+1)(N+1)
D00 = 2N 2+1
6 DNN = - 2N62+1
D
jj = -
xj
2(1-x2j ) j = 1, ..., N - 1
D
ij =
Ci
Cj
(-1) i+j
xi-xj
i 6= j
where
Ci =
??? 21 iotherwise= 0 or N
There is an FFT for Chebyshev spectral methods, but first we need to decide how to
implement a boundary condition. Consider the boundary value problem
u
xx = e4x -1 < x < 1
u(1) = 0
54
Using Chebyshev interp olation, uxx ? D(2)v. Let P(x) be the unique polynomial of
degree
= N such that P(1) = 0, P(xj) = vj 1 = j = N - 1. Set wj = D(2)u = P00(xj)
1 = j = N - 1. Then we have
??
?
w0
w1
.
wN-1
wN
??
?
=
??
?
Dn(2)
??
?
??
?
v0
v1
.
vN-1
vN
??
?
We will remove the first and last row and column to deal with the boundary layer.
6.5.1 FFT for Chebyshev Sp ectral Metho ds
Define z ? C, z = ei?. Then x = cos(?) = <(z) = 12 (z + z-1) ? [-1, 1] The
Chebyshev
polynomial of degree n is
Tn
= <(zn) = 1
2
(zn + z-n) = cos(n?)
And so
Tn
(x) = cos(n arccos(x))
The roots of this polynomial (doesnt look like one but it is!) are at x =
cos( (j+n1/2)p, which
work out to be the Chebyshev points. For those who dont believe that crazy looking
function
is a polynomial of degree n, consider:
T0 = 1
T1 = x
T2 = 1
2 (z2 + z-2) = 12 (z + z-1) - 1 = 2x2 - 1
and in general
Tn
(x) = 2xTn-1(x) - Tn-2(x)
The proof of this is by induction, and is now left as a (fairly easy, I promise)
exercise. This
relation makes it clear that T
n(x) is a polynomial of degree n, and also shows that the leading
coefficient is 2n-1. Furthermore, the set of Chebyshev polynomials of degree = n
form a basis
for the set of polynomials of degree n. This means that for any polynomial p(x) of
degree n,
p(x) =
n Xj
=0
ajTj(x)
The Chebyshev points are xj = cos( jpN ) = cos(?j), where ?j = jpN make a uniform
grid! We
can then write p(x) as a function of ? on a uniform grid, and we have ourselves a
periodic
function. This means we can use spectral differentiation.
The procedure for the Chebyshev spectral method is
55
1. Given v0, ..., vN at the Chebyshev points x0 = 1, ..., xN = -1, extend the data
to a
vector v of length 2n by assigning v2N - j = vj, j = 1, ..., N - 1. Now you have an
even set of data, and this can be extended to 8 in a periodic way.
2. Use FFT to get the discrete Fourier transform of the data
vk = p
N
2N
Xj
=1
e-ik?jv
j k = -N + 1, ..., N
3. set wk = -ikvk except wN = 0.
4. Use IFFT
Wj
=
1
2p
N X
k=N+1
eik?jw
j
5. The derivative with respect to x corresponding to xj is
wj =
Wj
q1 - x2j j = 1, ..., N - 1
from the chain rule, because Wj is a discretization of a function of ?.
Of course, if the ?th derivative is required, the third step becomes wk = (-ik) ?
vk, and we
need to set wN = 0 if ? is odd. The ending relation from the chain rule is also
different, the
chain rule can be carried out to figure out what it should be.
56
Chapter 7
Monte Carlo Metho ds
Monte Carlo metho ds are probabilistic metho ds, named for the casino in Monaco.
Consider
Z0 1 f(x)dx 0 = f(x) = 1
We can randomly select a point (x1, y1) in [0, 1] [0, 1], and count it as 1 if
f(x1) = y1 or 0
otherwise and then repeat. So for random (xi, yi) ? [0, 1] [0, 1],
zi =
??? 0 if1 if yi => f(xi)
and then
z1 + z2 + + zn
n
? Z0 1 f(x)dx
as n ? 8. This works by the Law of Large Numbers, which states that if x1, x2, ...,
xn
are independent identically distributed random variables, and E(xi) = ,(where E(y)
is the
expected value of a random variable y then
x =
x1 + x2 + + xn
n
?
as n ? 8,
E(f(xi)) = Z0 1 f(x)dx
and
z1 + z2 + + zn
n
= E(f(xi))
because it is the chance the point is below the curve.
An important theorem in probabilistic methods is The Central Limit Theorem
Theorem 19. If var(xi) = s2 < 8,
x -
s/vn ? N(0, 1)
where N(0, 1) is the normal distribution.
57
This means that x - ~ O # vsn# , and so the convergence rate to our integral
above is
O # v1n# . This does not increase with increased dimension, nor does the
computational cost.
In Rd the approximation is identical:
Z[0,1] d f(x)dx = n1 Xi=1n f(xi) + O #v1n#
In contrast, quadrature rules in Rd become much more computationally costly:
Z[0,1] d f(~x)dx = c iX1N=1 iX2N=1 ... iXdN=1 f(xi 1, xi 2, ..., xid) + O #N12 #
So the computational cost is Nd. If n = Nd, so the computational cost was the same
for
each, the accuracy of quadrature is N12 , while for the Monte Carlo method it is
N1d/2 . For
d >> 1, the Monte Carlo method is more accurate at the same computational cost. For
very
high dimensions, it is not even feasible to use quadrature, and the Monte Carlo
method is
the only choice.
7.1 Probability Basics
Let X be a continuous random variable. Then F(x) = P(X = x) is a distribution
function.
F(x) is monotone, F(-8) = 0, F(8) = 1 and 0 = F(x) = 1. Furthermore, f(x) = F0(x)
is
called the probability density function. As a result, P(a = X = b) = F(b)-F(a) =
Rab f(x)dx,
and
E(X) = Z-88 xf(x)dx
so
E(h(X)) = Z-88 h(x)f(x)dx
The variance of x is
var(X) = E((x - ) 2) = E(x2) - (E(x)) 2 = 0
and the standard deviation is s = pvar(x).For a discrete random variable, F(x) is
discontinuous, and has jumps of pi at xi. The expected value for a discrete random
variable
is
E(h(x)) =
N Xi
=1
h(xi)pi
7.1.1 Distributions
There are some important probability distributions of continuous random variables.
The
Normal (Gaussian) Distribution is N(, s2) where
f(x) = v21pse-(x -)/2s2
58
The Exponential Distribution Exp(?) where
f(x) =
??? ?e0- ?x x >x < 0
and = s = ?1 .
The Uniform Distribution Unif[a, b], where
f(x) =
??? b-0 otherwise1 a x ? [a, b]
Every computer has a (pseudo-)random number generator that will produce X ? Unif[0,
1],
and when using a higher level language like python or matlab, random number
generators
are available with many other distributions.
Important distributions of discrete random variables include the Binomial
Distribution Binom(n, ?),
which is the coin flip distribution (or cheated coin if ? 6= 1/2).
P(heads) = ? P(tails) = 1 - ?
For n flips, if X is the number of heads
P(X = x) = nx# ?x(1 - ?) n-x
E(X) = n?
s2 = n?(1 - ?)
The Geometrical Distribution gives a distribution for X, the number of tosses
before the
first heads:
P(X = x) = (1 - ?) x-1?
and
E(X) = 1
? s2(X) = 1 ?-2 ?
The Multinormal Distribution is outcome of tossing a k sided dice n times. In this
distribution, the variable is X = (x1, ..., xk) where x1 + + xk = n and
P(X = (x1, ..., xk)) = n!
x1! xk! ?1x 1 ?kxk
where ?
j is the probability of seeing side j in a single toss.
The Poisson Distribution is
P(X = x) = ?x
x! e-x x = 0, 1, ...
and
E(X) = var(x) = ?
This is the limit of the Binom(n, p) where np = ? as n ? 8.
59
7.2 Convergence of Random Variables
If we have the random variables x1, x2, ..., xn and the distribution functions
F1(x), F2(x), ..., Fn(x),
we can define a few different kinds of convergence:
If lim
n?8
Fn
(x) = F(x) we say that xn ? x in distribution
If ?# > 0, lim
n?8
P(| xn - x| > #) = 0, then we say that xn ? x in probability.
If P( lim
n?8
| xn - x| = 0) = 1 then we say xn ? x almost surely.
So, convergence almost surely ? convergence in probability ? convergence in
distribution.
The weak law of large numbers is
Theorem 20. If x1, ..., xn are independent identically distributed (iid) with
finite mean ,
then
x1 + x2 + + xn
n
?
in probability.
The strong law of large numbers is
Theorem 21. If the variance of x1, ..., xn is also finite, then
x1 + x2 + + xn
n
?
almost surely.
It is also worth restating the central limit theorem
Theorem 22. if x1, ..., xn are mutually independent with mean and variance s2,
then
vn(x - )
s
? N(0, 1)
in distribution as n ? 8.
7.3 Random Sampling
That a computer can generate a uniform distribution on [0, 1] is taken as a given,
but we
probably want more distributions than just that. Suppose ? ? Unif[0, 1]. Then
f(x)dx = d?,
so F(x) = ?, and, if F is invertible, x = F-1(?). For example,
f(x) = e-x
F(x) = R0x f(t)dt = 1 - e-x
1 - e-x = ?
e-x = 1 - ?x = - log(1 - ?)
60
and b ecause we have a uniform distribution on [0, 1], the same is true for 1-?, so
x = - log(?).
We can use this technique to generate any distribution as long as F is invertible.
If F is not invertible, we need some other ideas. One idea, due to Von-Neumann, is
the
acceptance-rejection method. In this method, we find w(x) such that Mw(x) > f(x)?x,
M >
1, such that W(x) = A1 intx0w(t)dt is easily invertible. The algorithm for this
method is
1. Choose ?1 ? Unif[0, 1], find x = W-1(?1)
2. Choose ?2 ? Unif[0, 1]. If f(x) = Mw(x)?2, accept x. If not, reject x and return
to
step 1.
Efficiency of this method depends on how close w(x) is to f(x).
P(accept x) = P # Mwf(x()x) > ?# = E # Mwf(x()x) # = Z-88 Mwf(x()x) wA(x) dx = MA1
7.3.1 Discrete Sampling
Let k ? {1, .., M} be a random integer so that k = i with probability wi. The
sampling
algorithm is
1. Wk =
k Xi
=1
wi1cmW0 = 0
2. Take ? with Unif[0, 1]
3. Find k such that Wk-1 = ? = Wk
If the wi are not explicitly known, find w = wi, and use the rejection-acceptance
algorithm,
like so:
1. select integer with Unif[0, M], by selecting k = [M?] + 1, where ? ? Unif[0, 1]
2. if wk > w? 2 accept, else reject
7.4 Multivariate Random Variables
We can have multivariate random variables X~ = (X1, ..., Xd). For these,
F(X~) = P n\=1d {Xi = xi}!
f(X~) = ?F(X~)
?X1?X2 ?Xd
and
E(h(x)) = ZRd h(~x)f(~x)d~x
61
The multivariate normal distribution f = f(~x, ~, S) is
f = 1
(2p)dpdet(S) exp{- 12 (~x - ~)TS(~x - ~)}
where S is a positive definite matrix.
If x1, ..., xd are independent with the joint density function f(x1, ..., xd), and
xi has the
density function fi(xi), then
f(x1, ..., xd) = f1(x1)f2(x2) fd(xd)
The concept of Covariance is defined as
cov(X, Y) = E(XY) - E(X)E(Y)
and if X and Y are independent, cov(XY) = 0. Covariance is needed because
var(x1 + + xn) =
d Xi
=1
var(xi) + 2 X
i<j
cov(xi, xj)
And, if x1, ..., xn are independent identically distributed random variables, then
var #x1 + n + xn # = var(nxi)
In general, x1, x2, ..., xd are not mutually independent. We need a transformation
T : x ?
? = T(x) such that
P(x1, x2, ..., xd) = P1(?1)P2(?2) Pd(?d)d?1d?2 d?d