Covariance of two random

variables
• Height and wake-up time are uncorrelated, but height and weight
are correlated.
• Covariance
Cov(X, Y) = 0 for X = height, Y = wake-up times
Cov(X, Y) > 0 for X = height, Y = weight
• Definition:

Cov( X , Y ) % C XY % E !( X $ # x )(Y $ # y ) "

Question: If Cov(X, Y) < 0 for two random variables X, Y , what

would a scatterplot of samples from X, Y look like?

Question: If we add arbitrary constants to the random variables X, Y ,

how does the covariance change?

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 13

 Estimating covariance from samples
Again, we assume that we do not know the + x1 x 2 ! xn (
underlying probability distributions. But consider )y
we sample n times and estimate: * 1 y2 ! yn &'

1 n
Cov ( X , Y ) - , ( xi $ mx )( yi $ m y ) “sample covariance”
n i -1

Questions:
What is Cov(X, X) ? Cov(X, X) = Var(X)
How are Cov(X, Y) and Cov(Y, X) related? Cov(X, Y) = Cov(Y, X)

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 14

 Estimating covariance in Matlab
• Sample mean:
x - [ x1 x2 x3 ! xn ] mx . m_x
my . m_y
y - [ y1 y2 y3 ! yn ]
• Covariance + y1 $ m y (
)y $ m &
1
Cov( X , Y ) - [ x1 $ mx x2 $ mx ! xn $ mx ] )
2 y&

n ) " &
) &
)* yn $ m y &'
Method 1: >> v = (1/n)*(x-m_x)*(y-m_y)’
Method 2: >> w = x-m_x
>> z = y-m_y
>> v = (1/n)*w*z’

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 15

 Coefficient of Correlation
The coefficient of correlation is defined as:

E !( X $ # x )(Y $ # y ) " Cov ( X , Y )

0 xy % -
/ x/ y / x/ y

Properties:
• -1 ! Cxy ! 1
• if Cxy = 0 : X and Y uncorrelated
• if Cxy bigger/smaller zero : X and Y are positively/negatively
correlated
• Advantage: we can multiply X and Y with arbitrary factors and
Cxy stays the same.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 16

 Correlation of two random
variables
Definition:
C orr ( X , Y ) - E! XY "
If X and Y have zero mean, this is the same as the covariance.
If in addition, X and Y have variance of one this is the same as
the coefficient of correlation.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 17

Correlation, Covariance, Corr.Coeff.

C orr ( X , Y ) - E! XY " Correlation

Cov( X , Y ) - E !( X $ # x )(Y $ # y ) " Covariance

E !( X $ # x )(Y $ # y ) " Cov ( X , Y )

C xy - - Coefficient of Corr.
/ x/ y / x/ y

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 18

 Covariance Matrix
Covariance: Cov( A, B ) - E !( A $ # A )( B $ # B ) "

Now consider random vector: X - ( x1 , x2 , ! , xn )T

We can compute covariance
between two components, say !
Cov( x2 , x5 ) - c25 - E ( x2 $ # x2 )( x5 $ # x5 ) "
between x2 and x5:

Doing this for all combinations

6 c11 c21 ! cn1 3
gives us the elements of the 4 1
covariance matrix: 4 c12 c22 1
7X % 4 " # " 1
4 1
4 1
4c ! cnn 12
5 1n

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 19

 Properties of Covariance Matrix
6 c11 c21 ! cn1 3
4 1
4 c12 c22 1
7X % 4 "
4
# " 1
1
!
Cov( xi , x j ) - cij - E ( xi $ # xi )( x j $ # x j ) "
4 1
4c ! cnn 12
5 1n

It is symmetric, because cij - c ji .

Its diagonal elements are the individual variances: cii - / i - var( xi ) .
2

If it is diagonal, the xi are all 6 / 12 0 ! 0 3

uncorrelated and we have: 4 1
4 0 / 2
1 2
7 X % 44 " # " 1
1
4 1
44 1
5 0 ! / n 2 12

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 20

 Example covariance matrix

people’s heights: time people woke up this

X ~ N(67, 20) morning: Y ~ N(9, 1)

Question: what is the covariance matrix of V = (X Y)T ?

X and Y should be uncorrelated:

+X ( +20 0(
V -) & ) 0 1&
*Y ' * '

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 21

Estimating the covariance matrix from
samples (including Matlab code)
Sample n times and find mean of samples:
v1 v2 ! vn
+ x1 x2 ! x n ( + mx (
V -) m-) &
* y1 y2 ! yn &' *m y '

Find the covariance matrix:

+ x1 $ mx y1 $ m y (
1 + x1 $ mx x2 $ m x ! xn $ mx ( )) x2 $ mx y2 $ m y &&
Cov (V ) - )y $ m
n * 1 y y2 $ m y ! yn $ m y &' ) " " &
) &
)* xn $ mx yn $ m y &'

>> m = (1/n)*sum(v,2)
>> z = v - repmat(m,1,n)
>> v = (1/n)*z*z’

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 22

Gaussian distribution in D dimensions
Of course, the most important distribution can also be extended to higher
dimensions. Recall that a 1-dimensional Gaussian is completely
determined by its mean, #, and its variance, /82:
( x$# )2
1 $
X ~ N!#: /82) p( x) -
2" /
e 2/ 2

A D-dimensional Gaussian (multivariate Gaussian) is completely

determined by its mean, #, and its covariance matrix, 79
1 $ 12 ( x $µ ) T 7 $1 ( x $µ )
X ~ N!#: 7) p ( x) -
(2") D/2
7
1/ 2 e
Question: what happens when D = 1 for the D-dimensional Gaussian?

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 23

 The Gaussian in D dimensions
Question: What does a set of equiprobable points look like for a
2-dim. Gaussian? What for a D-dim. Gaussian?

In 2D, it’s an ellipse. In D dimensions, it’s an ellipsoid.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 24

 Equiprobable contours
of a Gaussian
If a Gaussian random vector has covariance matrix that is diagonal
(all of the variables are uncorrelated, then the axes of the ellipsoid
are parallel to the coordinate axes.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 25

 Equiprobable contours
of a Gaussian
If a Gaussian random vector has covariance matrix that is not
diagonal (some of the variables are correlated), then the axes of the
ellipsoid are perpendicular to each other, but are not parallel to the
coordinate axes.

Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch 26