Compressed Sensing

Compressible/k-sparse signals
Stable measurements
Restricted isometric property
Signal reconstruction algorithms
Geometric interpretation
Constrained/Unconstrained optimization
Fast algorithms

IP, Jos Bioucas Dias, IST, 2007 1

K-sparse signals

Definition of k-sparse signals:

is k-sparse if only k coefficients are nonzero

IP, Jos Bioucas Dias, IST, 2007 2

Compressible signals

There exists a basis where the representation

has just a few large coefficients and many small

coefficients.

Compressible signals are well approximated by k-sparse

representations

Many natural and man-made signals are compressible

IP, Jos Bioucas Dias, IST, 2007 3

Example: wavelet representation

4
10

2
10

0
10

-2
10
0 1 2 3 4 5 6
4
x 10

IP, Jos Bioucas Dias, IST, 2007 4

Example: wavelet representation (cont.)

IP, Jos Bioucas Dias, IST, 2007 5

Classical transform coding of compressive signals
1. Acquire the full n-sample signal
2. Compute the set of n coefficients
3. Retain the k << n largest coefficients and encode
their locations and values

Shortcommings:

1. Acquire n samples and only k << n coefficients are retained

2. The encoder computes n coefficients
3. The locations, besides the values of the largest k coefficients, have to be
encoded

IP, Jos Bioucas Dias, IST, 2007 6

Goal of compressive sampling

Measurement matrix

Goal of CS

Design a measurement matrix and a reconstruction algorithm for k-sparse

and compressible signals such that is of the order of

IP, Jos Bioucas Dias, IST, 2007 7

Stable measurement matrix

The measurement processes must not erase the information present in the k
non-zero entries of signal

Restricted isometric property (RIP)

For any k-sparse vector there holds

for some > 0

Any submatrix of is well-conditioned

IP, Jos Bioucas Dias, IST, 2007 8

Another way to look at stable measurements

The measurement matrix must be incoherent with the sparsifying basis

in the sense that the vectors cannot sparsely represent the vectors and
vice versa.

Stable (in the RIP sense) measurement matrices

random Gaussian

random binary

randomly selected Fourier samples (extra log factors apply)

IP, Jos Bioucas Dias, IST, 2007 9

Signal reconstruction algorithms

Minimum -norm reconstruction

NP- complete problem !!!!

(Convex approximation) Replace with

Basis pursuit problem ( complexity)

Surprise: if the measurement matrix is Gaussian i.i.d, then

a k-sparse vector f is exactly recovered by solving the
optimization problem with observations

IP, Jos Bioucas Dias, IST, 2007 10

Geometrical interpretation

k-sparse vectors

IP, Jos Bioucas Dias, IST, 2007 11

Unconstrained optimization

Minimum -norm reconstruction

Equivalent

Basis denoising algorithm

Algorithms
l1_ls [Kim et al, 2007]
GPRS [Figueiredo et al, 2007]
TwIST [Bioucas-Dias & Figueiredo , 2007]

IP, Jos Bioucas Dias, IST, 2007 12

Example: Sparse reconstruction

is i.i.d. Gaussian

20
40
60
80
100
120
100 200 300 400 500 600 700 800 900 1000

Original data - f Observed data - g

1 0.15

0.8

0.6 0.1

0.4

0.2 0.05

-0.2 0

-0.4

-0.6 -0.05

-0.8

-1 -0.1
0 200 400 600 800 1000 1200 0 20 40 60 80 100 120 140

IP, Jos Bioucas Dias, IST, 2007 13

 Example: Sparse reconstruction

Pseudo-inverse regularization
1 1
f f
0.8 0.8
fpseudo-inv f

0.6 0.6

0.4 0.4

0.2 0.2

0 0

-0.2 -0.2

-0.4 -0.4

-0.6 -0.6

-0.8 -0.8

-1 -1
0 200 400 600 800 1000 1200 0 200 400 600 800 1000 1200

IP, Jos Bioucas Dias, IST, 2007 14

 Example: Total variation reconstruction

randomly selected Fourier samples (9%)

IP, Jos Bioucas Dias, IST, 2007 15

IP, Jos Bioucas Dias, IST, 2007 16
References
E. Cands, J. Romberg, and T. Tao, Robust uncertainty principles: Exact signal
Reconstruction from highly incomplete frequency information, IEEE Trans. Inform.
Theory, vol. 52, no. 2, pp. 489509, Feb. 2006.

D. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, vol. 52, no.
4, pp. 1289 1306, April 2006.

R. Baraniuk, Compressed sensing, IEEE Signal Processing Magazine, 2007

J. Bioucas-Dias, M. Figueiredo, A new TwIST: two-step iterative

shrinkage/thresholding algorithms for image restoration, IEEE Transactions on
Image Processing, December 2007.

M. Figueiredo, R. Nowak, S. Wright, Gradient projection for sparse

reconstruction: application to compressed sensing and other inverse problems,
IEEE Journal of Selected Topics in Signal Processing: Special Issue on Convex
Optimization Methods for Signal Processing, vol. 1, no. 4, pp. 586-598, 2007.

S Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, A method for

large-scale l1-regularized least squares, IEEE Journal on Selected Topics
in Signal Processing, vol. 4, no. 1, pp. 606-617, 2007.

IP, Jos Bioucas Dias, IST, 2007 17