Blind Deconvolution Algorithms for the Restoration of atmospherically

degraded Imagery: a comparative Analysis

Claudia S. Huebner*, Mario Greco
Dept. of Signatorics, FGAN-FOM, Gutleuthausstrasse 1, 76275 Ettlingen, Germany;
ABSTRACT
Suggestions from the field of image processing to compensate for turbulence effects and restore degraded images include
motion-compensated image integration after which the image can be considered as a non-distorted image that has been
blurred with a point spread function (PSF) the same size as the pixel motions due to the turbulence. Since this PSF is
unknown, a blind deconvolution is still necessary to restore the image. By utilising different blind deconvolution
algorithms along with the motion-compensated image integration, several variants of this turbulence compensation
method are created. In this paper we discuss the differences of the various blind deconvolution algorithms employed and
give a qualitative analysis of the turbulence compensation variants by comparing their respective restoration results. This
is done by visual inspection as well as by means of different image quality metrics that analyse the high frequency
components.
Keywords: Image restoration, atmospheric turbulence, blind deconvolution, image quality metrics

1. I
TRODUCTIO

In imaging applications the prevalent effects of atmospheric turbulence comprise image dancing as well as image
blurring. These image-degradation effects arise from random inhomogeneities in the temperature distribution of the
atmosphere, producing small but significant fluctuations in the index of refraction which are the most pronounced close
to the ground. Light waves propagating through the atmosphere will sustain cumulative phase distortions as they pass
through these turbulence-induced fluctuations. When imaging over horizontal paths, as opposed to vertical imaging, the
degree of image degradation is particularly severe. As a consequence, image resolution is generally limited by
atmospheric turbulence rather than by design and quality of the optical system being used.
A number of correction methods have been proposed over the years, prominent among them the hardware-based
Adaptive Optics (AO) systems. Their underlying principle is to measure the phase aberration of an incoming wavefront
and correct it, directly. Usually, only the higher Zernike modes are corrected, like tip/tilt and defocus, by use of a
deformable mirror. Although they have proven invaluable for imaging point sources in a variety of applications, e. g. in
astronomical or medical imaging, their effectiveness is somewhat limited where extended targets are concerned. Efforts
have been made to remedy some of that drawback by using a hybrid approach, i. e. by using soft-ware based methods in
addition to the hardware-based correction, e. g. in [1], either directly "in-the-loop" or in post-processing. Nevertheless,
the work involved in the build-up of such a system is comparatively complex [2], and the necessary equipment still quite
expensive, not to mention bulky and immobile. On this account, the relative simplicity of a software-based approach
which can be put into effect anytime and anywhere, at low cost, holds great appeal for many applications, such as ours.
Our intention is, ultimately, the implementation of a mobile turbulence-compensation system, which is able to correct
atmospheric image degradation effects in real-time on a live-stream. Ideally, this correction should be put into effect
without necessitating the operator to undergo a protracted initialization procedure while trying to find the optimal setting
for a multitude of parameters. The work presented in this paper could be considered as a step forward to that end.
1.1 Outline
Suggestions from the field of Image Processing to compensate for turbulence effects and restore such degraded images
include Motion-Compensated Averaging (MCA), based on an idea of [3], after which the image can be considered as a
non-distorted image that has been blurred with a PSF (Point Spread Function) the same size as the pixel motions due to
*huebner@fom.fgan.de; phone 49 7243 992-252; fax 49 7243 992-299; www.fom.fgan.de

Optics in Atmospheric Propagation and Adaptive Systems XI, edited by

Anton Kohnle, Karin Stein, John D. Gonglewski, Proc. of SPIE Vol. 7108
71080M 2008 SPIE CCC code: 0277-786X/08/$18 doi: 10.1117/12.800124
Proc. of SPIE Vol. 7108 71080M-1
2008 SPIE Digital Library -- Subscriber Archive Copy

the turbulence. Since this PSF is unknown, a (mostly) blind deconvolution is still necessary to restore the image. We are
utilising four different algorithms in combination with this motion-compensated image integration, namely the linear
Inverse Wiener Filter (IWF) method, the non-linear Lucy-Richardson Deconvolution (LRD) along with its Iterative
Blind Deconvolution (IBD) variant which both use maximum likelihood estimation and, last but not least, a
deconvolution technique based on Principal Component Analysis (PCA). The specifics of our implementation of the
motion compensated image integration are given in Section 2 along with some notions on the concept of image quality.
In Section 3 the basic principle behind the blind deconvolution algorithms we employed is briefly described and their
respective advantages and disadvantages are discussed. Our efforts to automate deconvolution parameter estimation
wherever possible, at least partially, are elaborated on in Section 4 and in Section 5 the respective restoration results of
the turbulence compensation variants are presented and evaluated.

(a)

iii

(b)

!CUI

(c)

Fig. 1. Downsized sample images from the sequence pairs with different test patterns; for (a)-(c) left: 1.0 ms exposure time,
right: <0.1 ms exposure time; (a) Weak turbulence: Cn~1e-15, (b) medium turbulence: Cn~1e-14, (c) strong
turbulence: Cn~1e-13

1.2 Data and Equipment Specifics

Instead of an image stream, we used image sequences that had been recorded in different turbulence conditions. As input
we selected a number of six video sequences from among more than a hundred that had been recorded by FGAN-FOM
during a field trial in 2005 on the White Sands Missile Range, New Mexico, USA [4], for sample images see Fig. 1. All
of the sequences contained a thousand frames each and had an image resolution of 240 256 pixels. Sequences were
chosen in pairs for the following reasons: for one thing, they were showing the same object, namely a panel with a test
pattern placed in a distance of 1 km from the optics, and for another, they had no more than half an hour between their
respective recording time, such that we can assume similar turbulence (and lighting) conditions, and most importantly,
because of their differing exposure time, which for one sequence was always 1.0 ms and for the other less than 0.1 ms.
Table 1. Compilation of all vital data for the input sequences, ordered by time of their recording
Rec. Time

0715

0720

0745

0748

0812

0822

1307

Exp. Time

1/11000

1e-3

1e-3

1/11000

1e-3

1/22000

1e-3

F-Number

5.6

16

22

5.6

22

5.6

16

1000

1000

1000

1000

1000

1000

1000

Num. Frames
Resolution

240 x 256 240 x 256 240 x 256 240 x 256 240 x 256 240 x 256 240 x 256

Path Length

1000

1000

1000

1000

1000

Cn

4e-15

2.36e-15

5.09e-14

4.85e-14

9.3e-14

Est. Sigma

1.0

1.0

1.5

1.5

3.0

1000

1000

1.23e-13 7.56e-13
4.5

9.5

Est. Edge Width

Est. Noise

0.0036

0.0076

0.0054

0.0052

0.0437

0.0088

0.0011

The propagation path was in all cases parallel to the ground at a height of about 1.8 m. Measurements of Cn, the
structure parameter of the refractive index fluctuations, were taken at the same height and made available for each of the
sequences. All data were recorded by a Photron Fastcam PCI-R2 camera at a resolution of 240 x 256 pixels with a Texas
Instruments TC237B CCD sensor with progressive scan and a Nikon Nikkor ED 800 optics with 800 mm fixed focal
length. Aperture scales used for the sequences were f/5.6 in the extremely short-exposure cases (<0.1 ms) and either f/16
or f/22 for the remaining sequences. All data for the selected sequences has been compiled in Table 1, including data for
an additional sequence which had been recorded during the midday heat and was selected for its severe degree of
turbulence degradation. "Est. Sigma", Est. Edge Width" and "Est. Noise" refer to estimated parameters and are discussed
in Section 4.

Proc. of SPIE Vol. 7108 71080M-2

1.3 Approach
It should be noted that during the development of the algorithm presented in this paper, we did not adhere too rigidly to
the "real-time" objective in view of the continuing increase in computational power available and with an additional code
optimisation in mind which will undoubtedly reduce computation time yet a bit further. Nevertheless, we tried to keep
computation time per frame within reasonable limits, especially considering that we are intending to process a stream of
images where the output rate should at least be a 30 fps (frames per second) even if the input rate may comprise up to a
1000 fps. Naturally, such a constraint will come at the expense of at least some of the quality of the results.
Consequently, we set out to find the best solution for our application in terms of deciding which deconvolution method
yields the best results for a given choice of parameters within reasonable time.

2. MOTIO
-COMPE
SATED AVERAGI
G
Motion Compensated Averaging (MCA) is in essence the same as normal image integration, the main difference being,
that before integrating the next frame of the input sequence it is shifted slightly within a given search space (see Fig. 2)
of a few pixels in every direction such that the input frame better matches the running average. A sliding window of
length W is used for calculating this running average or, alternatively, a temporal median as was proposed by [5]. The
window length ought to be increased with growing turbulence strength. The maximum window length depends, of
course, on available memory space. Keeping 100 frames with a 240 256 resolution in the memory posed no problem,
as this was the setting we used, but keeping 1000 will (at present) become problematic on most PCs.
Search space

In
M(x)
n (x) = In(x+M(x))
Sn+1
Sn
Sn+1 = a Sn + (1 a) In
Sn+1 = b [a Sn + (1 a) n ] + (1 b) In

Input
Shift
MC Input
Current output
Previous output
Normal Average

M()
by

MC-Average

Fig. 2. Illustration of the MCA algorithm

Increasing the search space, automatically also increases the computational complexity by (2*
+1) where
denotes the
maximum pixel shift. Accordingly, the shift operation results in a number of (2*
+1) shifted images from which the
one, that matches the previous output image the best, needs to be determined. There are a number of different approaches
to making that decision by employing quality metrics which will be addressed in the following section. To keep
calculation time down, the size of the search space was limited to no more than two or three pixels. Nevertheless,
calculation time per frame still is in the order of 0.1 sec for a search space of 2 pixels and about 0.15 sec for a search
space of 3 pixels and therefore much too high to be real-time capable. But given that hardware based implementations of
image shifting and stacking operations are comparatively easy to realise, e. g. by using an FPGA (Field Programmable
Gate Array), the somewhat extended calculation time of our software implementation shall be of no consequence here.
The idea of using MCA at all, apart from noise reduction which would obviously also result from normal averaging, is
the reduction of image dancing between one frame and the next. Unfortunately, in highly anisoplanatic conditions, such
as we are wont to face, the coherence length r0 of the turbulence is much smaller than our aperture, so that the logic
doesnt quite apply. Nevertheless, when compared to the results from normal averaging, the stabilization effect for the
results from the MCA is generally better.
2.1 IQM Image Quality Metrics
The problem of giving a qualitative measure about the degree of similarity between two given images is closely related
to the concept of image quality and its many interpretations. As such, there is no simple answer to the question about
image quality. For once, it strongly depends on the application at hand. In image or video compression, for instance, it is

Proc. of SPIE Vol. 7108 71080M-3

comparatively easy to decide on an images quality. Here, a higher compression rate is equivalent to lower image quality
and so-called full reference Image Quality Metrics (IQM) are employed to compare the perfect, i.e. uncompressed,
reference image to its compressed instance. As our task is to decide upon the closest match between two images, such an
IQM is the logical choice. But again calculation time is an obvious issue and for this reason simple error summation
methods as the Mean Square Error (MSE), Root Mean Square Error (RMSE) or Mean Absolute Error (MAE) are very
attractive. In [6] another universal IQM was suggested that makes use of the statistical values of two images x and y
measuring structural distortion rather than the error energy. It is defined by:

IQM1 =

x y 2x y
2 xy
2
(x ) + (y ) x y x2 y2
2

(2.1)

while x and y denote the respective mean values, x and y are the standard deviations and xy is the cross-covariance.
The basic idea seemed promising, but so far we could detect no discernible improvement in our application over simply
using the Absolute Error (AE), therefore we settled once more for the fastest solution.
However, given that we are also interested in evaluating the respective results of our restoration variants, there is another
class of measures we need to consider, the so-called no reference IQMs where no reference image is required. In the case
of evaluating restoration result such an ideal image obviously doesnt exist. Considering, that our aim is to improve the
resolution of image data, good image quality will consequently refer to high contrast of fine detail. Since the low-pass
filter effect of the atmosphere removes the high frequency components, it is a logical conclusion to consider a metric that
exploits this information. Some such metric was proposed by [7] and has been employed by us before in the context of a
different project concerning a synthetic imaging technique. This metric is the mathematical equivalent of an existing
physical system and is defined as:

IQM 2 =

FT{exp[i I(r )]} | 4 d 2 r

(2.2)

where FT{} denotes the Fourier transform, I(r) the intensity values of the image I at position r = [r1, r2].

3. BLI
D DECO
VOLUTIO

Essentially, a deconvolution describes the procedure of separating two convolved signals f and h. In the spatial domain
the blind deconvolution problem takes the general form:

g ( x, y ) = h ( x , y ) f ( x , y ) + n ( x, y )

(3.1)
where g denotes the blurred image, h the blurring or point spread function (PSF), f the true image, * the convolution
operator and n an additive noise component. To simplify further steps, it is common practice to transfer the problem into
the Fourier domain where the relatively complicated convolution-operation becomes a simple multiplication:

G (u, v ) = H (u, v ) F(u, v ) + N (u, v )

(3.2)
with G, H, F and N denoting the Fourier transforms of g, h, f and n, respectively. Many attempts at solving this
deceptively simple equation for a wide variety of applications can be found throughout literature. An overview of the
most popular of these blind image deconvolution algorithms is detailed in [8].
3.1 IWF Inverse Wiener Filtering
The simplest approach to a solution of Eq. (3.2) is to use direct inverse filtering, i. e. to form an estimate F of the
undegraded image:

G (u , v )
F(u , v ) =
H(u , v )

N(u , v )
F(u , v ) = F(u , v ) +
H(u , v )

(3.3)

It becomes obvious that blind deconvolution is, as such, an ill-posed problem. Even if the exact blurring function was
known, due to the random noise component, the true image can never be fully recovered. In addition, the filter function
H is likely to contain numerous zeros or at least near vanishing values, such that the quotient on the right hand side of
Eq. (3.3) will produce significant errors in the restoration estimate, even if noise
as such was negligible. For increasing
spatial frequencies the ratio will also increase and thus disproportionately enhance the relative effect of the noise.

Proc. of SPIE Vol. 7108 71080M-4

Deconvolution with an inverse Wiener filter uses the least mean square error between the estimate and the true image:

{(

err 2 = E f f

)}
2

(3.4)

where E{} denotes the expectation operator. The Fourier transform of the estimate F can now be expressed as [9]:
2

H(u , v )
F(u, v ) =
(3.5)
G (u , v )
2
2
2
H(u , v ) H(u , v ) + N(u , v ) F(u , v )
If the noise-to-signal power ratio |
|/|F| becomes zero, meaning the absence of additive noise, Eq. (3.5) will reduce to
the ideal inverse filter in Eq. (3.3).
3.2 LRD Lucy-Richardson Deconvolution
The Lucy-Richardson (LRD) algorithm was developed independently by [10] and [11] and is a nonlinear and basically
non-blind method, meaning the PSF, or at least a very good estimate, must be a priori known as is the case for the IWF.
The LRD has been derived from Bayesian probability theory where image data are considered to be random quantities
that are assumed to have a certain likelihood of being produced from a family of other possible random quantities. The
problem regarding the likelihood that the estimated true image, after convolution with the PSF, is in fact identical with
the blurred input image, except for noise, is formulated as a so-called likelihood function, which is iteratively maximized.
The solution of this maximization requires the convergence of [9]:

g ( x, y )
fk + 1 ( x, y ) = fk ( x, y ) h ( x, y )

h( x, y ) f ( x, y )

(3.6)

where k denotes the k-th iteration. It is the division by f that constitutes the algorithm's nonlinear nature. The image
estimate is assumed to contain Poisson distributed noise which is appropriate for photon noise in the data whereas
additive Gaussian noise, typical for sensor read-out, is ignored. In order to reduce noise amplification, which is a general
problem of maximum likelihood methods, it is common practice to introduce a dampening threshold below which further
iterations are (locally) suppressed. Otherwise high iteration numbers introduce artefacts to originally smooth image
regions.
3.3 IBD Iterative Blind Deconvolution
The iterative blind deconvolution (IBD) algorithm, proposed by [12], is mainly a blind version of the LRD algorithm
where the PSF needs not to be known, only its support. The IBD is a so-called Expectation Maximization (EM)
algorithm which is an optimization strategy for estimating random quantities corrupted by noise. In the case of blind
deconvolution this means that the likelihood function from the LRD algorithm is again maximized iteratively but with
specified constraints until an estimate for the blurring PSF is retrieved from the data along with the estimate for the true
image. The IBD algorithm is characterized by a computational complexity in the order of O(
log2
) per iteration where

is the total number of pixels in a single frame while normally more than one iteration is required for its convergence.
3.4 PCA Principal Component Analysis
Due to the iterative nature of most blind deconvolution algorithms, calculation time per frame is relatively high, thus
rendering any real-time applications virtually impossible. But like the Inverse Wiener Filter, the deconvolution algorithm
based on Principal Component Analysis (PCA), which has been proposed in literature only recently [13], is an explicit,
i.e. non-iterative, algorithm and executes very fast. The PCA algorithm aims at de-correlating the correlation between
image matrix columns that has been caused by atmospheric (and system) blurring. There exist two versions of it, a
"multiple-observation" version that operates on a sequence of images, yielding a single output image, and functions as a
truly blind deconvolution algorithm and another "single-observation" version that operates on single images and needs
additional information about the filter support size.
If a number of M blurred observations of the same scene are given they can be modelled as random vectors

{Y1 , Y2 ,..., YM } with mean value Y . The PCA based algorithm looks for a linear transformation or filter that maximizes
the variance of its output, i. e. its estimation for the true image, by exploiting the M blurred input images as follows:

X est = [Y1 Y , Y2 Y ,..., YM Y ] v + Y

Proc. of SPIE Vol. 7108 71080M-5

(3.7)

where X est is a random vector representing the estimated true image obtained by this linear combination. The solution s
to the maximization problem is a vector of length M and can be written as:

s = arg max (X est Y )

v

(X est

Y )

(3.8)

with E{} denoting the expectation operator as before and v the variance of the argument. This solution allows us to
extract the maximum spatial variance within the M-blurred input images, thus recovering some of the high spatial
frequency components that have been removed previously by the blur.
The most obvious advantage of the PCA-based algorithm, apart from being robust to white noise, is that according to
[13] its computational complexity is generally lower than that of the iterative deconvolution techniques discussed in this
context.
3.5 Discussion
The four blind deconvolution algorithms discussed in this section were chosen for a number of reasons. The IWF, for
instance, is a classic method which can be calculated directly and therefore executes very fast, i. e. within 0.05 and 0.09
seconds for a filter size of 7 7 pixels (which was used for all calculation time measurements). The results generally
improve if the process is repeated at least 2-3 times but with the real-time application in mind that theoretically only
allows for a maximum of 0.03 sec for the complete processing of a single frame, one calculation must suffice.
The IBD was mostly chosen for its general reliability and as a kind of benchmark with which to compare the respective
results of the other methods. Since it usually takes several iterations to converge, execution time for the 5 to 10 iterations
necessary to get a decent result is relatively long with 0.5 sec up to 1.1 sec. The main reason to include the non-blind
LRD as well was as a means to evaluating the effectiveness of our estimate for the point spread function in comparison
to the results yielded by passing only filter support size to the IBD algorithm. Calculation time per frame for the LRD
was in the range of 0.36 sec and 0.45 sec for a number of 5 and 10 iterations, respectively, whereas 20 iterations already
took 0.97 sec.
The algorithm based on Principal Component Analysis was chosen because it was claimed to be very fast and the
qualitative analysis in [14] which was done on synthetic data, had given rise to hope that the algorithm would achieve
similarly good results when applied to real data. Calculation time in the single-observation version proved to be about
0.18 sec per frame regardless of the number of principal components to be determined. It took the multiple-observation
version 0.45 sec for 100 frames to calculate the first component, containing the highest variance, 0.19 sec for 50 frames
and 0.03 sec for 10 frames. Calculation of all four principal components took 0.62 sec for 100 frames, and for 50 and 10
frames calculation times were almost identical. Thus far, the algorithm was not quite as fast as we had hoped for large
filter support sizes of 7 pixels and more.
It should be noted that all computations were done using Matlab such that the given calculation times are relative and
might very well be sped up by an efficient C/C++-Implementation.

4. PARAMETER ESTIMATIO

We tried automating the parameter estimation wherever possible by reducing the input parameters for external conditions
to turbulence strength, characterized by Cn, the structure parameter of the refractive index fluctuations, and approximate
path length. The input parameters for the optical system, like frame-rate and image resolution, ought to be possibly be
communicated directly from the sensor without requiring manual input as of now they are simply put down in a script
and read in.
4.1 Automatic
oise Estimation
For the automatic noise estimation we decided upon using the temporal variance of a pixel region with homogeneous
grey values. In order to find such a homogeneous region, we first determined the image entropy of a single frame. Since
we were interested in local information content we performed the calculations in blocks of 16 16 pixels. We then
picked the block with the lowest entropy and calculated the temporal variance for the corresponding pixels over an
interval of a 100 frames (in pre-processing) and chose the mean value of this variance as noise parameter. The resulting
values for the individual sequences have been listed in Table 1. The intrinsic snag in this method is, of course, that the
variance caused by light which is randomly scattered due to the turbulence fluctuations, becomes an inseparable part of

Proc. of SPIE Vol. 7108 71080M-6

the result. It is also interesting to note, if perhaps not all that surprising, that the values we derived this way differed
significantly from the one, namely 1.7519, we derived from a sequence we recorded recently using the same camera
directed at a sheet of white paper, with the same image resolution and exposure time but a different optics.
4.2 Estimation of the Point Spread Function
While the Optical Transfer Function (OTF) describes the response of an optical system to a dirac impulse in the
frequency domain, its inverse Fourier transform, the so-called Point Spread Function (PSF), characterises the degree to
which the optical system spreads, or rather, blurs the light from a point source in the spatial domain. The approach to
image restoration involves the previous determination of the atmospheric point spread function. The simplest
approximation to an atmospheric blurring function is a Gaussian filter. This is quite a reasonable assumption since the
optical turbulence basically acts as a low-pass filter by filtering out the high spatial frequencies and thereby blurring
sharp edges and point-like objects.
The main problem is to guess the correct filter size in terms of mean and standard deviation : Choosing too big will
introduce ringing effects and choosing it too small will retain too much blurring. For each of our sequences, statistics
regarding the turbulence conditions had been compiled and evaluated with special focus on edge width broadening [15],
so we were able to use the edge width information that was correlated with the Cn values that had been measured for our
respective sequences. The resulting estimates are listed in Table 1.

illililt
Seq.1, median 100

IBD median

LRD median

IWF median

"It" "It"

Seq.2, median 100

IBD median

LRD median

iiUiiLi
IWF median

I IILIMI41
Seq.1, PCA single C 1234 for 100

PCA multi C 1234 for 100

I 11111R4
Seq.2, PCA single C 1234 for 100

PCA multi C 1234 for 100

Fig. 3: Some characteristic deconvolution results from Seq. 1 & 2 (7.15 a.m. & 7.20 a.m.), beginning with median of 100
frames as reference, followed by the resp. IBD, LRD and IWF results (Seq. 1 top, Seq. 2 centre row) and concluding
with single and multiple version PCA results for components 1, 2, 3 and 4 (bottom row: Seq. 1 left, Seq. 2 right)

Proc. of SPIE Vol. 7108 71080M-7

5. RESULTS
For easier reference, the sequences will be numbered by time of their recording, meaning Seq. 1 will refer to the earliest
recording from 7.15 a.m., Seq. 2 to the next, etc. And due to lack of space all the images had to be reduced to 60% of
their original size. For Seq. 1-6 the respective average and median images of 100 frames look almost the same as the
sample frame, merely a little less noisy, so samples and averages were omitted and only the median image, which was
also input for the various deconvolution algorithms, is displayed. Results are presented for sequence pairs, so the image
labels have been coloured to group results by sequence. Fig. 3, 4 and 5 present a basic set of results for sequences 1 & 2
(7.15 a.m. & 7.20 a.m., low turbulence), 3 & 4 (7.45 a.m. & 7.48 a.m., medium turbulence) and 5 & 6 (8.12 a.m. & 8.22
a.m., medium to strong turbulence), respectively. This set includes the temporal median of 100 frames (the same 100
frames in every case) as reference to compare to restoration results, the respective IBD, LRD and IWF results, followed
by PCA single and multiple (only Seq. 1 & 2) version results for various principal components, where "C 14", for
instance, means that components 1 and 4 have been evaluated by exploiting the generalisation made in [14].

AIVS

AIVS
I

Seq.3, median 100

IBD median

LRD median

IWF median

Seq.4, median 100

IBD median

LRD median

IWF median

PCA single C 1234 for 100

Seq.4, PCA single C 23 for 100

PCA single C 1234 for 100

4
AIMS
I

Seq.3, PCA single C 14 for 100

Fig. 4: Some characteristic deconvolution results from Seq. 3 & 4 (7.45 a.m. & 7.48 a.m.), beginning with median of 100
frames as reference, followed by the resp. IBD, LRD and IWF results (Seq. 3 top, Seq. 4 centre row) and concluding
with various single version PCA results (bottom row: Seq. 3 left, Seq. 4 right)

Due to lack of space only one comprehensive set of results could be included. Exemplarily, the results for Seq. 7, i. e. the
"worst case scenario", were chosen, since they were the most significant. They are presented in Fig. 6, which spreads
over two pages, beginning with a sample, median as well as average of 100 frames and the total average of the sequence
as frame of reference. To illustrate the turbulence effect on the edges, the standard deviation of the total sequence is also
included. Subsequently, the respective IBD, LRD and IWF results for both, average and median of 100 frames, are
displayed; following next are results from the single observation PCA deconvolution when applied to the average of
100, median of 20 and 10 frames and finally results from the multi observation PCA applied to 10, 20 and 100 frames.

Proc. of SPIE Vol. 7108 71080M-8

_.ii
ill.

-1r
III.
Seq.5, median 100

IBD median

LRD median

IWF median

ill ill
Seq.6, median 100

IBD median

LRD median

IWF median

PCA single C 1234 for 100

Seq.6, PCA single C 2 for 100

PCA single C 1234 for 100

HI!.
Seq.5, PCA single C 1 for 100

Fig. 5: Some characteristic deconvolution results from Seq. 5 & 6 (8.12 a.m. & 8.22 a.m.), beginning with median of 100
frames as reference, followed by the resp. IBD, LRD and IWF results (Seq. 5 top, Seq. 6 centre row) and concluding
with various single version PCA results (bottom row: Seq. 5 left, Seq. 6 right)

5.1 Evaluation
5.1.1 Short exposure vs. long exposure
When imaging through turbulence, exposure time becomes an issue. For longer exposures, i. e. if exposure time outlasts
the duration of the turbulence, turbulence-cells of all sizes are averaged over. It is the largest cells which are
predominantly responsible for the image dancing as they move across the aperture of the optics and the resulting
"smoke" effect which arises from light that gets deflected by these larger cells. For short exposures, on the other hand,
these large cells are "frozen" and only the small ones, which are responsible for the blurring, remain. Essentially, that
means that the geometry of a scene will be preserved, i. e. straight lines will remain straight, but details will be blurred.
Consequently, the results from the sequences with exposure below 0.1 ms were generally more accurate but also blurrier
than those with 1 ms exposure time which in the case of sequences 2, 3, 5 and 7 definitely qualified for the long exposure
case. The long exposure results looked sharper, had higher contrast and contained more details.
5.1.2 Visual inspection
The deblurring efforts of IBD, LRD and IWF are barely distinguishable and offer comparatively little improvement over
the median for sequences 1, 2, 5 and 6, whereas in Seq.3 and 4 it becomes more apparent. Here, the IWF yields the
highest resolution of all, judging from its ability to resolve the first line of text "A BEAR". In that regard it even
outperforms the single observation PCA algorithm, in quality as well as in speed. Otherwise, the IBD yields the sharpest
result for Seq. 3 and the LRD the smoothest without being blurry, unlike the PCA result.

Proc. of SPIE Vol. 7108 71080M-9

sample image

average 100

IBD average

IBD median

standard deviation 1000

Mt
IWF average

median 100

average 1000

LRD average

LRD median

IWF median

PCA single C1234 for average 100

PCA single C1 for average 100

PCA single C2 for average 100

PCA single C3 for average 100

PCA single C1 for median 20

PCA single C1234 for median 20

PCA single C1 for median 10

Proc. of SPIE Vol. 7108 71080M-10

Mtr
PCA single C4 for average 100

PCA single C1234 for median 10

PCA multi C1 for 10

PCA multi C1234 for 10

PCA multi C1 for 20

PCA multi C1234 for 20

PCA multi C1 for 100

PCA multi C4 for 100

PCA multi C34 for 100

PCA multi C1234 for 100

Fig. 6 (spreads 2 pages): Comprehensive deconvolution results from Seq. 7 (1.07 p.m.), recorded in strong turbulence
conditions, beginning with sample and averages as reference, with standard deviation of total sequence illustrating
turbulence effect on edges, followed by the respective IBD, LRD and IWF results, each for average and median of 100
frames; next are results from PCA single version applied to average of 100, median of 20 and 10 frames and last are
results from PCA multi version applied to 10, 20 and 100 frames

For all other sequences, including Seq. 7, IBD and LRD look essentially the same. This, in conjunction with the passable
IWF results, means that at least our PSF estimates are quite satisfactory. As regards the noise parameter, we could
discern neither improvement nor worsening in the results compared to results created under the zero noise assumption.
The PCA single-observation algorithm has proven its ability to resolve vertical and horizontal lines. Even in Seq. 7 it
manages to distinctly resolve the vertical lines. The first principal component, corresponding to the greatest variance,
yields the best result. In Fig. 6 (previous page) all four components are depicted individually in order to illustrate their
respective characteristics and the false 3D-effect in particular, which is inherent to the first two components and
obviously increases along with filter support which again increases with growing turbulence strength. (It should be
mentioned that the black borders around the results of the single observation PCA version are caused by the algorithm
and depend on the size of the corresponding filter support.)
As to the multi-observation PCA version, for 100 input frames it fails completely, and increasingly so in accordance with
growing turbulence strength. Concerning its performance for only 10 or 20 input frames, the results for Seq. 7 indicate
that it does not meet either the standard established by the single version nor that of the other deconvolution algorithms,
or even of the simple average or median operation.
5.1.3 Image Quality Metrics
For lack of a better alternative we used the averages and median and also the input frame as reference for metric IQM1 in
Eq. (2.1), meaning the results were correlated to the deconvolution input (i. e. average or median).
As was to be expected, the values for IBD and LRD were almost identical for both metrics; those for the IWF from
metric IQM2 Eq. (2.2) were generally a bit better but ranged in the same order as the other two. Due to the (at times
extreme) contrast enhancement of the PCA algorithms, both versions received high quality marks from both metrics, not
always justified.
On the whole, the results were rather consistent with our initial assessment but especially the "high quality" of the
multiple observation PCA results for 100 frames signifies that deconvolution results are prone to noise amplification
resulting in unnaturally high frequency components and high contrast, thus rendering both metrics somewhat unreliable.

Proc. of SPIE Vol. 7108 71080M-11

6. CO
CLUSIO
S
We have described and discussed the various steps in our algorithm for the restoration of images that were degraded by
atmospheric turbulence. This includes a comparative analysis of four different deconvolution methods and a presentation
of the respective results for a number of real data image sequences, both long and short exposure. The partially
automated PSF estimation has produced satisfactory results. The PCA approach has yielded interesting results and
proven worthy of being pursued further. The general feasibility of the real-time objective for our project has been
confirmed even if not quite as of yet.
Future work includes replacing the simple Gaussian filter by a more accurate estimate for the average atmospheric PSF
and finding an analytical description for the correspondence between Cn and edge width. We also intend to apply MCA
locally, by using block matching, in order to improve compensation for image dancing in anisoplanatic conditions.
Furthermore, the use of thresholded edge images rather than greyscale images might help improve the reliability of the
quality metrics.
It should also be mentioned that the full stabilizing effect of the restoring algorithm(s) can best be demonstrated when
directly comparing the original to the restored images in a video sequence.

ACK
OWLEDGME
TS
The authors would like to thank their colleagues, Gabriele Marchi, Endre Repasi and Wolfgang Schuberth, for supplying
last minute measurements, helpful insights and an abundance of literature on atmospheric transfer functions.

REFERE
CES
[1]

[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]

[15]

Vladimir I. Polejaev, Pierre R. Barbier, Gary W. Carhart, Mark L. Plett, David W. Rush, Mikhail A. Vorontsov,
Adaptive compensation of dynamic wavefront aberrations based on blind optimization technique, Proc. of SPIE,
Vol. 3760, pp. 88-95, (1999).
G. Marchi, R. Wei, "Evaluation and progress in the development of an adaptive optics system for ground object
observation", Proc. of SPIE, Vol. 6747, (2007)
E. Mauer, "Investigation of atmospheric turbulence effects on extended objects in high-speed image sequences
applying automatic image analysis", Proc. of SPIE, Vol. 5237, pp. 39-48, (2004)
E. Repasi, "Image Catalogue of Video Sequences recorded by FGAN-FOM during the RTG-40 Field Trial",
Distributed to group members, (2006)
J. Gilles, "Restoration algorithms and system performance evaluation for active imagers", Proc. of SPIE, Vol. 6739,
6739B, (2007)
Z. Wang, A. C. Bovik, "A universal image quality index", IEEE Signal Processing Letters, vol. 9, no. 3, pp. 8184,
(2002).
M. A. Vorontsov, G. W. Carhart, D. V. Pruidze, J. C. Ricklin, D. G. Voelz, "Image quality criteria for an adaptive
imaging system based on statistical analysis of the speckle field", J. Soc. Am. A, Vol. 13, No. 7, (1996)
D. Kundur, D. Hatzinakos, "Blind Image Deconvolution", IEE Signal Processing Magazine, 1053-5888/96, pp. 4364, (1996)
R. C. Gonzalez, R. E. Woods, "Digital Image Processing", 2nd ed., Prentice Hall, NJ, (2002)
W. H. Richardson, "Bayesian-Based Iterative Method of Image Restoration", J. Opt. Soc. Am. 62 (1), pp. 55-60,
(1972)
L. Lucy, "An iterative technique for the rectification of observed distributions", Astron. J. 79, pp. 745, (1974)
Ayers G. R., and Dainty J. C., Iterative blind deconvolution method and its applications, Opt. Letters, vol. 13, no.
7, pp. 547549, (1988).
D. Li, M. Mersereau, S. Simke, "Atmospheric Turbulence-Degraded Image Restoration Using Principal Component
Analysis", IEEE Geoscience and Remote Sensing Letters, Vol. 4, No. 3, pp. 340-344, (2007)
M. Greco, C. S. Huebner, G. Marchi, "Quantitative performance evaluation of a blurring restoration algorithm based
on principal component analysis", to be published on Proc. SPIE of Optics in Atmospheric Propagation and
Adaptive Systems XI, (2008)
E. Repasi, R. Weiss, "Analysis of Image Distortions by Atmospheric Turbulence and Computer Simulation of
Turbulence Effects", Proc. of SPIE, Vol. 6941,69410S, (2008)

Proc. of SPIE Vol. 7108 71080M-12