MULTIDIMENSIONAL SIGNAL,

IMAGE, AND VIDEO PROCESSING

AND CODING

JOHN W. WOODS
Rensselaer Polytechnic Institute
Troy, New York

»iBllfllfiii
. . i.
AMSTERDAM • BOSTON • HEIDELBERG • LONDON
NEW YORK • OXFORD • PARIS • SAN DIEGO
SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO
ELSEVIER Academic Press is an imprint of Elsevier
 CONTENTS

v
Preface xiii
Acknowledgments xvii

1 T W O - D I M E N S I O N A L SiGNALS A N D SYSTEMS 1
1.1 Two-Dimensional Signals 2
1.1.1 Separable Signals 6
1.1.2 Periodic signals 7
1.1.3 2-D Discrete-Space Systems 9
1.1.4 Two-Dimensional Convolution 11
1.1.5 Stability of 2-D Systems 13
1.2 2-D Discrete-Space Fourier Transform 14
1.2.1 Inverse 2-D Fourier Transform 18
1.2.2 Fourier Transform of 2-D or Spatial Convolution 19
1.2.3 Symmetry Properties of Fourier Transform 26
1.2.4 Continuous-Space Fourier Transform 28
1.3 Conclusions 31
1.4 Problems 31
References 33

2 SAMPLING IN TWO DIMENSIONS 35

2.1 Sampling Theorem—Rectangular Case 36
2.1.1 Reconstruction Formula 40
2.1.2 Ideal Rectangular Sampling 43
2.2 Sampling Theorem—General Regulär Case 48
2.2.1 Hexagonal Reconstruction Formula 52
2.3 Change of Sample Rate 5 7
2.3.1 Downsampling by Integers Mi x M2 57
2.3.2 Ideal Decimation 58
2.3.3 Upsampling by Integers L\ x Lj 61
2.3.4 Ideal Interpolation 62
2.4 Sample-Rate Change—General Case 64
2.4.1 General Downsampling 64
VI CONTENTS

2.5 Conclusions 66
1.6 Problems 66
References 70

3 T w O - D i M E N S I O N A L SYSTEMS A N D
Z-TRANSFORMS 71
3.1 Linear Spatial or 2-D Systems 72
3.2 Z-Transforms 16
3.3 Regions of Convergence 79
3.3.1 More General Case 82
3.4 Some Z-Transform Properties 83
3.4.1 Linear Mapping of Variables 84
3.4.2 Inverse Z-Transform 85
3.5 2-D Filter Stability 89
3.5.1 First-Quadrant Support 91
3.5.2 Second-Quadrant Support 91
3.5.3 Root Maps 96
3.5.4 Stability Criteria for NSHP Support Filters 98
3.6 Conclusions 100
3.7 Problems 101
References 103

4 T w O - D i M E N S I O N A L D i S C R E T E TRANSFORMS 105
4.1 Discrete Fourier Series 106
4.1.1 Properties of the DFS Transform 109
4.1.2 Periodic Convolution 111
4.1.3 Shifting or Delay Property 112
4.2 Discrete Fourier Transform 113
4.2.1 DFT Properties 115
4.2.2 Relation of DFT to Fourier Transform 120
4.2.3 Effect of Sampling in Frequency 121
4.2.4 Interpolating the DFT 122
4.3 2-D Discrete Cosine Transform 123
4.3.1 Review of 1-D DCT 125
4.3.2 Some 1-D DCT Properties 128
4.3.3 Symmetrie Extension in 2-D DCT 131
4.4 Subband/Wavelet Transform (SWT) 132
4.4.1 Ideal Filter Case 132
4.4.2 1-D SWT with Finite-Order Filter 135
4.4.3 2-D SWT with FIR Filters 137
4.4.4 Relation of SWT to DCT 138
4.4.5 Relation of SWT to Wavelets 138
 CONTENTS vi/

4.5 Fast Transform Algorithms 140

4.5.1 Fast DFT Algorithm 140
4.5.2 Fast DCT Methods 141
4.6 Sectioned Convolution Methods 142
4.7 Conclusions 143
4.8 Problems 144
References 147

5 T W O - D I M E N S I O N A L FiLTER D E S I G N 149
5.1 FIR Filter Design 150
5.1.1 FIR Window Function Design 150
5.1.2 Design by Transformation of 1-D Filter 156
5.1.3 Projection-Onto-Convex-Sets Method 161
5.2 HR Filter Design 165
5.2.1 2-D Recursive Filter Design 165
5.2.2 Fully Recursive Filter Design 171
5.3 Subband/Wavelet Filter Design 174
5.3.1 Wavelet (Biorthogonal) Filter Design Method 178
5.4 Conclusions 182
5.5 Problems 182
References 187

6 INTRODUCTORY IMAGE PROCESSING 189

6.1 Light and Luminance 190
6.2 Still Image Visual Properties 194
6.2.1 Weber's Law 195
6.2.2 Contrast Sensitivity Function 196
6.2.3 Local Contrast Adaptation 198
6.3 Time-Variant Human Visual System Properties 199
6.4 Image Sensors 201
6.4.1 Electronic 201
6.4.2 Film 203
6.5 Image and Video Display 204
6.5.1 Gamma 205
6.6 Simple Image Processing Filters 206
6.6.1 Box Filter 206
6.6.2 Gaussian Filter 207
6.6.3 Prewitt Operator 208
6.6.4 Sobel Operator 208
6.6.5 Laplacian Filter 209
vüi CONTENTS

6.7 Conclusions 2 1 1
6.8 Problems 2 1 1
References 213

7 IMAGE ESTIMATION AND RESTORATION 215

7.1 2-D R a n d o m Fields 216
7.1.1 Filtering a 2-D Random Field 218
7.1.2 Autoregressive Random Signal Models 222
7.2 Estimation for R a n d o m Fields 2 2 4
7.2.1 Infinite Observation Domain 225
7.3 2-D Recursive Estimation 2 2 9
7.3.1 1-D Kaiman Filter 229
7.3.2 2-D Kaiman Filtering 233
7.3.3 Reduced Update Kaiman Filter 235
7.3.4 Approximate RUKF 236
7.3.5 Steady-State RUKF 236
7.3.6 LSI Estimation and Restoration Examples with RUKF 237
7.4 Inhomogeneous Gaussian Estimation 2 4 1
7.4.1 Inhomogeneous Estimation with RUKF 243
7.5 Estimation in the Subband/Wavelet D o m a i n 2 4 4
7.6 Bayesian and M A P Estimation 2 4 8
7.6.1 Gauss Markov Image Models 249
7.6.2 Simulated Annealing 253
7.7 Image Identification and Restoration 2 5 7
7.7.1 Expectation-Maximization Algorithm Approach 258
7.7.2 EM Method in the Subband/Wavelet Domain 262
7.8 Color Image Processing 2 6 3
7.9 Conclusions 2 6 3
7.10 Problems 2 6 3
References 266

8 DIGITAL IMAGE COMPRESSION 269

8.1 Introduction 2 7 0
8.2 Transformation 272
8.2.1 DCT 272
8.2.2 SWT 274
8.2.3 DPCM 275
8.3 Quantization 2 7 6
8.3.1 Uniform Quantization 278
8.3.2 Optimal MSE Quantization 278
 CONTENTS ix

8.3.3 Vector Quantization 280

8.3.4 LBG Algorithm [7] 282
8.4 Entropy Coding 2 8 4
8.4.1 Huffman Coding 285
8.4.2 Arithmetic Coding 286
8.4.3 ECSQandECVQ 287
8.5 D C T Coder 2 8 9
8.6 SWT Coder 2 9 2
8.6.1 Multiresolution SWT Coding 298
8.6.2 Nondyadic SWT Decompositions 300
8.6.3 Fully Embedded SWT Coders 300
8.6.4 Embedded Zero-Tree Wavelet (EZW) Coder 301
8.6.5 Set Partitioning in Hierarchical Trees (SPIHT) Coder 304
8.6.6 Embedded Zero Block Coder (EZBC) 306
8.7 JPEG 2 0 0 0 308
8.8 Color Image Coding 3 0 9
8.8.1 Scalable Coder Results Comparison 311
8.9 Robustness Considerations 3 1 1
8.10 Conclusions 3 1 2
8.11 Problems 3 1 2
References 315

9 THREE-DIMENSIONAL AND SPATIOTEMPORAL

PROCESSING 317
9.1 3-D Signals and Systems 318
9.1.1 Properties of 3-D Fourier Transform 320
9.1.2 3-D Filters 321
9.2 3-D Sampling and Reconstruction 3 2 1
9.2.1 General 3-D Sampling 323
9.3 Spatiotemporal Signal Processing 3 2 5
9.3.1 Spatiotemporal Sampling 325
9.3.2 Spatiotemporal Filters 326
9.3.3 Intraframe Filtering 328
9.3.4 Intraframe Wiener Filter 328
9.3.5 Interframe Filtering 330
9.3.6 Interframe Wiener Filter 331
9.4 Spatiotemporal M a r k o v Models 3 3 2
9.4.1 Causal and Semicausal 3-D Field Sequences 333
9.4.2 Reduced Update Spatiotemporal Kaiman Filter 335
9.5 Conclusions 338
9.6 Problems 338
References 339
x CONTENTS

10 DIGITAL VIDEO PROCESSING 341

10.1 Interframe Processing 342
10.2 M o t i o n Estimation and M o t i o n Compensation 348
10.2.1 Block Matching Method 350
10.2.2 Hierarchical Block Matching 353
10.2.3 Overlapped Block Motion Compensation 354
10.2.4 Pel-Recursive Motion Estimation 355
10.2.5 Optical flow methods 356
10.3 M o t i o n - C o m p e n s a t e d Filtering 358
10.3.1 MC-Wiener Filter 358
10.3.2 MC-Kalman Filter 360
10.3.3 Frame-Rate Conversion 363
10.3.4 Deinterlacing 365
10.4 Bayesian M e t h o d for Estimating M o t i o n 3 7 1
10.4.1 Joint Motion Estimation and Segmentation 373
10.5 Conclusions 3 7 7
10.6 Problems 378
References 379
10.7 Appendix: Digital Video Formats 380
SIF 381
CIF 381
ITU 601 Digital TV (aka SMPTE D l and D5) 381
ATSC Formats 382

11 DIGITAL VIDEO COMPRESSION 385

11.1 Intraframe Coding 3 8 7
11.1.1 M-JPEG Pseudo Algorithm 388
11.1.2 DV Codec 391
11.1.3 Intraframe SWT Coding 392
11.1.4 M-JPEG 2000 394
11.2 Interframe Coding 395
11.2.1 Generalizing 1-D DPCM to Interframe Coding 396
11.2.2 MC Spatiotemporal Prediction 397
11.3 Interframe Coding Standards 398
11.3.1 MPEG 1 399
11.3.2 MPEG 2—"a Generic Standard" 401
11.3.3 The Missing MPEG 3—High-Definition Television 403
11.3.4 MPEG 4—Natural and Synthetic Combined 403
11.3.5 Video Processing of MPEG-Coded Bitstreams 404
11.3.6 H.263 Coder for Visual Conferencing 405
11.3.7 H.264/AVC 405
11.3.8 Video Coder Mode Control 408
11.3.9 Network Adaptation 410
 CONTENTS XI

11.4 Interframe SWT Coders 4 1 0

11.4.1 Motion-Compensated SWT Hybrid Coding 412
11.4.2 3-D or Spatiotemporal Transform Coding 413
11.5 Scalable Video Coders 4 1 7
11.5.1 M o r e o n M C T F 420
11.5.2 Detection of Covered Pixels 421
11.5.3 Bidirectional MCTF 423
11.6 Obj ect-Based Video Coding 4 2 6
11.7 Comments on the Sensitivity of Compressed Video 4 2 8
11.8 Conclusions 4 2 9
11.9 Problems 4 3 0
References 431

12 VIDEO TRANSMISSION OVER NETWORKS 435

12.1 Video o n IP N e t w o r k s 4 3 6
12.1.1 Overview of IP Networks 437
12.1.2 Error-Resilient Coding 440
12.1.3 Transport-Level Error Control 442
12.1.4 Wireless Networks 443
12.1.5 Joint Source-Channel Coding 444
12.1.6 Error Concealment 446
12.2 Robust SWT Video Coding (Bajic) 4 4 7
12.2.1 Dispersive Packetization 447
12.2.2 Multiple Description FEC 453
12.3 Error-Resilience Features of H.264/AVC 4 5 8
12.3.1 Syntax 458
12.3.2 Data Partitioning 459
12.3.3 Slice Interleaving and Flexible Macroblock Ordering 459
12.3.4 Switching Frames 459
12.3.5 Reference Frame Selection 461
12.3.6 Intrablock Refreshing 461
12.3.7 Error Concealment in H.264/AVC 461
12.4 Joint S o u r c e - N e t w o r k Coding 4 6 3
12.4.1 Digital Item Adaptation (DIA) in MPEG 21 463
12.4.2 Fine-Grain Adaptive FEC 464
12.5 Conclusions 4 6 9
12.6 Problems 4 6 9
References 471

Index 477