A BRIEF REVIEW

ON THE REPRODUCING KERNEL HILBERT SPACE

AND THE DISCRETIZED TIKHONOV REGULARIZATION
O MA
.
NH KHANG - V ANH KHOA
Ho Chi Minh City - July 7, 2013
Contents
Contents 1
1 REPRODUCING KERNEL HILBERT SPACES 2
1.1 INTRODUCTION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 EXAMPLES OF REPRODUCING KERNELS AND HISTORICAL REMARKS. . . . . . 5
2 TIKHONOV REGULARIZATION AND THE REPRESENTER THEOREM 6
2.1 TIKHONOV REGULARIZATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Bibliography 7
1
Chapter 1
REPRODUCING KERNEL
HILBERT SPACES
1.1 INTRODUCTION.
Denition 1.1. Let H be a real Hilbert space of functions dened on R
n
with the inner product
(., .)
H
. A function : R is called a reproducing kernel for H if the two followings are satised:
1. For every x ,
x
(y) = (y, x) as a function of y belongs to H.
2. For every x and every f H, then f (x) = (f,
x
)
H
.
Remark. The second is called the reproducing property as well.
By applying the second one to the function
y
at f, we get

y
(x) = (
y
,
x
)
H
, for x, y .
And then using the rst one,
(x, y) = (
y
,
x
)
H
= (
x
,
y
)
H
, for x, y . (1.1.1)
This is called symmetric property of a reproducing kernel .
Denition 1.2. A Hilbert space H is called a reproducing kernel Hilbert space (RKHS) if there exists a
reproducing kernel of H. And its norm is symbolized by .
H
.
Moreover, for all f H in the form of f =
N

k=1

x
k
with x
k
,
k
R, its norm is dened by
f
2
H
=
N

k=1
N

j=1

j
(x
k
, x
j
) . (1.1.2)
The following theorem formally establishes the relationship between the RKHS and a reproducing
kernel.
Theorem 1.3 (Moore - Aronszajn). The reproducing kernel of a RKHS is uniquely determined. Con-
versely, a symmetric positive denite reproducing kernel denes a unique RKHS.
2
CHAPTER 1. REPRODUCING KERNEL HILBERT SPACES 3
Proof. To prove the rst statement, we assume that there exists
x
and
x
are simultaneously
determined on a RKHS H.
At this point, using the second of (1.1), we have
(f,
x
)
H
= (f,
x
)
H
, for f H.
It means that
(f,
x

x
)
H
= 0, for f H.
Choosing f =
x

x
, we obtain
x

H
= 0. Hence, the reproducing kernel is unique.
For the second statement, given we can construct the RKHS H
0
as the completion of space of
functions spanned by the set {
x
|x } with an inner product dened as follows:
Given two functions f and g in {
x
|x }, we set
f (x) =
N

k=1

x
k
(x) , (1.1.3)
g (x) =
N

k=1

k
(x) , (1.1.4)
we dene their inner product to be
(f, g)
H
0
=
N

k=1
N

j=1

j
_

x
k
,
x

j
_
H
0
=
N

k=1
N

j=1

_
x
k
, x

j
_
. (1.1.5)
Thus for all x , we have
(f,
x
)
H
0
=
N

k=1

k
(
x
k
,
x
)
H
0
=
N

k=1

k
(x
k
, x) = f (x) . (1.1.6)
This means that H
0
has the reproducing property. So the denition of the inner product above does
not depend on the representations of the functions f and g in H
0
. Since is positive denite, (f, f)
H
0
0
for all f H
0
and we obtain the Cauchy - Schwartz for (f, f)
H
0
. In addition, if (f, f)
H
0
= 0 and then
by (1.1.6), for all x ,
|f (x)| f
H
0

H
0
= 0,
which implies that f 0. Thus,
_
H
0
, (., .)
H
0
_
is a pre - Hilbert space.
(continue)
According to Mercers theorem that can be found in [1], we obtain the following theorem.
Theorem 1.4. Let be in L
2
() with a compact metric space , we dene
T

() (x) =

(x, y) (y) dy. (1.1.7)

Then there is an orthonormal basis {e
j
} of L
2
() consisting of eigenfuntions of T

such that the

CHAPTER 1. REPRODUCING KERNEL HILBERT SPACES 4
eigenvalues
j
are non - negative. The eigenfunctions corresponding to non - zero eigenvalues are con-
tinuous on and can be represented as
(x, y) =

j=1

j
e
j
(x) e
j
(y) , (1.1.8)
and the convergence of the sum is uniform on , and absolute for each pair (x, y) .
Proof. Firstly, we show that the operator T

is well - dened, i.e., T

() is a continuous function
for all L
2
(). Indeed, for all x, y , we have
|T

() (x) T

() (t)| =

((x, y) (t, y)) (y) dy

= |(
x

t
, )
L
2
|

x

L
2

L
2

((x, y) (t, y))

2
dy
_
1/2

L
2

max
y
|(x, y) (t, y)|
L
2
.
Since is a continuous function on a compact domain and is in L
2
(), we obtain T

() is a
continuous function on .
Secondly, ...
Theorem 1.5. Let be a compact metric space and H
1
dened by
H
1
=
_
_
_
f L
2
() :

j=1

1
j

(f, e
j
)
L
2
()

2
<
_
_
_
, (1.1.9)
and the inner product can be written as
(f, g)
H1
=

j=1

1
j
(f, e
j
)
L
2
()
(g, e
j
)
L
2
()
, f, g H
1
. (1.1.10)
Then H
1
= H (they are the same spaces of functions with the same inner product).
Proof. Firstly, we show that (1.1.10) denes an inner product. Indeed, we easily realize that it is
linear, symmetric and (f, f)
H
1
0 since
j
are non - negative. Moreover, (f, f)
H
1
= 0 if and only if
f 0.
For the second step, by (1.1.8), we have
x
=

j=1

j
e
j
(x) e
j
. Thus we see that

i=1

1
i

(
x
, e
i
)
L
2
()

2
=

i=1

1
i

j=1

j
e
j
(x) (e
j
, e
i
)
L
2
()

2
=

i=1

i
|e
i
(x)|
2
< ,
which implies that
x
H
1
for all x .
Furthermore, since f L
2
(), we probably have the form f =

i=1
a
i
e
i
with a
i
R. Then, by the
denition of its inner product,
CHAPTER 1. REPRODUCING KERNEL HILBERT SPACES 5
(f,
x
)
H
1
=

j=1

1
j
_

i=1
a
i
e
i
, e
j
_
L
2
()
_

k=1

k
e
k
(x) e
k
, e
j
_
L
2
()
=

j=1

1
j
a
j
e
j

j
e
j
(x)
= f (x) .
Thus, H
1
is a Hilbert space of functions with a reproducing kernel , so it must be equal to the
original H by the uniqueness of RKHS.
Conclusion. The outcome of the above theorem is that although space H
1
is dened using the integral
operator T

and its associated eigenfunctions {e

j
} which depend on some prerequisite conditions, it still
coincides exactly with the original RKHS H of by the uniqueness of RKHS.
1.2 EXAMPLES OF REPRODUCING KERNELS AND HIS-
TORICAL REMARKS.
RKHS were explicitly introduced in learning theory by F. Girosi (1997). However, this idea was used
for the rst time at the beginning of the 20th century by S. Zaremba in his work on boundary value
problems for harmonic and biharmonic functions. But he did not develop this theory after that.
For a long time these results were not investigated and seemed to be forgotten. Then the idea once
again appeared in some papers and works of S. Bergman, S. Bochner, E. H. Moore, N. Aronszajn,
S. Saitoh and so on in many subjects, for instance, approximation theory, computer vision, partial
dierential equations, probability theory.
For now, they are used to deal with many important problems, and can be considered as a nearly
new tool in Mathematics.
In statistical problems, people frequently use the three following kernels.
Linear kernel
(x, y) = x y,
Gaussian kernel
(x, y) = e

xy
2

2
, > 0,
Polynomial kernel
(x, y) = (x y + 1)
n
, n N.
Chapter 2
TIKHONOV REGULARIZATION
AND THE REPRESENTER
THEOREM
2.1 TIKHONOV REGULARIZATION.
The goal of regularization is to restore the well - posedness (specically, making the result depend
smoothly on the data) of the empirical risk minimization technique by eectively restricting the hypoth-
esis space H.
Let E be an arbitrary set, and let H

be the RKHS admitting a reproducing kernel (., .) on E.

For any Hilbert space H we consider a bounded linear opterator K : H

H. We shall study the best

approximate problem
inf
fH

Kf b
H
, (2.1.1)
for a vector b in H.
6
Bibliography
[1] Holger Wendland, Scattered Data Approximation, Cambridge University Press, 2005.
[2] Y. C. Hon, Tomoya Takeuchi, Discretized Tikhonov regularization by reproducing kernel Hilbert space
for backward heat conduction problem, Adv Comput Math (2011) 34 : 167 - 183.
7