Reproducing kernel Hilbert space
It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion.
The reproducing kernel was first introduced in the 1907 work of Stanisław Zaremba concerning boundary value problems for harmonic and biharmonic functions.
James Mercer simultaneously examined functions which satisfy the reproducing property in the theory of integral equations.
The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of Gábor Szegő, Stefan Bergman, and Salomon Bochner.
The subject was eventually systematically developed in the early 1950s by Nachman Aronszajn and Stefan Bergman.
For ease of understanding, we provide the framework for real-valued Hilbert spaces.
While property (1) is the weakest condition that ensures both the existence of an inner product and the evaluation of every function in
A more intuitive definition of the RKHS can be obtained by observing that this property guarantees that the evaluation functional can be represented by taking the inner product of
This function is the so-called reproducing kernel[citation needed] for the Hilbert space
Nontrivial reproducing kernel Hilbert spaces often involve analytic functions, as we now illustrate by example.
Consequently, using Plancherel's theorem, we have Thus we obtain the reproducing property of the kernel.
The theorem first appeared in Aronszajn's Theory of Reproducing Kernels, although he attributes it to E. H. Moore.
via the integral operator using Mercer's theorem and obtain an additional view of the RKHS.
be a compact space equipped with a strictly positive finite Borel measure
Mercer's theorem states that the spectral decomposition of the integral operator
is a reproducing kernel so that the corresponding RKHS can be defined in terms of these eigenvalues and eigenfunctions.
The spectral theorem for self-adjoint operators implies that there is an at most countable decreasing sequence
The first sections presented the connection between bounded/continuous evaluation functions, positive definite functions, and integral operators and in this section we provide another representation of the RKHS in terms of feature maps.
Conversely, every positive definite function and corresponding reproducing kernel Hilbert space has infinitely many associated feature maps such that (3) holds.
Another classical example of a feature map relates to the previous section regarding integral operators by taking
Lastly, feature maps allow us to construct function spaces that reveal another perspective on the RKHS.
This view of the RKHS is related to the kernel trick in machine learning.
More formally, we define a vector-valued RKHS (vvRKHS) as a Hilbert space of functions
This definition can also be connected to integral operators, bounded evaluation functions, and feature maps as we saw for the scalar-valued RKHS.
We can equivalently define the vvRKHS as a vector-valued Hilbert space with a bounded evaluation functional and show that this implies the existence of a unique reproducing kernel by the Riesz Representation theorem.
Mercer's theorem can also be extended to address the vector-valued setting and we can therefore obtain a feature map view of the vvRKHS.
In particular, we find that every vvRKHS is isometrically isomorphic to a scalar-valued RKHS on a particular input space.
One can construct a ReLU-like nonlinear function using the theory of reproducing kernel Hilbert spaces.
Below, we derive this construction and show how it implies the representation power of neural networks with ReLU activations.
It has the inner product To construct the reproducing kernel it suffices to consider a dense subspace, so let
