Reproducing Kernel Hilbert Space

• A Reproducing Kernel Hilbert Space (RKHS) is an infinite-dimensional vector space of functions that includes a continuous point evaluation for each point in a given domain, defined by a unique reproducing kernel.
• RKHSs are often used in machine learning for tasks such as kernel regression and SVM, as they enable efficient representation of complex models using kernels that describe the geometry of the functions and their interpolations.
• Positive semi-definite kernels play a crucial role in reproducing kernel methods, as they allow for the construction of RKHS norms and help establish norms for function spaces adapted to differential and stochastic structures.

Reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions on a set $X$ in which every point-evaluation functional $f\mapsto f(x)$ is continuous. By the Riesz representation theorem, for each $x\in X$ there is a unique kernel section $K(\cdot,x)$ such that

$f(x)=\langle f,K(\cdot,x)\rangle_H.$

The function $K:X\times X\to\mathbb C$, called the reproducing kernel, encodes simultaneously the Hilbert-space inner product, the embedding of $H$ into the function space on $X$, and the geometry of evaluations. RKHSs are in one-to-one correspondence with positive-semidefinite kernels through the Moore–Aronszajn construction. They provide canonical feature representations, minimum-norm interpolation, kernel-based regularization, covariance and Gaussian-process models, operator-valued observations, geometric embeddings, and function spaces adapted to differential, integral, stochastic, and representation-theoretic structures.

1. Definition, reproducing property, and kernel correspondence

Let $X$ be a set and let $H\subseteq\mathbb C^X$ be a Hilbert space. For each $f\mapsto f(x)$0, define the evaluation functional

$f\mapsto f(x)$1

The space $f\mapsto f(x)$2 is an RKHS if every $f\mapsto f(x)$3 is continuous. Equivalently, for every $f\mapsto f(x)$4, there is a unique $f\mapsto f(x)$5 satisfying

$f\mapsto f(x)$6

The element $f\mapsto f(x)$7 is the Riesz representer of evaluation at $f\mapsto f(x)$8, and the reproducing kernel is

$f\mapsto f(x)$9

with the order of variables determined by the convention for linearity of the inner product.

The reproducing identity yields the evaluation estimate

$x\in X$0

Thus norm convergence in $x\in X$1 implies pointwise convergence. This distinguishes RKHSs from general Hilbert spaces of functions, such as many $x\in X$2-spaces, in which point evaluation need not be continuous (Manton et al., 2014).

The kernel is Hermitian and positive semidefinite. For every finite collection $x\in X$3 and scalars $x\in X$4,

$x\in X$5

Indeed,

$x\in X$6

Conversely, every positive-semidefinite kernel determines a unique RKHS. One begins with

$x\in X$7

defines

$x\in X$8

and completes the resulting pre-Hilbert space. This is the Moore–Aronszajn construction. The kernel sections form a canonical, generally overcomplete coordinate system: $x\in X$9

A positive-semidefinite kernel also admits a feature representation. There are a Hilbert space $K(\cdot,x)$0 and a map $K(\cdot,x)$1 such that

$K(\cdot,x)$2

The associated RKHS consists of functions

$K(\cdot,x)$3

with the quotient norm induced by the feature representation. In the canonical realization, $K(\cdot,x)$4.

2. Feature geometry, distances, and regularity

The canonical feature map induces the kernel pseudometric

$K(\cdot,x)$5

whose squared form is

$K(\cdot,x)$6

For a real symmetric kernel,

$K(\cdot,x)$7

It is a genuine metric when the feature map is injective; otherwise distinct points with identical kernel sections are identified.

Every $K(\cdot,x)$8 is automatically Lipschitz with respect to this intrinsic metric: $K(\cdot,x)$9 This property does not require an externally specified topology on $f(x)=\langle f,K(\cdot,x)\rangle_H.$0. It is a direct consequence of the Hilbert-space geometry of kernel sections (Fiedler, 2023).

The intrinsic metric differs from regularity measured in a pre-existing metric $f(x)=\langle f,K(\cdot,x)\rangle_H.$1 on $f(x)=\langle f,K(\cdot,x)\rangle_H.$2. If kernel sections are locally $f(x)=\langle f,K(\cdot,x)\rangle_H.$3-Hölder with respect to $f(x)=\langle f,K(\cdot,x)\rangle_H.$4, then, generally,

$f(x)=\langle f,K(\cdot,x)\rangle_H.$5

and every RKHS function is $f(x)=\langle f,K(\cdot,x)\rangle_H.$6-Hölder with a bound proportional to its RKHS norm. In particular, ordinary Lipschitz continuity of kernel sections generally yields only $f(x)=\langle f,K(\cdot,x)\rangle_H.$7-Hölder continuity of all RKHS functions. A feature map that is itself $f(x)=\langle f,K(\cdot,x)\rangle_H.$8-Hölder in Hilbert-space norm avoids this exponent-halving phenomenon and gives

$f(x)=\langle f,K(\cdot,x)\rangle_H.$9

If the kernel is bounded,

$K:X\times X\to\mathbb C$0

then every RKHS function is bounded: $K:X\times X\to\mathbb C$1 Bounded kernels therefore produce RKHSs continuously embedded into $K:X\times X\to\mathbb C$2.

The feature-space distance is distinct from the projective distance obtained by normalizing kernel sections. The ordinary feature-space distance is

$K:X\times X\to\mathbb C$3

When $K:X\times X\to\mathbb C$4, define

$K:X\times X\to\mathbb C$5

The projective sine distance is

$K:X\times X\to\mathbb C$6

It measures the angle between normalized kernel sections and is invariant under nonzero rescaling of those sections. It also equals the operator norm distance between the rank-one projections $K:X\times X\to\mathbb C$7 and $K:X\times X\to\mathbb C$8: $K:X\times X\to\mathbb C$9 Related Skwarczyński and Fubini–Study distances have the same infinitesimal geometry, although their global formulas differ (Arcozzi et al., 2010).

3. Interpolation, approximation, and inclusion relations

Given distinct points $H$0, the finite-dimensional kernel space is

$H$1

An interpolant

$H$2

satisfying $H$3 is obtained from the Gram system

$H$4

If the kernel is strictly positive definite, $H$5 is nonsingular and the interpolant is unique.

The cardinal functions $H$6 satisfy

$H$7

so the interpolant is

$H$8

The orthogonal decomposition

$H$9

has

$X$0

Consequently, the kernel interpolant is the orthogonal projection of $X$1 onto $X$2 and preserves the prescribed nodal values (Azarnavid et al., 2017).

The kernel also determines minimum-norm interpolation. For constraints $X$3, the minimum-norm solution belongs to $X$4. For a single constraint $X$5,

$X$6

provided $X$7.

Inclusion of RKHSs is governed by kernel domination rather than pointwise comparison. For positive-semidefinite kernels $X$8 and $X$9,

$X$0

where $X$1 is positive semidefinite. The optimal constant

$X$2

determines the norm of the inclusion: $X$3

Equivalently, if

$X$4

then

$X$5

if and only if there exists a bounded operator $X$6 such that

$X$7

Thus the feature representation of the smaller RKHS must be obtainable from that of the larger one by a bounded linear transformation (Zhang et al., 2011).

For translation-invariant kernels on $X$8, Bochner’s theorem gives

$X$9

Then

$H\subseteq\mathbb C^X$0

if and only if

$H\subseteq\mathbb C^X$1

where $H\subseteq\mathbb C^X$2 is the representing measure of $H\subseteq\mathbb C^X$3. The optimal inclusion constant is

$H\subseteq\mathbb C^X$4

For Hilbert–Schmidt kernels, the analogous condition is coefficientwise domination: $H\subseteq\mathbb C^X$5 for all $H\subseteq\mathbb C^X$6. Equivalent norms require two-sided domination rather than one-sided inclusion.

The inclusion order is stable under several kernel operations. Sums, positive scalar multiplication, products through the Schur product theorem, uniformly bounded pointwise limits, and analytic functions with nonnegative power-series coefficients preserve appropriate forms of inclusion.

4. Analytic, geometric, stochastic, and operator-valued RKHSs

RKHSs arise naturally from differential operators and geometric structures. On a complete $H\subseteq\mathbb C^X$7-dimensional Riemannian manifold of bounded geometry, the Sobolev space $H\subseteq\mathbb C^X$8 is an RKHS when

$H\subseteq\mathbb C^X$9

Its kernel is the integral kernel of the elliptic smoothing operator

$f\mapsto f(x)$00

where $f\mapsto f(x)$01 is the Laplace–Beltrami operator: $f\mapsto f(x)$02 On a compact manifold with Laplacian eigenfunctions $f\mapsto f(x)$03 and eigenvalues $f\mapsto f(x)$04,

$f\mapsto f(x)$05

Diffusion spaces

$f\mapsto f(x)$06

are RKHSs for every $f\mapsto f(x)$07, with reproducing kernel equal to the heat kernel

$f\mapsto f(x)$08

They impose exponential spectral decay and are smoother than every fixed-order Sobolev space (Vito et al., 2019).

Reachable spaces of boundary-controlled heat equations also possess RKHS structures. If a state is represented as

$f\mapsto f(x)$09

then its range, equipped with the quotient norm, is an RKHS with kernel

$f\mapsto f(x)$10

For a finite rod, analytically continued reachable states form RKHSs on squares; for the half-line, the reachable space is a sum of Gaussian-weighted pullbacks of Bergman and Hardy spaces on the right half-plane (Lopez-Garcia, 2019).

The operator reproducing kernel Hilbert space (ORKHS) generalizes point evaluation. Let $f\mapsto f(x)$11 and $f\mapsto f(x)$12 be Hilbert spaces and let

$f\mapsto f(x)$13

be a family of continuous linear operators. An ORKHS is a Hilbert space for which the operators $f\mapsto f(x)$14 are continuous and separate elements of $f\mapsto f(x)$15. Its operator reproducing kernel is

$f\mapsto f(x)$16

and

$f\mapsto f(x)$17

The induced operator-valued kernel is

$f\mapsto f(x)$18

This framework includes Fourier coefficients, wavelet coefficients, integral measurements, local averages, vector-valued observations, and operator-valued data. Perfect ORKHSs reproduce both ordinary point evaluations and another family of operators. Regularized learning from operator-valued observations has minimizers of the form

$f\mapsto f(x)$19

an operator-valued representer theorem (Wang et al., 2015).

Positive-definite kernels are also covariance kernels. Every such kernel is the covariance kernel of a centered Gaussian process: $f\mapsto f(x)$20 The associated RKHS is the deterministic Hilbert space generated by the same covariance structure. Kernel transforms map measures or bounded linear functionals into RKHS elements: $f\mapsto f(x)$21 with

$f\mapsto f(x)$22

This identifies RKHS geometry with moment, measure, and stochastic-process geometry (Jorgensen et al., 2022).

Group and groupoid representations produce kernels through matrix coefficients. For a unitary representation $f\mapsto f(x)$23 of a group $f\mapsto f(x)$24 and $f\mapsto f(x)$25 in the representation space,

$f\mapsto f(x)$26

Conversely, a positive-definite kernel satisfying the appropriate covariance identity generates a unitary representation by translating kernel sections. The groupoid version decomposes the RKHS over range fibers and connects kernel geometry with convolution, harmonic analysis, complex transformations, and representation theory (Drewnik et al., 2021).

5. Kernel methods, distributions, and mean-field limits

In machine learning, a datum $f\mapsto f(x)$27 is mapped to its kernel section

$f\mapsto f(x)$28

Linear functionals in the RKHS correspond to nonlinear functions of the original input. Minimum-norm interpolation, kernel ridge regression, support-vector methods, and regularized empirical risk minimization exploit this feature-space representation.

For a loss depending only on values at training points, the representer theorem reduces an infinite-dimensional optimization problem to the finite span of training sections: $f\mapsto f(x)$29 For squared loss with RKHS penalty,

$f\mapsto f(x)$30

where $f\mapsto f(x)$31 is the Gram matrix.

Probability measures can themselves serve as inputs. For a probability measure $f\mapsto f(x)$32, the kernel mean element is

$f\mapsto f(x)$33

A kernel is characteristic if $f\mapsto f(x)$34 is injective. In one-dimensional distribution regression, the quadratic Wasserstein distance is the $f\mapsto f(x)$35-distance between quantile functions: $f\mapsto f(x)$36 The kernel

$f\mapsto f(x)$37

is positive definite. For $f\mapsto f(x)$38, under compactness and regularity assumptions, it is universal on $f\mapsto f(x)$39, allowing uniform approximation of continuous functions of probability distributions (Bui et al., 2018).

Mean-field limits extend this construction to systems with many measurement variables. A symmetric configuration

$f\mapsto f(x)$40

is represented by its empirical measure

$f\mapsto f(x)$41

If a sequence of kernels $f\mapsto f(x)$42 is permutation invariant, uniformly bounded, and equicontinuous through empirical measures, then a subsequence converges uniformly to a continuous positive-definite kernel

$f\mapsto f(x)$43

The associated finite-$f\mapsto f(x)$44 RKHS functions have subsequences converging uniformly, on empirical configurations, to functions in the limiting RKHS. Double-sum kernels provide an important example: $f\mapsto f(x)$45 whose limit is

$f\mapsto f(x)$46

The convergence is subsequential and function-wise; it does not assert convergence of the entire sequence of RKHSs in operator norm (Fiedler et al., 2023).

RKHSs also arise in meshless collocation. Cardinal functions constructed from kernel sections yield differentiation matrices

$f\mapsto f(x)$47

and pointwise differentiation errors satisfy

$f\mapsto f(x)$48

where $f\mapsto f(x)$49 is a kernel-determined interpolation error. Sobolev RKHS trial spaces can incorporate homogeneous boundary conditions directly, producing meshless collocation schemes for boundary-value, parabolic, and Burgers equations (Azarnavid et al., 2017).

6. Algebraic structure, stability, and limitations

Pointwise multiplication does not generally preserve an RKHS. A reproducing kernel Hilbert algebra (RKHA) is an RKHS for which the diagonal map

$f\mapsto f(x)$50

extends boundedly from $f\mapsto f(x)$51 to $f\mapsto f(x)$52. Its adjoint

$f\mapsto f(x)$53

is pointwise multiplication: $f\mapsto f(x)$54 Hence

$f\mapsto f(x)$55

The comultiplication is coassociative and cocommutative, and the RKHA structure is closed under Hilbert-space tensor products and pullbacks (Giannakis et al., 2024).

For applications to system identification, an RKHS of impulse responses is called stable when every element is absolutely integrable: $f\mapsto f(x)$56 in continuous time, or

$f\mapsto f(x)$57

in discrete time. If $f\mapsto f(x)$58 is the kernel-induced operator

$f\mapsto f(x)$59

then stability is equivalent to boundedness

$f\mapsto f(x)$60

For continuous Mercer kernels under the stated integrability assumptions, this can be tested using only sign-valued functions $f\mapsto f(x)$61 satisfying $f\mapsto f(x)$62 almost everywhere: $f\mapsto f(x)$63 The discrete-time analogue holds for the entire class of kernels as a boundedness test for $f\mapsto f(x)$64 (Bisiacco et al., 2023).

The existence of an RKHS between two Banach function spaces is constrained by Hilbertian geometry. For proper Banach function spaces $f\mapsto f(x)$65, there exists an intermediate RKHS

$f\mapsto f(x)$66

if and only if the inclusion operator $f\mapsto f(x)$67 factors through a Hilbert space. Such 2-factorability implies type $f\mapsto f(x)$68 and cotype $f\mapsto f(x)$69 conditions. For classical smoothness spaces on a $f\mapsto f(x)$70-dimensional domain, the resulting thresholds involve dimension-dependent smoothness gaps. For example, an RKHS with bounded kernel containing a Sobolev-type space of smoothness $f\mapsto f(x)$71 requires, under the stated hypotheses,

$f\mapsto f(x)$72

This obstruction reflects the abundance of localized, nearly independent functions in high-dimensional domains rather than pointwise smoothness alone (Schölpple et al., 2023).

Finally, RKHS inclusion and metric properties have important degeneracies. Kernel-induced distances may be pseudometrics if kernel sections fail to separate points. A zero kernel section makes the corresponding point invisible to the RKHS. Kernel inclusion is not determined by pointwise kernel magnitude, and equivalent norms are stronger than set inclusion. Regularity of a kernel in an external metric does not automatically give the same regularity for all RKHS functions without exponent loss or additional feature-map structure. Likewise, mean-field convergence generally requires compactness, symmetry, uniform boundedness, equicontinuity, and often passage to a subsequence. These qualifications are intrinsic to the theory: an RKHS is determined not merely by a similarity formula, but by the complete interaction among positive definiteness, Hilbert geometry, evaluation continuity, feature representations, and the function-space topology.