Vector-Valued RKHSs Overview

• Vector-valued RKHSs are Hilbert spaces of functions mapping inputs to a Hilbert space, characterized by an operator-valued positive definite kernel and a reproducing property.
• They generalize classical scalar RKHS theory by employing spectral representations and feature map constructions to address multi-task and structured learning challenges.
• Refinement strategies and extensions to Banach spaces and Hilbert C*-modules enable rigorous analysis and practical applications in infinite-dimensional settings and stochastic processes.

Vector-valued reproducing kernel Hilbert spaces (RKHSs) are Hilbert spaces of functions mapping a set into a vector space or Hilbert space, equipped with a reproducing property governed by an operator- or matrix-valued positive definite kernel. These spaces generalize classical scalar RKHS theory and form the rigorous analytic foundation for a wide range of methodologies in modern signal processing, machine learning, multi-task learning, functional analysis, and operator theory. The theory encompasses constructions via operator-valued kernels, integral representations, extensions of classical theorems such as Mercer’s theorem, kernel refinement strategies, functional-analytic tools for infinite-dimensional outputs, as well as connections to Banach space theory and Hilbert C*-module frameworks.

1. Foundations of Vector-Valued RKHSs and Operator-Valued Kernels

Vector-valued RKHSs $\mathcal{H}_K$ consist of functions $f : X \to \Lambda$, where $X$ is the input set and $\Lambda$ is a Hilbert space (possibly infinite-dimensional), with the key property that point-evaluation functionals are continuous:

$f \mapsto f(x) \in \Lambda,\quad \forall x \in X.$

This is realized via an operator-valued positive-definite kernel $K: X \times X \to \mathcal{L}(\Lambda)$ such that for all finite $(x_i, y_i) \in X \times \Lambda$:

$$\sum_{i,j} \langle K(x_i, x_j) y_i, y_j \rangle_\Lambda \geq 0,$$

and the reproducing property holds:

$$\langle f(x), y \rangle_\Lambda = \langle f, K(\cdot, x) y \rangle_{\mathcal{H}_K} \;\; \forall f \in \mathcal{H}_K,\ x \in X,\ y \in \Lambda.$$

The inner product and norm on $\mathcal{H}_K$ are inherited from the kernel structure, which induces the geometry and completeness.

A feature map representation exists for any such kernel:

$K(x, y) = \Phi(x)^* \Phi(y)$

where $\Phi: X \to \mathcal{L}(\Lambda, \mathcal{W})$ is a feature map into an auxiliary Hilbert space $\mathcal{W}$. This construction forms the basis for explicit computation and analysis of the function space, as well as for the derivation of generalizations such as kernel refinement and integral representations (Xu et al., 2011).

2. Spectral Representations, Mercer Theorem, and Topology

The extension of the Mercer theorem to vector-valued measurable kernels establishes that, for a measurable (matrix/operator-valued) kernel $K$ with a separable associated RKHS, there exists a series expansion in terms of eigenfunctions of a compact integral operator:

$$K(x, t)_{l,j} = \sum_{i \in I} \sigma_i f_i(t) f_i(x),$$

where $\{f_i\}$ are orthonormal in $L^2(X, v; \mathbb{C}^n)$ and $\{\sigma_i\}$ are strictly positive eigenvalues, with the expansion converging uniformly on compacts after introducing an appropriate second countable topology induced from the kernel (Vito et al., 2011).

Critical steps involve defining a pseudo-metric:

$$d(x, t) = \sup_{\|y\| = 1} \|K(x, y) - K(t, y)\|_{\Lambda},$$

which is used to generate the topology under which the continuity and convergence properties of the kernel and its Mercer representation are guaranteed.

This spectral theory is pivotal in high-dimensional and infinite-dimensional applications for representing kernels, understanding their approximation power, and implementing kernel PCA and related methods.

3. Refinement and Adaptivity via Feature and Integral Representations

Adjusting (refining) operator-valued kernels is fundamental in multi-task learning, where hypothesis spaces must be adaptable to avoid underfitting or overfitting. The refinement theory formalizes the construction of a new kernel $G$ such that $\mathcal{H}_K \subset \mathcal{H}_G$ isometrically, with refinement typically achieved by augmenting the feature space:

$G(x, y) = K(x, y) + \Theta(x)^* \Theta(y)$

for an auxiliary feature map $\Theta$. For translation-invariant kernels, an integral representation is used:

$$K(x, y) = \int_\Omega \varphi(x, t) \Gamma(t) \varphi(y, t)^* d\gamma(t),$$

with the refinement realized by adjusting $\Gamma$ (for example, $\Gamma(t) \preceq \tilde{\Gamma}(t)$) (Xu et al., 2011). This refinement mechanism preserves continuity, universality, and orthogonal decomposability, and enables theoretical and practical adaptation of vector-valued kernel methods to changing learning requirements.

4. Functional-Analytic Techniques for Infinite-Dimensional Outputs

When the output space $\Lambda$ is infinite-dimensional, as in functional regression or structured prediction, standard compactness properties of the kernel integral operator may fail. Spectral theory for non-compact, self-adjoint operators is then employed. For regularized least-squares regression in a vector-valued RKHS $\mathcal{H}$, with $Y$ an $\mathcal{H}$-valued random variable,

$$R_\lambda(f) = \mathbb{E} \|f(X) - Y\|^2_\mathcal{H} + \lambda \|f\|^2_\mathcal{H},$$

the optimal solution is characterized by the integral operator inclusion and its adjoint:

$$f_\lambda = (\iota^* \iota + \lambda I_{\mathcal{H}})^{-1} \iota^* f^*,$$

where $$\iota: \mathcal{H} \hookrightarrow L^2(\mathcal{X}, P_X; \Lambda)$$ is the inclusion. The spectral theorem allows explicit control of the resolvent $(T_n + \lambda I_\mathcal{H})^{-1}$ without compactness, leading to universal consistency under minimal assumptions (Park et al., 2020).

5. Applications: Multi-Task Learning, Manifold Regularization, Functional Models

Vector-valued RKHSs provide a unifying framework for manifold regularization and co-regularized multi-view learning, with operator-valued kernels enabling the modeling of structured outputs and inter-view dependencies. Closed-form solutions arise via the representer theorem applied to least-squares and SVM formulations:

$(K + \lambda_A I + \lambda_I K L) c = y,$

where $L$ is a graph Laplacian encoding geometric data structure. Quadratic programming formulations underpin SVMs with operator-valued kernels for multi-output, multi-class, and multi-view settings (Minh et al., 2014).

In functional analysis and operator theory, vector-valued RKHSs support the construction of functional models for multiplication operators and their commutants (Chavan et al., 2017). For vector-valued de Branges spaces, the synthesis of operator-valued analytic kernels linked to J-contractive functions is used to derive RKHSs of entire functions with rich structural and spectral properties, supporting parametrization of selfadjoint extensions and functional models for symmetric operators with infinite deficiency indices (Mahapatra et al., 2023, Garg et al., 4 Nov 2024).

6. Banach Space Generalizations, Hilbert C*-Modules, and Relative Kernels

The theory of vector-valued RKHSs extends into Banach space analogs via generalized Mercer kernels, leading to Reproducing Kernel Banach Spaces (RKBSs) built upon $p$-norm sequence spaces, admitting representations

$$f(x) = \sum_{n} a_n \phi_n(x),\quad \|f\|_{B_K^p} = \|(a_n)\|_{\ell^p}$$

with vector-valued expansions defined analogously, allowing sparsity and geometric structure control (Xu et al., 2014).

Reproducing kernel Hilbert C*-modules (RKHM) further generalize by allowing the inner product to take values in a C*-algebra $A$, with module-valued kernels $k : \mathcal{X} \times \mathcal{X} \to A$ and corresponding structural theorems, representer theorems, and injectivity/universality analysis for kernel mean embeddings (Hashimoto et al., 2021, Moslehian, 2021, Hashimoto et al., 2020, Hashimoto et al., 2022). These frameworks provide enhanced flexibility for representing continuous and operator-theoretic structure—critical for functional data, operator models, and invariant feature extraction.

Relative reproducing kernels in both Hilbert and Banach settings extend the reproducing paradigm to differences:

$f(y) - f(x) = \langle f, M_{x, y} \rangle,$

introducing cocycle and additive properties required for applications focusing on variational or invariant tasks (Ebadian et al., 2016).

7. Connections to Sampling, Frames, and Stochastic Processes

Sampling theory in vector-valued RKHSs is formulated using point-mass evaluation and interpolation via kernels, with the conditions for reconstructibility depending on the inclusion of Dirac masses, Gramian matrices, and symmetric operator pairs (Jorgensen et al., 2016). Frame theory links with RKHS structures, yielding kernel representations as $K^G(s, t) = \langle l(s), G^{-1} l(t) \rangle_2$ when the vectors $(\varphi_n(t))$ form a frame, and isometric isomorphisms between Hilbert spaces and RKHSs are constructed via the frame operator (Jorgensen et al., 2016).

Gaussian processes and stochastic analysis are intertwined with vector-valued RKHS theory, particularly through infinite-dimensional transforms and the construction of spectral and sampling representations suitable for multivariate signals and processes (Jorgensen et al., 2022).

---

In summary, vector-valued RKHSs provide a versatile, scalable, and analytically tractable framework for multi-output learning, functional data analysis, structured prediction, and operator-theoretic spectral modeling. The breadth of the theory encompasses spectral decomposition, refinement and adaptation, Banach and C*-module extensions, and rich connections to sampling, frame theory, and stochastic process representations, making them a central tool across several domains of mathematical analysis and modern machine learning.