Deep Vector-Valued RKHS in Modern Learning

• Deep vector-valued RKHS are Hilbert spaces where functions map inputs to vector outputs using operator-valued kernels.
• Deep vvRKHS architectures layer multiple kernel-based maps, allowing complex multi-output functions to be represented via finite-dimensional expansions.
• This framework unifies multi-output regression, deep kernel networks, and neural operator models with optimization tractability via representer theorems.

Deep vector-valued reproducing kernel Hilbert spaces (vvRKHS) provide a rigorous mathematical framework for modeling and learning vector-valued functions, especially in the context of multi-layer architectures and neural operator models. vvRKHS generalize classical scalar-valued RKHS to handle outputs in Hilbert spaces, enabling kernel-based treatment of high-dimensional, multi-output, and operator-valued tasks. This theory underpins many modern developments in deep learning, kernel methods, and neural operators, particularly those involving infinite-width limits, structured outputs, and operator regression.

1. Foundations of Scalar and Vector-Valued RKHS

A scalar-valued RKHS $\mathcal{H}$ consists of functions $f:\mathcal{X}\to\mathbb{R}$ with an inner product $\langle\cdot,\cdot\rangle_{\mathcal{H}}$ such that for every $x\in\mathcal{X}$, point evaluation is continuous and is represented via a reproducing kernel $K:\mathcal{X}\times\mathcal{X}\to\mathbb{R}$ by the property $f(x) = \langle f, K(\cdot,x)\rangle_{\mathcal{H}}$ (Diwale et al., 2018).

In the vector-valued setting, one replaces the output space $\mathbb{R}$ with a Hilbert space $\mathcal{Z}$, commonly $\mathbb{R}^d$. A function $f:\mathcal{X}\to\mathcal{Z}$ belongs to a vvRKHS $\mathcal{H}$ if there exists a unique positive semidefinite, self-adjoint operator-valued kernel $$K:\mathcal{X}\times\mathcal{X}\to\mathcal{L}(\mathcal{Z},\mathcal{Z})$$ such that for all $z\in\mathcal{Z}$ and $f\in\mathcal{H}$,

$$\langle f(x), z\rangle_{\mathcal{Z}} = \langle f, K(\cdot, x)z\rangle_{\mathcal{H}}.$$

Point-evaluation is bounded as $$\|f(x)\|_{\mathcal{Z}} \leq C_x \|f\|_{\mathcal{H}}$$ for all $f$ (Dummer et al., 30 Sep 2025).

2. Generalized Representer Theorem for vvRKHS

The representer theorem in vvRKHS extends classical results to Hilbert space-valued functionals, allowing for variational problems with multiple linear constraints and general regularization. Given continuous linear operators $L_1,\ldots,L_m,L_{m+1}$ on $\mathcal{H}$ and functionals $$C: \mathcal{Z}_1\times\cdots\times\mathcal{Z}_m\to\mathbb{R}\cup\{+\infty\}$$ and $$\Omega:\mathcal{Z}_{m+1}\to\mathbb{R}\cup\{+\infty\}$$, the minimization problem

$$\min_{f\in\mathcal{H}} J(f),\quad J(f)=C(L_1f,\dots,L_mf)+\Omega(L_{m+1}f)$$

admits a minimizer in a finite-dimensional subspace explicitly determined by the data, the linear operators, and the regularizer's orthomonotonicity with respect to subspace-valued maps (Diwale et al., 2018). For classical empirical risk minimization with convex loss $L$ and squared vvRKHS norm, the optimal $f^*$ has finite-sample expansion

$$f^*(\cdot) = \sum_{i=1}^n K(\cdot,x_i)c_i, \quad c_i\in\mathcal{Z}.$$

This reduction is key for algorithmic tractability in both shallow and deep networks (Dummer et al., 30 Sep 2025).

Table 1: Summary of Representer Theorem Aspects

Aspect	Scalar RKHS	Vector-Valued RKHS (vvRKHS)
Output Space	$\mathbb{R}$	Hilbert space $\mathcal{Z}$
Kernel	$K: \mathcal{X}\times\mathcal{X}\to\mathbb{R}$	$$K: \mathcal{X}\times\mathcal{X}\to\mathcal{L}(\mathcal{Z})$$
Representer Expansion	$\sum_{i=1}^n K(\cdot, x_i)c_i$ where $c_i\in\mathbb{R}$	$\sum_{i=1}^n K(\cdot, x_i)c_i$, $c_i\in\mathcal{Z}$

3. Deep vvRKHS Architectures

Deep vvRKHS models are constructed by composing multiple vvRKHS layers. Let $N$ denote the number of layers, each associated with a Hilbert space $\mathcal{H}^{(\ell)}$, an operator-valued kernel $K^{(\ell)}$, and a subspace-valued map $S_{\ell}$. Layer $\ell$ realizes a map $$f^{(\ell)}:\mathcal{X}^{(\ell-1)}\to\mathcal{X}^{(\ell)}$$, and intermediate activations $y^{(\ell)}$ satisfy $y^{(\ell)}=f^{(\ell)}(y^{(\ell-1)})$. The composite function is $y = f^{(N)}\circ\cdots\circ f^{(1)}(x)$. The resulting variational problem sums layerwise data-fit and RKHS norm penalties:

$$\min_{\{f^{(\ell)},y^{(\ell)}\}} \sum_{\ell=1}^N \|y^{(\ell)}-f^{(\ell)}(y^{(\ell-1)})\|^2 + \lambda_\ell\|f^{(\ell)}\|^2_{\mathcal{H}^{(\ell)}}.$$

Each $f^{(\ell)}$ admits a kernel expansion

$$f^{(\ell)}(\cdot) = \sum_{i=1}^m K^{(\ell)}(\cdot, y_i^{(\ell-1)}) c_i^{(\ell)}$$

so the model is a multi-layer kernel machine (Diwale et al., 2018). The layerwise expansions yield explicit, finite-dimensional parametrizations for deep kernel networks and facilitate optimization by standard solvers at each layer.

4. Operator-Valued and Neural Kernels

The kernel $K$ in vvRKHS can be specialized to encode deep neural architectures and operator learning models:

• Arc-cosine and NTK kernels: For deep ReLU networks, scalar arc-cosine kernel recursions are lifted to operator-valued kernels by multiplying by the output space identity, $K^{(\ell)}(x,x') = k^{(\ell)}(x,x') I_{d_y}$. The Neural Tangent Kernel is constructed analogously, capturing the functional limit of infinitely wide neural networks (Dummer et al., 30 Sep 2025).
• DeepONet and hypernetworks: DeepONet maps $z\in Z$ to $C(X;\mathbb{R}^{d_y})$ using a branch-trunk construction, producing a kernel $K((z,x),(z',x')) = k_Z(z,z')k_X(x,x')I_{d_y}$ in the infinite-width limit. Hypernetwork architectures, conditioning on $z$ to obtain network weights, similarly yield operator-valued kernels over mixed domains.

These constructions demonstrate that deep, vector-valued architectures and neural operator models are instances of vvRKHS with explicit operator-valued kernels reflecting the network structure and data domains (Dummer et al., 30 Sep 2025).

5. Computational Aspects and Algorithmic Implications

Deep vvRKHS architectures enable finite-dimensional optimization by reducing infinite-dimensional function search to coefficient learning via the representer theorem. Each layer's kernel matrix must be computed and factorized, commonly incurring $\mathcal{O}(m^3)$ cost for $m$ data points, motivating low-rank methods, inducing-point schemes, or randomized features. Layerwise convexity (when activations are fixed) contrasts with the overall nonconvexity due to composition. Automatic differentiation can be performed through layers of kernel expansions, supporting integration with deep learning toolkits and optimization pipelines (Diwale et al., 2018).

6. Theoretical Unification and Applications

The deep vvRKHS framework unifies diverse learning tasks:

• Multi-output regression: $\mathcal{Z}=\mathbb{R}^d$, yielding $K(x,x')\in\mathbb{R}^{d\times d}$ and multi-output least squares.
• Kernel-based deep networks: Application of nonlinear activations (e.g., ReLU) between vvRKHS layers, solved by multiple-shooting variational principles.
• Operator learning: Infinite-width DeepONet and hypernetwork models correspond to vvRKHSs over function-space or mixed domains, providing a representer-based algorithmic structure.
• Stochastic regression: Gaussian processes over vector-valued outputs are recovered as special cases, with posterior mean predictors as finite expansions.

A plausible implication is that this framework enables principled construction, training, and theoretical analysis of multi-layer, vector-valued, and operator neural architectures, supporting both universal approximation and algorithmic tractability (Diwale et al., 2018, Dummer et al., 30 Sep 2025).