Automatic RKHS Regularization

• Automatic RKHS regularization is a framework that uses data-adaptive and probabilistic penalties within reproducing kernel Hilbert spaces to control function complexity and enhance generalization.
• It leverages Bayesian priors, adaptive penalty rules, and techniques like Lepskii’s principle to automatically determine regularization strength, reducing dependence on manual tuning.
• Extensions include operator-induced RKHSs, structured sparsity, and deep compositional models that incorporate variable weighting and layerwise norm control to improve interpretability and performance.

Automatic RKHS regularization designates a family of methods in which estimation in a reproducing kernel Hilbert space is stabilized by a regularizing mechanism that is intrinsic to the RKHS, induced by a probabilistic model, adapted to the data or forward operator, or selected from the data without a fixed hand-chosen penalty schedule. In the classical setting, one minimizes empirical loss plus an RKHS norm penalty, and the representer theorem converts the infinite-dimensional problem into a finite kernel expansion over the sample points (Aravkin et al., 2013, Chen et al., 2017). Subsequent work broadens the meaning of “automatic” in several directions: Bayesian derivations in which the penalty is determined by prior and noise scales (Aravkin et al., 2013), adaptive penalties that grow slower than the standard quadratic RKHS norm (Mendelson et al., 2010, Audiffren et al., 2013), data-driven parameter-choice rules such as Lepskii’s principle and stochastic approximation along regularization paths (Mücke, 2018, Tarrès et al., 2011), operator-induced RKHSs for inverse problems (Lu et al., 2022, Li et al., 2024, Li et al., 16 Jul 2025), structured sparsity in both variables and dual coefficients (Chen et al., 2017), and deep compositional models with layerwise RKHS control (Bietti et al., 2018, Chen et al., 13 May 2026).

1. Classical formulation and the basic RKHS mechanism

The canonical RKHS regularization problem takes the form

$$f^*=\arg\min_{f\in\mathcal H_K}\Big\{\sum_{i=1}^n V\big(y_i-f(x_i)\big)+\lambda \|f\|_{\mathcal H_K}^2\Big\},$$

or, with an intercept when constant functions are not contained in the RKHS, $f=\tilde f+b$ with only $\|\tilde f\|_{\mathcal H_K}^2$ penalized (Aravkin et al., 2013, Chen et al., 2017). Under mild assumptions on the loss, the representer theorem yields

$f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$

so that optimization reduces to coefficients in the finite-dimensional span of kernel sections. In matrix form, with Gram matrix $K$, one has $\|f\|_{\mathcal H_K}^2=c^\top K c$, and standard cases include kernel ridge regression for squared loss and kernel support vector machines for hinge loss (Aravkin et al., 2013, Chen et al., 2017).

In this baseline formulation, regularization acts by controlling smoothness or complexity in function space. The squared RKHS norm provides a global stabilizer and underlies the classical spline and kernel machinery. At the same time, it does not automatically zero out irrelevant input coordinates, does not generally induce sparsity in the dual coefficients, and does not by itself adapt to structural properties such as sparse variable relevance or sparse kernel expansions (Chen et al., 2017). This limitation is one of the main points of departure for later notions of automatic regularization.

2. Probabilistic and distribution-level interpretations

A central statistical interpretation identifies RKHS regularization with Gaussian random field or Gaussian process priors. If $f \sim \mathcal{GP}(0,\alpha K)$ and the observation model is $y_i=f(x_i)+\epsilon_i$ with $\epsilon_i \sim \mathcal N(0,\sigma^2)$, then the RKHS estimator with squared loss coincides with the posterior mean, with coefficients

$$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$

so the regularization parameter is the ratio of noise variance to prior scale (Aravkin et al., 2013). For non-Gaussian losses such as absolute value, Vapnik, or Huber, the finite-dimensional MAP estimate of the vector of function values at any sampled locations coincides with the RKHS regularized solution evaluated at those locations, and the representer theorem extends the estimate to the full domain (Aravkin et al., 2013). This gives a rigorous sense in which RKHS regularization can arise automatically from a prior-likelihood specification rather than from an externally imposed penalty.

Kernel Bayesian inference extends the same theme from functions to posterior embeddings in vector-valued RKHSs. In that setting, the posterior embedding is recovered as the optimizer of a vector-valued regression problem, and the resulting objective induces a new posterior regularization based on thresholded weights $f=\tilde f+b$0 rather than only squared operator regularization (Song et al., 2016). The corresponding estimator has the representer form

$f=\tilde f+b$1

and the paper proves consistency of the thresholding mechanism in the finite-$f=\tilde f+b$2 setting, including $f=\tilde f+b$3 in probability (Song et al., 2016). It also introduces distribution-level posterior regularization, denoted kRegBayes, in which additional target embeddings $f=\tilde f+b$4 enforce posterior concentration directly in RKHS geometry rather than through moment constraints (Song et al., 2016).

These probabilistic constructions do not eliminate regularization; instead, they reinterpret it. The penalty is the finite-dimensional expression of a prior covariance, and the regularization strength is inherited from variance scales or posterior constraints. This suggests a notion of automaticity rooted in model specification rather than in post hoc hyperparameter heuristics.

3. Adaptive penalties and automatic choice of regularization strength

A separate line of work makes regularization automatic by changing either the growth of the penalty or the rule used to choose its strength. In regression with squared loss, one analysis shows that under weak-$f=\tilde f+b$5 eigenvalue decay of the kernel integral operator, one can replace the standard quadratic penalty by a slower term of order

$f=\tilde f+b$6

up to the logarithmic envelope appearing in the oracle inequality (Mendelson et al., 2010). The resulting regularizer grows slower than the standard quadratic RKHS norm and yields excess-risk rates $f=\tilde f+b$7 up to logarithmic factors under the source condition $f=\tilde f+b$8 (Mendelson et al., 2010).

A related modification replaces $f=\tilde f+b$9 by $\|\tilde f\|_{\mathcal H_K}^2$0 for an arbitrary exponent $\|\tilde f\|_{\mathcal H_K}^2$1. In $\|\tilde f\|_{\mathcal H_K}^2$2-power regularized least squares regression, $\|\tilde f\|_{\mathcal H_K}^2$3 recovers kernel ridge regression, $\|\tilde f\|_{\mathcal H_K}^2$4 gives a sub-quadratic penalty, and $\|\tilde f\|_{\mathcal H_K}^2$5 produces a nonconvex objective (Audiffren et al., 2013). The representer theorem still applies, and the finite-dimensional problem can be reduced to a one-dimensional optimization in a scalar $\|\tilde f\|_{\mathcal H_K}^2$6, after which $\|\tilde f\|_{\mathcal H_K}^2$7 (Audiffren et al., 2013). Under the eigenvalue condition $\|\tilde f\|_{\mathcal H_K}^2$8, the choice

$\|\tilde f\|_{\mathcal H_K}^2$9

removes dependence of the optimal $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$0 scaling on the approximation exponent $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$1, which the paper presents as a “parameter-free” prescription once $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$2 is known (Audiffren et al., 2013).

Automaticity can also be achieved at the parameter-choice stage. In RKHS regression with a general linear regularization scheme $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$3, Lepskii’s principle selects $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$4 by comparing estimators along a geometric grid and balancing empirical approximation against an empirical variance proxy built from the empirical effective dimension $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$5 (Mücke, 2018). The resulting rule is minimax optimal adaptive, up to $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$6, and an especially notable result is that balancing in $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$7 automatically yields optimal balancing in all stronger interpolation norms between $f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$8 and the RKHS (Mücke, 2018).

An online analogue appears in stochastic approximation of regularization paths. With updates

$f^*(x)=\sum_{i=1}^n c_i\,K(x,x_i),$9

and schedules $K$0, $K$1 with $K$2, the product $K$3 links the iteration count to a Tikhonov regularization path $K$4 (Tarrès et al., 2011). Under the source condition $K$5, the method matches the best-known strong RKHS convergence rate of batch learning and attains the minimax weak rate $K$6 for prediction error, up to the stated lower-order and logarithmic terms (Tarrès et al., 2011).

Taken together, these approaches make the penalty itself, or the parameter controlling it, depend on spectral geometry, smoothness assumptions, or empirically estimated complexity. The regularization is still explicit, but it is no longer restricted to a fixed quadratic template with a manually chosen constant.

4. Data-adaptive RKHSs for inverse problems, operators, and kernel learning

In inverse problems, automatic RKHS regularization often means that the RKHS is itself induced by the operator and the data. For Fredholm integral equations with Gaussian measurement noise, an exploration measure

$K$7

defines an ambient space $K$8, and the operator-induced kernel

$K$9

generates an RKHS $\|f\|_{\mathcal H_K}^2=c^\top K c$0 whose closure is the function space of identifiability (Lu et al., 2023). The corresponding Tikhonov estimators are $\|f\|_{\mathcal H_K}^2=c^\top K c$1 for the $\|f\|_{\mathcal H_K}^2=c^\top K c$2 penalty and $\|f\|_{\mathcal H_K}^2=c^\top K c$3 for the RKHS penalty, and the small-noise analysis shows that both attain the same linear rate in $\|f\|_{\mathcal H_K}^2=c^\top K c$4 while the RKHS regularizer has a smaller multiplicative constant (Lu et al., 2023).

DARTR generalizes this principle to learning kernels in operators. It constructs a system-intrinsic data-adaptive RKHS from an exploration measure $\|f\|_{\mathcal H_K}^2=c^\top K c$5 and a reweighted kernel $\|f\|_{\mathcal H_K}^2=c^\top K c$6, with $\|f\|_{\mathcal H_K}^2=c^\top K c$7 compact and positive on $\|f\|_{\mathcal H_K}^2=c^\top K c$8 (Lu et al., 2022). The resulting SIDA-RKHS $\|f\|_{\mathcal H_K}^2=c^\top K c$9 has norm $f \sim \mathcal{GP}(0,\alpha K)$0, and its closure is the function space of identifiability $f \sim \mathcal{GP}(0,\alpha K)$1, so regularization in $f \sim \mathcal{GP}(0,\alpha K)$2 automatically restricts learning to identifiable directions (Lu et al., 2022). The discrete implementation uses a generalized eigenproblem $f \sim \mathcal{GP}(0,\alpha K)$3, and the regularization parameter is selected by the L-curve curvature criterion (Lu et al., 2022).

iDARR replaces direct spectral solvers by an iterative scheme on RKHS-restricted Krylov subspaces. In matrix form the automatic penalty is

$f \sim \mathcal{GP}(0,\alpha K)$4

and generalized Golub-Kahan bidiagonalization constructs bases adapted to the inner product induced by $f \sim \mathcal{GP}(0,\alpha K)$5 (Li et al., 2024). The method solves projected least-squares problems in subspaces where the solution is unique, tracks both residual norm and RKHS norm for L-curve or discrepancy-principle stopping, and has total complexity $f \sim \mathcal{GP}(0,\alpha K)$6 for $f \sim \mathcal{GP}(0,\alpha K)$7-by-$f \sim \mathcal{GP}(0,\alpha K)$8 matrices and $f \sim \mathcal{GP}(0,\alpha K)$9 iterations (Li et al., 2024).

For learning convolution kernels, the same operator-driven idea is made explicit in a data-adaptive RKHS with kernel

$y_i=f(x_i)+\epsilon_i$0

where $y_i=f(x_i)+\epsilon_i$1 is built from the forward operator and sampled inputs (Li et al., 16 Jul 2025). The semi-continuum basis functions $y_i=f(x_i)+\epsilon_i$2 satisfy $y_i=f(x_i)+\epsilon_i$3, and the finite-dimensional representer theorem shows that the minimal-norm least-squares solution, the Tikhonov solution, and conjugate-gradient iterates all lie in $y_i=f(x_i)+\epsilon_i$4 (Li et al., 16 Jul 2025). Hybrid iterative–Tikhonov procedures and weighted-GCV are then defined entirely in this automatic basis (Li et al., 16 Jul 2025).

Learning Lévy densities from nonlocal Fokker–Planck data follows the same pattern but adds a bi-level selection of $y_i=f(x_i)+\epsilon_i$5. The adaptive kernel $y_i=f(x_i)+\epsilon_i$6 defines $y_i=f(x_i)+\epsilon_i$7, and the lower-level Tikhonov problem yields coefficients $y_i=f(x_i)+\epsilon_i$8 (Yang et al., 29 Dec 2025). A GSVD-based bilevel optimization then minimizes validation error over $y_i=f(x_i)+\epsilon_i$9, with the same GSVD reused for the lower-level solution and the hypergradient (Yang et al., 29 Dec 2025). Under source and spectral decay conditions, the reconstruction error decays in the mesh size at the near optimal rate stated in Theorem 3.1 (Yang et al., 29 Dec 2025).

Automatic RKHS regularization also appears when the object being learned is itself a kernel. In hyper-RKHS, one learns a bivariate function $\epsilon_i \sim \mathcal N(0,\sigma^2)$0 by minimizing pairwise regression loss plus $\epsilon_i \sim \mathcal N(0,\sigma^2)$1, and the representer theorem gives

$\epsilon_i \sim \mathcal N(0,\sigma^2)$2

This lifts Tikhonov regularization from functions to kernels, accommodates positive definite and indefinite outputs, and supports out-of-sample extension from a learned similarity function (Liu et al., 2018).

A further extension appears in derivative estimation from noisy time series. There, the derivative $\epsilon_i \sim \mathcal N(0,\sigma^2)$3 is estimated in a vector-valued RKHS by minimizing

$\epsilon_i \sim \mathcal N(0,\sigma^2)$4

and an integral-form representer theorem reduces the problem to a finite-dimensional linear system with integrated kernel blocks $\epsilon_i \sim \mathcal N(0,\sigma^2)$5 (Guo et al., 2 Apr 2025). The regularization parameter is chosen automatically by maximizing the curvature of the L-curve, and the same strategy is reused in a second vRKHS regression step for the state-space dynamics (Guo et al., 2 Apr 2025).

5. Structural sparsity, variable weighting, and posterior constraints

Some automatic regularizers do more than smooth the fitted function; they also induce sparsity or impose structure. DOSK is the most explicit example. It introduces a variable-weighted kernel

$\epsilon_i \sim \mathcal N(0,\sigma^2)$6

with $\epsilon_i \sim \mathcal N(0,\sigma^2)$7, so that $\epsilon_i \sim \mathcal N(0,\sigma^2)$8 removes predictor $\epsilon_i \sim \mathcal N(0,\sigma^2)$9 from the kernel (Chen et al., 2017). The optimization problem is

$$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$0

with $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$1 (Chen et al., 2017). Here $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$2 yields data sparsity, $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$3 yields variable sparsity, and $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$4 maintains smoothness. Under the stated assumptions and tuning conditions, DOSK achieves estimation error $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$5, variable selection consistency, and a generalization gap of order $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$6 (Chen et al., 2017).

The DOSK objective is convex in $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$7 conditional on $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$8 but generally nonconvex in $$c=(K+\lambda I)^{-1}y,\qquad \lambda=\sigma^2/\alpha,$$9, so the algorithm alternates an $f=\tilde f+b$00-step, a one-dimensional $f=\tilde f+b$01-step, and a convex quadratic program for $f=\tilde f+b$02 obtained by local linearization of the variable-weighted kernel (Chen et al., 2017). This illustrates a recurring theme in automatic RKHS regularization: regularization can be “automatic” in the sense of structural induction, even when optimization remains nonconvex and initialization-sensitive.

Distribution-level posterior regularization provides a different structural mechanism. In kernel Bayesian inference, thresholding $f=\tilde f+b$03 automatically suppresses negative components in the empirical posterior embedding objective, and kRegBayes augments the weighted regression loss with terms of the form $f=\tilde f+b$04 (Song et al., 2016). This allows posterior information to be encoded directly at the level of distributions rather than by tuning only a scalar smoothness parameter (Song et al., 2016).

Hyper-RKHS regularization extends structural control to the kernel-learning problem itself. Because the hyper-kernel $f=\tilde f+b$05 regularizes a bivariate kernel function $f=\tilde f+b$06, the Tikhonov term $f=\tilde f+b$07 controls the complexity of the learned similarity function over $f=\tilde f+b$08, not merely a scalar predictor on $f=\tilde f+b$09 (Liu et al., 2018). This is a finer distinction than the familiar contrast between feature selection and function smoothing: the regularized object is the kernel.

These examples indicate that automatic RKHS regularization is not limited to choosing $f=\tilde f+b$10. It can operate by selecting variables, training points, posterior constraints, or even the kernel function itself, while still using the RKHS norm as the analytic device that stabilizes the optimization.

6. Deep and compositional extensions

Deep learning motivated a different reinterpretation of RKHS regularization. One line of work treats a deep convolutional network as lying in the RKHS of a hierarchical convolutional kernel, and uses upper and lower bounds on the intractable RKHS norm to derive practical regularizers (Bietti et al., 2018). Lower bounds include adversarial perturbation penalties, gradient penalties, and deformation stability penalties, while upper bounds are expressed through spectral norms of the layer operators (Bietti et al., 2018). In this view, spectral norm penalties or constraints, predictor-gradient penalties, and adversarial training become computable surrogates for $f=\tilde f+b$11, and hybrid strategies combine lower and upper approximations to obtain stronger control on both local stability and global complexity (Bietti et al., 2018). The same framework yields an adversarially robust margin bound in which the RKHS norm controls the perturbation term $f=\tilde f+b$12 (Bietti et al., 2018).

Wahkon pushes the construction further by placing each univariate link function in a deep Kolmogorov-style superposition network inside an RKHS and penalizing it layerwise: $f=\tilde f+b$13 A deep representer theorem then gives finite expansions

$f=\tilde f+b$14

so training reduces to a finite-dimensional coefficient problem (Chen et al., 13 May 2026). Under independent GP priors $f=\tilde f+b$15 and Gaussian noise, the MAP estimator coincides exactly with the penalized objective when $f=\tilde f+b$16, which extends the spline/GP duality from single-layer RKHS models to deep compositions (Chen et al., 13 May 2026). The same paper derives an entropy bound, an explicit architecture factor

$f=\tilde f+b$17

and a convergence rate balancing approximation, regularization bias, and stochastic error (Chen et al., 13 May 2026).

These deep constructions preserve the central RKHS logic—control of function-space complexity through norms or equivalent GP priors—while relocating it to learned feature hierarchies. They also clarify an important limitation. Exact RKHS norms are often intractable in deep models, and practical methods rely on surrogates, profile objectives, inducing points, or last-layer conditioning (Bietti et al., 2018, Chen et al., 13 May 2026). This suggests that “automatic” does not mean the disappearance of approximation, tuning, or optimization issues. Rather, it means that complexity control is tied to an explicit RKHS or GP geometry instead of being left implicit.

Across the literature, automatic RKHS regularization therefore denotes a broad methodological principle rather than a single algorithm. In some papers it refers to penalties that emerge from Bayesian priors; in others, to data-adaptive RKHSs induced by operators and sampling geometry; in others still, to adaptive selection of $f=\tilde f+b$18, sparsity-inducing structures, or deep layerwise norm control. What unifies these variants is the use of RKHS geometry to turn ill-posedness, overfitting, or architectural flexibility into a regularized optimization problem whose bias-variance, identifiability, or generalization properties can be analyzed explicitly.