Scale-Relative Kernel Parameterization

• Scale-Relative Kernel Parameterization is a framework that treats scale as a learnable parameter to dynamically shape the kernel's functional form and estimation impact.
• It is applied in robust estimation, adaptive kernel regression, and CNN architectures, significantly enhancing model expressivity and optimization.
• The approach unifies multi-scale data analysis by integrating learnable bandwidths and hyperparameters, boosting stability and performance across applications.

Scale-Relative Kernel Parameterization

A scale-relative (or scale-variant) kernel parameterization is a formal and algorithmic framework in which the notion of "scale"—whether as a kernel width, bandwidth, or a domain-adaptive parameter—plays a central, learnable role in shaping both the functional form of kernels and their operational impact on model estimation, learning dynamics, or structured transformations. This paradigm appears across statistical estimation, structured regression, deep learning, signal processing, and dynamical systems, unifying approaches that adapt or parameterize kernels according to (often learned) scale settings rather than fixing them a priori. Key developments have been documented in robust estimation (Das et al., 2022), kernel regression and learning (&&&1&&&, Li et al., 17 Feb 2025), convolutional architectures (Romero et al., 2021, Chen et al., 2024), manifold learning (Lindenbaum et al., 2017), scale-space and signal analysis (Luxemburg et al., 2023), and infinite-width neural tangent kernels (Sohl-dickstein et al., 2020).

1. Theoretical Motivation and Definitions

Scale-relative kernel parameterizations generalize classical fixed-kernel schemes by representing the effective action range, sensitivity, or structure of the kernel as a function of explicit, often learnable parameters. These parameters may encode bandwidth (as in Gaussian kernels), adaptive shapes (in robust estimation), dilation or anisotropy (in high-dimensional or multi-modal data), architectural support (CNN kernel size), or the choice of atomic agent and ensuing kernel in dynamical systems.

A unifying formulation replaces the fixed kernel $k(x,x')$ by $k(x, x'; \theta, \sigma)$, where $\sigma$ is a (possibly vector- or matrix-valued) scale/bandwidth parameter and $\theta$ includes additional kernel hyperparameters. In the scale-relative setting, $\sigma$ and $\theta$ are not externally fixed but optimized alongside model coefficients, or even treated as domain-adaptive latent variables (Norkin et al., 24 Jan 2025, Li et al., 17 Feb 2025).

2. Adaptive Scale in Robust Kernel Estimation

The robust estimation literature has developed explicit scale-variant losses based on parametric kernel forms. Consider the loss $\phi(r;\alpha,\sigma)$ defined as (Das et al., 2022):

$$\phi(r;\alpha,\sigma) = \begin{cases} \frac{1}{2}(r/\sigma)^2 & \text{if } \alpha = 2 \ \log(1 + \frac{1}{2}(r/\sigma)^2) & \text{if } \alpha = 0 \ 1 - \exp(-\frac{1}{2}(r/\sigma)^2) & \text{if } \alpha \rightarrow -\infty \ \frac{|\alpha-2|}{\alpha} \left[ (1 + |\alpha-2|(r/\sigma)^2)^{\alpha/2} - 1 \right] & \text{otherwise} \end{cases}$$

Here, $\sigma > 0$ serves as the scale (residual dispersion), and $\alpha$ modulates the loss's "shape," directly interpolating between classical least-squares ($\alpha=2$), Cauchy ($\alpha=0$), and Welsch ($\alpha\to-\infty$) penalties.

Optimization employs alternating minimization: (i) update $\alpha$ on a discretized grid given $\sigma$, (ii) update $\sigma$ with fixed $\alpha$, (iii) update the primary model parameters using IRLS with weights $w(r;\alpha,\sigma) = \phi'(r;\alpha,\sigma)/r$. Decoupling scale and shape parameters by preestimating $\sigma$ using a robust median-based statistic further improves stability and outlier resistance (variant SRKO*). This removes manual threshold tuning and allows adaptation to input noise levels (Das et al., 2022).

3. Scale-Relative Kernels in Learning and Regression

In classical kernel regression and SVM, the hypothesis space is traditionally an RKHS determined by a fixed kernel $k_\sigma(x,x')$ with a chosen $\sigma$ (often bandwidth). Scale-relative kernel methods generalize this as follows:

• Each basis kernel at training location $x_i$ is allowed its own scale $\sigma_i$, learning the optimal bandwidth per datum.

$f(x) = \sum_{i=1}^m a_i\, k_{\sigma_i}(x, x_i)$

with an $L_2$-regularized empirical risk objective jointly in $a = (a_1,...,a_m)$ and $\sigma = (\sigma_1,...,\sigma_m)$ (Norkin et al., 24 Jan 2025).

• The underlying matrix $K_{ij}(\sigma)$ involves analytic expressions for the overlap integral between differently-scaled basis functions (for Gaussians, a closed form is given). Alternating optimization is used: solve for $a$ with fixed $\sigma$, then update each $\sigma_i$ (e.g., via gradient search), and repeat.
• This approach creates a much larger hypothesis space than a single-RKHS model, improving fit and expressiveness, particularly in heterogeneous or multi-scale data (Norkin et al., 24 Jan 2025).

A complementary perspective involves parameterizing kernels via a linear transformation $U$ on input data: $K_U(x, x') = K(Ux, Ux')$. Optimization of a coupled objective in both $f$ and $U$ acts as a joint scale detector and feature selector, with $U$ shaping the effective kernel anisotropy and bandwidth along different input dimensions (Li et al., 17 Feb 2025). The solution landscape reveals multiple vacua, corresponding to different latent scales or data substructures.

4. Kernel Parameterization in Deep and Convolutional Architectures

Scale-relative kernel parameterizations underpin recent innovations in convolutional network design:

• FlexConv implements continuous kernels $$\psi(x) = \mathrm{MLP}^\psi(x)\cdot w_\text{gauss}(x)$$; the Gaussian mask parameters $(\mu_X, \sigma_X^2, \ldots)$ are fully differentiable and learned by backpropagation. The result is a kernel that adapts its effective spatial scale during training. Maximum frequency is analytically controlled for alias-free upsampling, and differentiable penalization ensures no violation of the Nyquist bound. This mechanism grants high expressive bandwidth and efficient support adaptation, outperforming classical dilation- or basis-expanded kernels on vision and sequence tasks (Romero et al., 2021).
• PeLK (Parameter-efficient Large Kernel ConvNets) achieves scale adaptivity by configuring parameter sharing over concentric rings: neighborhoods with the same Chebyshev radius share a weight, with adaptive switching from fine-grained to coarse-grained sharing as a function of distance. This reduces parameter complexity from $O(K^2)$ to $O(\log K)$ for a $K \times K$ kernel, enabling extremely large kernels (up to $101\times101$) with dense coverage and minimal parameter overhead (Chen et al., 2024).

5. Manifold Learning and Data-Driven Bandwidth Selection

In manifold learning and classification, the Gaussian RBF kernel scale $\sigma$ is a fundamental parameter that governs the connectivity structure in the induced affinity graph. Scale-relative methods optimize this by aligning the resulting Markov operator's behavior with the intrinsic dimension of the data (manifold alignment) or by maximizing class separation metrics post-embedding. Empirically, strategies such as maximizing the ratio of between- to within-class distances in the diffusion map embedding, maximizing eigengaps for class block-diagonality, or within-class random-walk probabilities have been shown to robustly approximate scales that maximize classification accuracy (Lindenbaum et al., 2017).

6. Scale Adaptation in Infinite-Width Neural Network Kernels

The infinite-width limit of deep neural networks reduces the trainable model to a kernel regression problem governed by the so-called Neural Tangent Kernel (NTK). The standard NTK parameterization loses sensitivity to the ratios of individual layer widths; the "improved standard" (scale-relative) parameterization restores this dependence by introducing a "baseline" width $N^l$ per layer, along with a scaling factor. The resulting infinite-width kernel incorporates layer width as an explicit scale parameter, matching the finite-width network's training dynamics and enabling hyperparameter tuning at the level of per-layer widths (Sohl-dickstein et al., 2020). The improved standard parameterization matches or outperforms NTK schemes in classification performance and preserves fine-grained control over weight and bias update dynamics.

7. Extensions: Multi-Parameter Scale Spaces and Dynamical Systems

Multi-parameter linear scale-spaces for signal analysis are constructed using a maximal family of convolution kernels:

$$K(x;\alpha,\beta,a,p,\rho) = \alpha\,\rho^{p+1}\,\Lambda^e_p(a\rho x) + \beta\,\rho^{p+1}\,\Lambda^o_p(a\rho x)$$

where $\rho$ is bandwidth, $a$ a dilation, $p$ is fractional derivative order, and $(\alpha, \beta)$ weight even/odd branches (Luxemburg et al., 2023). Maximality theorems guarantee no further, nontrivial, scale-invariant kernels exist within the axiomatic system.

In dynamical systems, scale-relative kernel parameterizations systematize the selection and transformation of "atomic" units—cells, firms, regimes—whose interaction, reproduction, and transmission are all encoded in a kernel triple $(\rho_S, w_S, M_S)$, themselves parameterized by scale $S$. Coarse-graining or aggregation over scales is governed by precise "lumpability" conditions, ensuring form-invariance of the dynamics under changes of scale (Farzulla, 10 Jan 2026). This facilitates cross-domain application from biology (cellular mitosis) to political philosophy (legitimacy/friction dynamics).

---

Summary Table: Representative Contexts for Scale-Relative Kernel Parameterization

Domain/Context	Core Scale Parameterization	Key Reference
Robust Estimation	Loss $\phi(r;\alpha,\sigma)$, learnable $\alpha,\sigma$	(Das et al., 2022)
Kernel Regression	Per-datum scale $\sigma_i$, joint $(a,\sigma)$ optimization	(Norkin et al., 24 Jan 2025, Li et al., 17 Feb 2025)
CNNs/ConvNets	Differentiable kernel size/shape, adaptive parameter sharing	(Romero et al., 2021, Chen et al., 2024)
Manifold Learning	Data-driven kernel bandwidth $\sigma$ aligned to embedding geometry	(Lindenbaum et al., 2017)
Infinite-Width DNNs	Layer width as explicit "scale" in kernel recursion	(Sohl-dickstein et al., 2020)
Signal Classification	Multi-parameter (bandwidth/order/shift) scale-spaces	(Luxemburg et al., 2023)
Dynamical Systems	Kernel triple $(\rho_S, w_S, M_S)$, atomic unit as scale param	(Farzulla, 10 Jan 2026)

Scale-relative kernel parameterization defines a nexus between model expressivity, statistical efficiency, data adaptivity, and structural interpretability, making it central in contemporary kernel-based learning, signal analysis, and modeling of complex systems.