Kernel Least Squares (KLS) Framework

• Kernel Least Squares (KLS) is a framework for nonlinear regression and classification that maps input data into a reproducing kernel Hilbert space (RKHS) and solves a regularized least-squares problem.
• KLS encompasses various algorithmic variants such as kernel ridge regression, recursive methods for online learning, and federated approaches that enhance scalability and privacy.
• Computational accelerations like random Fourier features and Nyström approximations reduce the inherent computational and memory overhead while maintaining high accuracy.

Kernel Least Squares (KLS) is a foundational framework in machine learning and signal processing, enabling nonlinear regression and classification by mapping input data into a reproducing kernel Hilbert space (RKHS) and solving a regularized least-squares problem. The KLS paradigm subsumes a broad class of algorithms, including kernel ridge regression (KRLS), least-squares support vector machines (LS-SVM), and recursive kernel methods. It is central in both batch and online learning, in exact and approximate form, and underpins robust, scalable, and privacy-enhanced modern algorithms.

1. Core Formulation and Solution Structure

Given data $\{(x_i, y_i)\}_{i=1}^n$ with $x_i \in \mathbb{R}^d$, $y_i \in \mathbb{R}$, and a positive-definite kernel $k(\cdot, \cdot)$ inducing an RKHS $\mathcal{H}$, the KLS objective is

$$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$

where $\lambda > 0$ controls regularization. By the representer theorem, the solution admits an expansion $f(x) = \sum_{j=1}^n \alpha_j k(x_j, x)$. Defining $K \in \mathbb{R}^{n \times n}$ by $K_{ij} = k(x_i, x_j)$ and $x_i \in \mathbb{R}^d$0, one obtains the linear system

$x_i \in \mathbb{R}^d$1

with closed-form solution $x_i \in \mathbb{R}^d$2 (Damiani et al., 2024, Chang et al., 2022). Predictions at new points $x_i \in \mathbb{R}^d$3 utilize $x_i \in \mathbb{R}^d$4 for all $x_i \in \mathbb{R}^d$5.

2. Algorithmic Variants: Recursive, Robust, and Federated KLS

Recursive and Online KLS

Recursive kernel least-squares algorithms, such as KRLS and its ALD-sparsified form, maintain a dynamic dictionary of kernel centers to allow efficient online adaptation. At each step, the approximate linear dependency (ALD) criterion controls model growth by testing representability of new samples, thus bounding computational cost and memory (Zhao, 2015). The recursive update equations employ either block-matrix inversion or rank-one modification, yielding $x_i \in \mathbb{R}^d$6 per-step complexity for dictionary size $x_i \in \mathbb{R}^d$7.

Robust Kernel Least Squares

Sensitivity to noise and outliers is mitigated by replacing the $x_i \in \mathbb{R}^d$8 loss with generalized alternatives, such as the $x_i \in \mathbb{R}^d$9-loss, which interpolates between $y_i \in \mathbb{R}$0 and $y_i \in \mathbb{R}$1 behavior and guarantees bounded influence functions. Robust KLS is solved via iteratively reweighted least squares (IRLS), where each iteration solves a weighted KLS system with appropriately updated weight functions derived from the loss's derivative (Dong et al., 2019). This approach delivers improved breakdown properties and noise resilience.

Federated and Hybrid KLS

Federated settings, relevant in privacy-preserving scenarios (e.g., medical and omics data), require distributed computation without sharing raw data. Hybrid federated KLS decomposes the Gram matrix construction by utilizing kernels with multiplicative separability, enabling each data silo (hospital, omics center) to compute partial Gram blocks locally. A random Nyström approximation is employed for scalability, constructing $y_i \in \mathbb{R}$2 from $y_i \in \mathbb{R}$3 landmarks shared via random seeds. Two algorithmic variants are outlined: a naïve federated CG (single communication round) and a secure iterative CG (per-iteration secure aggregation of matrix-vector products with label masking and privacy defenses) (Damiani et al., 2024). Empirically, these methods match centralized accuracy (within $y_i \in \mathbb{R}$4), with $y_i \in \mathbb{R}$5 training overhead and robust privacy guarantees.

3. Computational Acceleration and Large-Scale Approximations

Conventional (batch) KLS faces formidable $y_i \in \mathbb{R}$6 time and $y_i \in \mathbb{R}$7 memory requirements for Gram matrix operations. Several approximation strategies enable KLS to scale:

• Random Fourier Features (RFF): For shift-invariant kernels, RFF approximates $y_i \in \mathbb{R}$8 by an explicit, finite-dimensional feature map $y_i \in \mathbb{R}$9, facilitating "linear" least-squares in $k(\cdot, \cdot)$0 with fixed $k(\cdot, \cdot)$1. Both online and batch KLS admit efficient RFF surrogates with provable convergence and error floors saturating as $k(\cdot, \cdot)$2 increases (Bouboulis et al., 2016).
• Nyström Approximation and Randomized Block Methods: The Nyström method samples $k(\cdot, \cdot)$3 columns/rows of $k(\cdot, \cdot)$4, constructing a rank-$k(\cdot, \cdot)$5 low-rank approximation, substantially reducing computational requirements. Randomized block Kaczmarz and Matching Pursuit iterate over small blocks of the full system, providing controlled accuracy-memory tradeoffs suitable for datasets with $k(\cdot, \cdot)$6 or beyond (Andrecut, 2017, Chang et al., 2022).
• Deep/Hierarchic KLS: High-dimensional and spatiotemporal data are efficiently handled by reorganizing KLS into a hierarchy of layers, each modeling one input dimension and propagating weights. This reduces the cost from $k(\cdot, \cdot)$7 to $k(\cdot, \cdot)$8, with substantial empirical gains in both accuracy and speed for multidimensional, grid-structured problems (Mohamadipanah et al., 2017).

4. Extensions: Classification, GLMs, Additive Structure, and Robust Criteria

KLS generalizes to classification via multiclass expansions (one-vs-all or direct multiclass system), e.g., least-squares SVMs or kernel classifiers with representative selection using K-means centroids, enabling reductions in both computation and memory ($k(\cdot, \cdot)$9 for $\mathcal{H}$0) while preserving accuracy on large datasets such as MNIST ($\mathcal{H}$1 test error for $\mathcal{H}$2 representatives) (Andrecut, 2020, Andrecut, 2017).

Generalized kernel regularized least squares (gKRLS) further extends KLS to mixed-effects models, combining penalized kernel terms with random and fixed effects, or accommodating non-Gaussian outcomes via generalized linear modeling (GLM) machinery with penalized IRLS solvers and REML smoothing selection (Chang et al., 2022).

Robustification and reduction of computational overhead in time-series filtering is achieved by integrating information-theoretic loss functions (e.g., minimum error entropy, generalized MEE), quantized via codebook centroids to accelerate kernel sum computations (from $\mathcal{H}$3 to $\mathcal{H}$4, $\mathcal{H}$5), while maintaining mean and mean-square-error convergence properties (He et al., 2023).

5. Theory: Convergence, Robustness, and Regularization

Convergence

For the $\mathcal{H}$6-convergence of kernel collocation methods (KLS in the context of PDEs), rigorous error bounds are established under smoothness and denseness conditions on the trial space, with optimal rates $\mathcal{H}$7 for kernels reproducing $\mathcal{H}$8 (Cheung et al., 2018). In recursive and randomized schemes, convergence to the unique regularized minimizer (or within the approximation error) is established under mild spectral and step-size conditions (Zhao, 2015, Bouboulis et al., 2016, Andrecut, 2017, He et al., 2023).

Robustness

Use of bounded-gradient loss functions (e.g., $\mathcal{H}$9) and entropy criteria result in bounded influence functions, reducing estimator sensitivity to contamination or outliers (Dong et al., 2019, He et al., 2023). Empirical results demonstrate substantial reductions in RMSE/MAE under adversarial noise.

Regularization

The quadratic penalty $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$0 stabilizes the solution, controls model complexity, and ensures invertibility for ill-conditioned (or nearly colinear) kernel matrices (Chang et al., 2022, Zhao, 2015). Optimal regularization is tractable via cross-validation or marginal likelihood (REML) (Chang et al., 2022).

6. Security and Privacy in Modern KLS Applications

In privacy-sensitive domains, federated KLS integrates multiple defense mechanisms:

• Random Nyström landmarks: prevent any client from reconstructing full input data.
• Masked gradients/labels: ensure partial products reveal no raw information; the server or aggregator can remove synchronized noise.
• Secure aggregation (secret sharing, homomorphic encryption): obfuscate partial products so that intercepted information remains unusable.
• Differential privacy (Gaussian noise addition, gradient clipping): optionally provide formal privacy guarantees.
• Randomized RBF widths: foil Euclidean distance matrix (EDM) reconstruction attacks by adversaries (Damiani et al., 2024).

Empirical tests with state-of-the-art EDM-completion algorithms show relative errors $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$1 in reconstructing $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$2 from partial distances $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$3 for typical $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$4 landmarks, which slowly decays for larger $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$5. Random widths remove EDM structure, providing further protection.

7. Practical Guidelines and Applications

Effective deployment of KLS and its variants involves:

• Kernel selection and tuning: RBF kernels with bandwidth set to feature dimension or empirically tuned; use higher-order Sobolev kernels for PDEs.
• Regularization parameter: Optimize $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$6 by grid search, leave-one-out, or REML.
• Approximation/sparsity controls: Nyström rank $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$7, RFF dimension $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$8, ALD threshold $$\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n \left( y_i - f(x_i) \right)^2 + \lambda \|f\|_{\mathcal{H}}^2,$$9, and IRLS iterations.
• Hierarchic/deep structure: Preferred for multidimensional grid problems.
• Privacy controls: For federated/hybrid settings, align with threat models and available infrastructure (e.g., secure multi-party computation).

Typical applications include nonlinear regression, classification (multiclass, large-scale), system identification, time-series prediction, PDE solution via collocation, and privacy-sensitive distributed modeling. KLS provides a unifying methodological backbone for contemporary nonlinear and nonparametric statistical learning (Damiani et al., 2024, Chang et al., 2022, Andrecut, 2017, Bouboulis et al., 2016, Zhao, 2015, Andrecut, 2020, He et al., 2023, Cheung et al., 2018, Mohamadipanah et al., 2017, Dong et al., 2019).