---
title: Deterministic Ridge Leverage Score Sampling
url: https://www.emergentmind.com/topics/deterministic-ridge-leverage-score-sampling
type: topic
---

# Deterministic Ridge Leverage Score Sampling

Deterministic Ridge Leverage Score Sampling is a data-dependent, subset selection technique for linear algebraic and statistical tasks that balances the goals of low-rank approximation and regularization. Unlike randomized variants, the deterministic approach provides interpretability, repeatability, and strong worst-case guarantees, making it particularly attractive for applications in scientific data analysis, regression, matrix approximation, and feature selection.

## 1. Formal Definition and Structural Properties

Given a matrix $A\in\mathbb R^{n\times d}$ and a regularization parameter $\lambda>0$, the ridge leverage score for the $i$th column $a_i$ is defined as
\[
\bar\tau_i(A) = a_i^T (A A^T + \lambda I_n)^+ a_i,
\]
where $(\cdot)^+$ denotes the Moore–Penrose pseudoinverse. In terms of the thin SVD $A=U\Sigma V^T$, with singular values $\sigma_1\ge\cdots\ge\sigma_r>0$ and right singular vectors $V$, this score becomes
\[
\bar\tau_i(A) = \sum_{j=1}^r \frac{\sigma_j^2}{\sigma_j^2+\lambda} V_{ij}^2.
\]
Ridge leverage scores interpolate between subspace leverage scores (unregularized) and their regularized counterparts, smoothly down-weighting directions associated with small singular values. This stabilization makes ridge leverage scores adaptive for both regularized regression and low-rank matrix approximations, simultaneously capturing the informative structure and dampening the effect of noise or degeneracy [1803.06010].

## 2. Deterministic Sampling Algorithms

Deterministic ridge leverage score sampling proceeds by ranking columns according to their ridge leverage scores and selecting the subset with maximal cumulative score. The canonical procedure is:

1. Compute all ridge leverage scores $\bar\tau_i = \bar\tau_i(A)$.
2. Sort the column indices so that $\bar\tau_{\pi(1)} \geq \bar\tau_{\pi(2)} \geq \cdots$.
3. Initialize an empty selection set $\Theta$ and a partial sum $s=0$.
4. Iteratively add $\pi(j)$ to $\Theta$ and update $s \gets s + \bar\tau_{\pi(j)}$ until $s \geq \bar t - \epsilon$, with $\bar t = \sum_{i=1}^d \bar\tau_i(A) \leq 2k$ for target rank $k$ and tolerance $\epsilon$.
5. If $|\Theta| < k$, continue selecting the largest remaining scores to ensure the subset has at least $k$ columns.
6. Construct the sampling matrix $S$ and resulting sketch $C = A S$.

This routine yields an unweighted, deterministic subset of columns, with computational complexity $O(\min\{n,d\}nd)$ for score computation and $O(d\log d)$ for sorting [1803.06010]. For kernelized and feature-map settings, the deterministic variant is implemented by sorting data points by (kernel) ridge leverage scores and taking the top set [2104.02373, 2103.05238].

## 3. Theoretical Guarantees

Deterministic ridge leverage score sampling provides strong spectral and prediction risk bounds:

- **Additive-multiplicative spectral bound**: For $C$ the selected column subset,
  \[
  (1-\epsilon) AA^T - \frac{\epsilon}{k} \|A_\backslash k\|_F^2 I_n \preceq CC^T \preceq AA^T,
  \]
  where $A_{\backslash k} = A - A_k$ and $A_k$ is the best rank-$k$ approximation [1803.06010].

- **Projection-cost preservation**: For any orthogonal projector $X$ of rank $k$,
  \[
  (1-\alpha\epsilon)\|A-XA\|_F^2 \leq \|C-XC\|_F^2 \leq \|A-XA\|_F^2,
  \]
  with constant $\alpha = 2(2+\sqrt{2})$, preserving objectives such as low-rank approximation and $k$-means cost up to $O(\epsilon)$ factors [1803.06010].

- **Risk inflation in regression**: For ridge regression on the sketch $C$ versus the full data, the statistical risk $\mathcal R(\hat y_C) \leq (1+\beta\epsilon)\mathcal R(\hat y_A)$, where $\beta = \frac{2\alpha(-1+2\alpha + 3\alpha^2)}{(1-\alpha)^2}$ [1803.06010].

- **Sample complexity**: When ridge leverage scores decay as a power law $\bar\tau_{\pi(i)}(A) = L i^{-a}$, $a>1$, the deterministic subset size matches or exceeds the efficiency of randomized sampling, i.e., $m = O(k)$ for $a \geq 2$ [1803.06010].

## 4. Applications in Regression, Feature Selection, and Kernel Methods

Deterministic ridge leverage sampling is applicable and effective in a range of settings:

- **Ridge regression and classification**: Using a sketch $C$ formed from deterministic RLS sampling, regression coefficients corresponding to non-selected columns are forced to zero, resulting in built-in feature selection with provable risk control. Selecting with respect to the regularized leverage scores yields a risk bound competitive with alternatives such as elastic net [1803.06010].
  
- **Design and data valuation**: Ridge leverage scores measure marginal gain under A- and D-optimality criteria, and when normalized can serve as Shapley-like data value surrogates [2511.02100].

- **Active learning and data subset selection**: In deterministic active learning, acquiring samples with the highest ridge leverage scores yields models whose test accuracy closely matches or exceeds classical uncertainty-based and geometric selection strategies [2511.02100].

- **Nyström approximations in kernel ridge regression**: Deterministic selection of kernel landmarks by (approximate) ridge leverage yields Nyström approximations with the same in-sample risk as full KRR and near-linear computational complexity, especially in cases with stationary kernels [2103.05238].

## 5. Connections to Spectral Sparsification and Feature Selection

Deterministic feature selection via RLS connects closely to single-set spectral sparsification (BSS), wherein a greedy procedure selects rows with weights to spectrally approximate the Gram matrix $X^T X$ up to $(1\pm\epsilon)$, ensuring that the risk of ridge regression on the reduced feature space inflates by at most $1+O(\epsilon)$ relative to the original [1506.05173]. Both methods yield deterministic, interpretable subset selectors with explicit sample complexity guarantees—typically $O(n/\epsilon^2)$ for rank $n$ and error $\epsilon$.

## 6. Practical Considerations and Empirical Observations

- **Parameter selection**: Good practice suggests setting $k$ via spectrum elbowing, $\lambda = \|A - A_k\|_F^2 / k$, and $\epsilon \in [0.01, 0.1]$ to balance sketch size and error [1803.06010].
- **Computational cost**: Direct SVD or Cholesky is $O(nd^2)$ for dense problems, with further acceleration possible via randomized or approximate techniques for large $n,d$ [2104.02373, 2103.05238].
- **Empirical efficacy**: In applications such as multi-omic cancer data and deep-learning model training, deterministic RLS sampling yields compact, interpretable data sketches (small $m\ll d$) with negligible loss in predictive accuracy, and in GAN training, empirically corrects mode drop and improves rare-mode coverage [1803.06010, 2104.02373].
- **Feature and landmark selection**: Deterministic top-$k$ RLS selection matches or outperforms standard baselines (uncertainty, margin, entropy) in data-efficient regimes, particularly for high-dimensional or overparameterized models [2511.02100].

## 7. Extensions and Kernel Generalizations

Kernelized deterministic ridge leverage score sampling extends the theory and algorithmic guarantees to non-linear settings. In those contexts, the scores are computed in dual (kernel) or primal (feature) form and can leverage the structure of stationary kernels for efficient approximation. A one-dimensional integral formula, based on input density and kernel spectral density, enables linear-time computation (up to poly-log) of approximate scores. Sorting and selecting the top estimated scores yields a deterministic Nyström approximation matching the statistical risk of full-data solutions under regularity assumptions [2103.05238]. Tuning of regularization, landmark count, and kernel hyperparameters directly impact both efficiency and downstream generalization.

---

Key references: [1803.06010], [2104.02373], [2511.02100], [1506.05173], [2103.05238].

Source: https://www.emergentmind.com/topics/deterministic-ridge-leverage-score-sampling