--- title: Leverage Score Sampling Overview url: https://www.emergentmind.com/topics/leverage-score-sampling type: topic --- # Leverage Score Sampling Overview Searching arXiv for recent and foundational papers on leverage score sampling to ground the article. Leverage score sampling is a family of importance-sampling methods that select rows, columns, data points, features, or neurons according to quantities derived from projection matrices or their regularized analogues. In its classical matrix form, the statistical leverage score of a row measures how strongly that row participates in the relevant subspace of the data matrix, and leverage-based sampling yields smaller sketches that preserve the geometry of the original problem for least squares and related tasks [2009.09829]. Subsequent work extends this principle to ridge-regularized kernel methods, random features, neural tangent kernels, active learning, distributed optimization, generative modeling, streaming algorithms, and explainable machine learning, with the common goal of biasing computation toward directions that are structurally important rather than uniformly frequent [2103.05238]. ## 1. Classical definition and geometric interpretation For a data matrix \(A \in \mathbb{R}^{n \times d}\), with rows \(a_i^\top\), the statistical leverage scores are defined through the orthogonal projector onto the column space of \(A\). Writing \[ P \;=\; A (A^\top A)^\dagger A^\top \in \mathbb{R}^{n \times n}, \] the \(i\)-th leverage score is \[ \sigma_i(A) \;:=\; P_{ii} \;=\; a_i^\top (A^\top A)^\dagger a_i. \] If \(A = U \Sigma V^\top\) is a thin singular value decomposition, then \(P = UU^\top\) and \[ \sigma_i(A) = \|U_{i,:}\|_2^2. \] These scores lie in \([0,1]\), sum to \(\mathrm{rank}(A)\), and the coherence is \(\|\sigma(A)\|_\infty\) [2009.09829]. This formulation identifies leverage as a measure of alignment with the principal subspace. High-leverage rows contain directions that are poorly represented by other rows; low-leverage rows are near-redundant. The same interpretation appears in matrix and kernel settings: ALEVS defines leverage scores for a symmetric positive semidefinite kernel matrix \(K = U\Sigma U^\top\) using the top \(k\) eigenvectors \(U_1\), with \[ \ell_j = \|(U_1)_{(j)}\|_2^2, \] so a point has high leverage when it strongly participates in the top \(k\) eigendirections of the kernel matrix [1507.04155]. Leverage score sampling converts these scores into a probability distribution. In the matrix case, \[ p_i \;=\; \frac{\sigma_i(A)}{\sum_{j=1}^n \sigma_j(A)} \;=\; \frac{\sigma_i(A)}{\mathrm{rank}(A)}. \] Sampling rows according to \(p_i\) and rescaling each selected row by \(1/\sqrt{mp_i}\) produces a sketch \(\widetilde{A}\) that preferentially retains high-leverage structure while reducing the problem size [2009.09829]. A closely related deterministic variant selects the columns with the largest leverage scores until their cumulative mass exceeds a threshold, and can achieve relative-error guarantees under a power-law decay assumption on leverage scores [1404.1530]. ## 2. Matrix sketching, regression, and deterministic variants Leverage score sampling originated in numerical linear algebra and theoretical computer science, where it underlies randomized algorithms for least squares, low-rank approximation and CUR decompositions, linear programming, semidefinite programming, graph sparsification, maximum matching, max-flow, and random spanning trees [2009.09829]. The common pattern is to replace a large matrix or graph by a sampled and rescaled subset whose spectral or geometric properties approximate the original. For overdetermined least squares, \[ \min_x \|Ax - b\|_2^2, \] one forms a sampling matrix \(S \in \mathbb{R}^{m \times n}\), obtains \(\widetilde{A}=SA\) and \(\widetilde{b}=Sb\), and solves \[ \min_x \|\widetilde{A}x - \widetilde{b}\|_2^2. \] With high probability, leverage-score-based sampling yields a solution \(\hat{x}\) satisfying \[ \|A\hat{x} - b\|_2^2 \le (1+\varepsilon)\min_x \|Ax-b\|_2^2 \] when \(m = \tilde{O}(d/\varepsilon^2)\) or \(\tilde{O}(\mathrm{rank}(A)/\varepsilon^2)\) [2009.09829]. These sketches also preserve quadratic forms and spectral structure through subspace embedding bounds of the form \(A^\top A \approx \widetilde{A}^\top \widetilde{A}\). Deterministic leverage-score selection replaces randomized sampling by thresholding the largest rank-\(k\) leverage scores \(\ell_i^{(k)} = \|[V_k]_{i,:}\|_2^2\). If the selected columns capture all but \(\varepsilon\) of the total leverage, then \[ \|A - C C^\dagger A\|_\xi < (1-\varepsilon)^{-1}\|A-A_k\|_\xi,\qquad \xi\in\{2,\mathrm{F}\}, \] and under power-law decay of the leverage scores, the number of selected columns is polynomial in \(k\) and \(1/\varepsilon\) [1404.1530]. This provides relative-error guarantees in both Frobenius and spectral norm for a purely deterministic column subset selection rule. Ridge leverage scores regularize the classical notion by replacing \(AA^\top\) with \(AA^\top+\lambda_2 I\). For a matrix \(A\in\mathbb{R}^{n\times d}\), the ridge leverage score of column \(i\) is \[ \bar{\tau}_i(A) = a_i^\top (AA^\top+\lambda_2 I)^+ a_i, \qquad \lambda_2 = \frac{1}{k}\|A-A_k\|_F^2. \] A deterministic ridge leverage score sampler selects columns until the leftover ridge leverage mass is below \(\epsilon\), producing a column subset \(C\) that satisfies additive-multiplicative spectral bounds, \((1+{\epsilon})\) error column subset selection, \((1+{\epsilon})\) error projection-cost preservation, and a \((1+\epsilon)\) bound on statistical risk for ridge regression under the prescribed regularization [1803.06010]. This suggests a direct connection between leverage-based sampling and sparse, interpretable feature selection in regularized regression. A separate line of work uses leverage-aware sampling to accelerate iterative solvers rather than merely reduce matrix dimension. For regression \(\min_x \frac12\|Ax-b\|_2^2\), leverage-score sampling combined with proximal point methods and accelerated coordinate descent yields a running time \[ \widetilde{O}\Big(\big(n+\sqrt{d\cdot\kappa_{\text{sum}}}\big)\cdot s\cdot \log\epsilon^{-1}\Big), \] improving over the previous \(\widetilde{O}((n+\sqrt{n\cdot\kappa_{\text{sum}}})\cdot s\cdot\log\epsilon^{-1})\), where \(\kappa_{\text{sum}}=\mathrm{tr}(A^\top A)/\lambda_{\min}(A^\top A)\) and \(s\) is the maximum number of non-zero entries in a row of \(A\) [1711.08426]. In that framework, leverage scores act as a preconditioning device that balances coordinate-wise smoothness constants. ## 3. Ridge leverage scores in kernel methods and random features In kernel ridge regression, with kernel matrix \(K\in\mathbb{R}^{n\times n}\), labels \(Y\in\mathbb{R}^n\), and regularization \(\lambda>0\), one solves \[ \min_\alpha \frac{1}{2}\|Y-K\alpha\|_2^2 + \frac{1}{2}\lambda \alpha^\top K \alpha, \] or equivalently predicts at a new point \(z\) via \[ f(z)=K(z,X)^\top (K+\lambda I_n)^{-1}Y. \] The cubic cost of forming \((K+\lambda I)^{-1}\) motivates random feature and Nyström approximations [2009.09829]. For kernels admitting a random feature representation \[ K(x,z)=\mathbb{E}_{w\sim p}[\phi(x,w)^\top \phi(z,w)], \] ridge leverage scores quantify the importance of random features relative to the regularized kernel matrix. Defining \[ \Phi(w)= \begin{bmatrix} \phi(x_1,w)^\top\\ \vdots\\ \phi(x_n,w)^\top \end{bmatrix}, \] the ridge leverage function is \[ q_\lambda(w)=p(w)\,\mathrm{Tr}\!\left[\Phi(w)^\top (K+\lambda I_n)^{-1}\Phi(w)\right]. \] Its total mass \[ s_\lambda(K)=\int q_\lambda(w)\,dw=\mathrm{Tr}[(K+\lambda I_n)^{-1}K] \] is the statistical dimension of \(K\) at regularization \(\lambda\) [2009.09829]. Sampling random features according to \[ q(w)=\frac{q_\lambda(w)}{s_\lambda(K)} \] and reweighting by \(\sqrt{p(w)/q(w)}\) yields a feature matrix \(\Psi\) satisfying the spectral approximation \[ (1-\varepsilon)(K+\lambda I_n)\preceq \Psi\Psi^\top+\lambda I_n \preceq (1+\varepsilon)(K+\lambda I_n) \] with \(m=\tilde{O}(s_\lambda(K)/\varepsilon^2)\) samples, rather than a feature count tied directly to \(n\) [2009.09829]. This extends earlier Fourier-feature results to any kernel representable as an expectation over finite-dimensional features, including random Fourier features, dot-product kernels, and neural tangent kernels [2009.09829]. A separate line investigates approximation of the diagonal entries of \[ K_n(K_n+n\lambda I_n)^{-1}, \] which are the statistical leverage scores for KRR in Nyström methods. For stationary kernels, an analytic approximation uses the input density \(p(x)\) and the spectral density \(m(s)\) of the kernel. The central approximation is \[ \widetilde{K}_\lambda(x_i,x_i)=\int_{\mathbb{R}^d}\frac{1}{p(x_i)+\lambda/m(s)}\,ds, \] which approximates the rescaled leverage score \(G_\lambda(x_i,x_i)\) in large samples [2103.05238]. This gives a linear-time, modulo poly-log terms, algorithm for stationary-kernel-based KRR, with asymptotic relative error guarantees and Nyström risk guarantees matching full KRR up to constants [2103.05238]. Conceptually, it shows that leverage-score non-uniformity is governed by local input density and kernel smoothness. Random-feature leverage scores can also be approximated empirically. Given \(M_0\) candidate features and \(N_0\) data points, the empirical matrix \[ Q = \Phi_{M_0,N_0}\Phi_{M_0,N_0}^\top\big(\Phi_{M_0,N_0}\Phi_{M_0,N_0}^\top + \lambda I\big)^{-1} \] induces a discrete distribution \[ \hat{q}(\tilde{\omega}_i)=\frac{[Q]_{ii}}{\mathrm{Tr}(Q)}. \] Sampling features from this empirical leverage distribution yields improved out-of-sample performance over vanilla Monte Carlo sampling, with a generalization bound that separates a kernel approximation term from a spectral term involving the smallest eigenvalue of the feature Gram matrix \(G\) [1903.08329]. This suggests a trade-off: increasing the initial feature pool improves approximation of the target kernel but can worsen conditioning in feature space. ## 4. Neural tangent kernels and neural-network initialization Leverage score sampling has been extended to deep learning theory through the neural tangent kernel (NTK). For a two-layer ReLU network \[ f_{\mathrm{nn}}(W,a,x)=\frac{1}{\sqrt{m}}\sum_{r=1}^m a_r \sigma(w_r^\top x), \] with fixed second-layer weights \(a_r\in\{-1,+1\}\), the NTK with respect to \(W\) can be written for two-layer ReLU as \[ \phi(x,w)=x\,\sigma'(w^\top x)\in\mathbb{R}^d,\qquad K_{\mathrm{ntk}}(x,z)=\mathbb{E}_{w\sim\mathcal{N}(0,I_d)}[\phi(x,w)^\top\phi(z,w)]. \] Thus the NTK is a kernel of the same random-feature form addressed by generalized leverage score sampling [2009.09829]. From this viewpoint, finite-width network initialization selects a finite set of random features that approximates the continuous NTK. The initial kernel \(H(0)\) is a Monte Carlo approximation of the continuous NTK matrix \(H^{\mathrm{cts}}\). Replacing Gaussian initialization by leverage-score-based initialization samples first-layer weights from \[ q(w)=\frac{q_\lambda(w)}{s_\lambda(H^{\mathrm{cts}})}, \qquad q_\lambda(w)=p(w)\,\mathrm{Tr}\!\left[\Phi(w)^\top (H^{\mathrm{cts}}+\lambda I_n)^{-1}\Phi(w)\right], \] and reweights neurons by \(\sqrt{p(w)/q(w)}\) [2009.09829]. Under this initialization, \[ (1-\varepsilon)\bigl(H^{\mathrm{cts}}+\lambda I_n\bigr) \preceq H(0)+\lambda I_n \preceq (1+\varepsilon)\bigl(H^{\mathrm{cts}}+\lambda I_n\bigr) \] with \(m=\tilde{O}(s_\lambda(H^{\mathrm{cts}})/\varepsilon^2)\) neurons [2009.09829]. The same work establishes equivalence between \(\ell_2\)-regularized training of a two-layer ReLU network and NTK kernel ridge regression under both classical Gaussian initialization and leverage-score-based initialization [2009.09829]. For Gaussian initialization, the neural network predictions converge to the NTK KRR solution in the infinite-width limit, on both training data and test points, under appropriate width and gradient-flow assumptions. With leverage-score-based initialization, an analogous training-data equivalence theorem holds, with the theoretical advantage that the required width can scale with the statistical dimension \(s_\lambda(H^{\mathrm{cts}})\) rather than a worst-case dimension [2009.09829]. This suggests a reinterpretation of initialization in the NTK regime: initialization is not merely a random seed but a feature-sampling mechanism. Under that interpretation, leverage score sampling identifies neurons that are important for approximating the regularized NTK, thereby reducing the width needed to emulate NTK ridge-regression dynamics [2009.09829]. ## 5. Sampling data points, coalitions, and partitions Leverage score sampling has also been adapted to settings where the sampled objects are data points, active-learning queries, Shapley coalitions, or data partitions rather than matrix rows in a classical least-squares sketch. In active learning, ALEVS computes leverage scores on class-specific kernel matrices. At each round, a classifier \(h_t\) partitions data into predicted-positive and predicted-negative subsets; for each class-specific kernel matrix, the top \(k\) eigenvectors are computed, and the leverage score of point \(j\) is \[ \ell_j = \|(U_1)_{(j)}\|_2^2. \] The query rule is deterministic: \[ x_q = \arg\max_{x_j\in\mathcal{D}_u^t}\ell_j. \] On ten binary classification tasks, ALEVS outperforms the baselines on most datasets for a substantial portion of the query range, particularly in early iterations, while class-wise leverage scores outperform leverage scores computed on all data at once [1507.04155]. The method is structure-based rather than margin-based: it queries points that are influential in the top eigenspaces of their predicted class manifolds. In explainable machine learning, Leverage SHAP formulates Shapley value estimation as a linearly constrained weighted least-squares problem and applies leverage score sampling to the projected design matrix arising in Kernel SHAP. After reducing the constraint, the projected matrix \(\mathbf{A}\) has analytically tractable leverage scores: \[ \ell_{\mathbf{z}} = \binom{n}{\|\mathbf{z}\|_1}^{-1}, \] for any coalition indicator \(\mathbf{z}\) with \(0<\|\mathbf{z}\|_12\) faces lower bounds [2406.00339]. Even with these limitations, a unifying pattern is evident. Leverage scores, whether classical, ridge-regularized, empirical, continuous, or \(\ell_p\)-generalized, measure how much a sampled object contributes to the effective subspace or function class relevant to the task. Sampling according to those scores yields approximation guarantees that depend on effective rank, statistical dimension, or leverage decay rather than on ambient problem size. In this sense, leverage score sampling functions as a common principle linking matrix sketching, kernel approximation, neural tangent theory, active data acquisition, robust distributed learning, mode-balanced generative modeling, continual learning, and dynamic-stream coresets [2009.09829]. Source: https://www.emergentmind.com/topics/leverage-score-sampling