Proxy/Low-Rank Score Approximation

• Proxy/Low-Rank Score Approximation is a set of methods that approximate complex, high-dimensional score functions using structured, compact representations for efficient inference and optimization.
• It employs techniques such as projection onto low-rank families, score-matching, contour-integral expansions, and randomized sampling to reduce computation and storage costs.
• These methods enable significant reductions in computational complexity from quadratic to linear or sublinear scales, benefiting applications in variational inference, kernel approximation, and large-scale regression.

Proxy and low-rank score approximation refer to a family of algorithmic and analytic methods that approximate complex, high-dimensional score functions—such as leverage, regression, KL divergence, or log-likelihood gradients—using compact, typically low-rank representations. These proxies serve both as computational surrogates and as statistical regularizers, enabling efficient estimation, inference, and optimization in high-dimensional statistical, machine-learning, and signal-processing pipelines. The unifying feature is the exploitation of algebraic, geometric, or analytic structure to reduce computational and sample complexity, while retaining provable guarantees on approximation quality.

1. Analytical and Algorithmic Foundations

Proxy/low-rank score approximation arises in several contexts, including variational inference (Modi et al., 2024), randomized numerical linear algebra [(Pan et al., 2019); (Cohen et al., 2015); (Bhojanapalli et al., 2014)], analytic kernel and potential theory (Ye et al., 2019, Lepilov et al., 22 May 2026), high-dimensional regression (Jääsaari et al., 2024), and structured comparison/ranking (Li et al., 28 May 2026). The central idea is to reduce the dimensionality or complexity of a score matrix or operator (gradient, inner-product, likelihood, etc.)—often of size $d \times d$ or $n \times n$—to an efficiently parameterizable family, such as diagonal-plus-low-rank, factorized, or contour-integral expansions.

Common schemes include:

• Projection to low-rank or structured families: Constraining covariance (e.g., in Bayesian VI) or regression matrices (e.g., in ANN search) to forms such as $\Psi + \Lambda\Lambda^\top$, $AB$, or analytical bases.
• Score-matching and dimension-reduction: Matching score functions (gradients of log-densities, regression outputs) in restricted subspaces, often leveraging SVDs or tailored sampling.
• Analytical proxy expansions: Contour integration or complex-analytic constructions transform a kernel or operator into low-rank, rapidly convergent series (Ye et al., 2019, Lepilov et al., 22 May 2026).
• Sampling-based score proxies: Use of leverage or ridge-leverage scores, or information-theoretically justified sampling, to condense the computational burden of estimating the impact or representativeness of data/features [(Cohen et al., 2015); (Bhojanapalli et al., 2014); (Pan et al., 2019)].

2. Low-Rank Score Parameterization in Variational Inference

In high-dimensional black-box variational inference, direct estimation of a full-rank covariance matrix in a Gaussian approximation is computationally prohibitive due to $O(d^2)$ storage and runtime. The patch-augmented Batch-and-Match (pBaM) framework (Modi et al., 2024) integrates the following steps:

• Score-matching via BaM: Instead of classical reverse KL minimization via stochastic gradient descent, BaM minimizes a covariance-metric weighted “score divergence” between the gradients of the log density of the target and the variational posterior. Proximal updates have closed-form for Gaussian families.
• Low-rank patching: After an unconstrained covariance update, project $\Sigma_{full}$ onto the diagonal-plus-low-rank family $\Psi+\Lambda\Lambda^\top$ by solving a KL minimization between multivariate Gaussians. This reduces to the infinite-data-limit EM for factor analysis, with each EM step decreasing KL. One obtains tractable $O(dr)$ memory and $O(d)$ per-iteration computational cost for $r\ll d$.
• Empirical performance: On both synthetic and real-world problems, pBaM achieves near-full-rank accuracy at orders-of-magnitude reduced time and space, outperforming classical ADVI-LR and diagonal schemes.

3. Sampling and Sketching for Proxy Leverage Score Approximation

In randomized numerical linear algebra, low-rank proxies for leverage scores, regression scores, or spectral scores are constructed via sketching and sampling approximations (Pan et al., 2019, Cohen et al., 2015, Fahrbach et al., 2021):

• Leverage/ridge-leverage scores: Fundamental to importance sampling (e.g., in CUR decompositions, kernel methods), the leverage score proxies—computed via subspace sketches or via proxy integrals—approximate the influence of rows or columns for down-sampling and subspace embedding.
• Recursive/streaming algorithms: Techniques like recursive halving with ridge leverage scores enable optimal low-rank approximations in input sparsity time, supporting streaming, distributed, and merge-and-reduce regimes with robust additive/multiplicative error guarantees (Cohen et al., 2015).
• Tensor and kernel extension: Proxy scores extend to high-order tensors (e.g., core updates in Tucker ALS), where Kronecker factor leverage structures enable efficient sampling without forming the full design (Fahrbach et al., 2021).

Table: Score Proxy Types and Their Context

Method/Class	Proxy Structure	Application Domain
Diag + low-rank projection	$n \times n$0	BBVI, Gaussian VI (Modi et al., 2024)
Ridge leverage score	$n \times n$1	LRA, CUR, kernel approx. (Cohen et al., 2015)
Analytical proxy points	Adaptive contour points	Kernel compression (Ye et al., 2019, Lepilov et al., 22 May 2026)
Reduced-rank regression	$n \times n$2 with $n \times n$3	ANN, inner-product approx. (Jääsaari et al., 2024)
Low-rank matrix/tensor	SVD/truncated decomposition	LLM ranking, matrix/tensor LRA (Li et al., 28 May 2026, Fahrbach et al., 2021)

4. Analytical Proxy Point Methods for Kernel Matrices

For analytic kernel matrices generated from well-separated sets, the proxy point method constructs explicit low-rank factorizations by contour-integral representations (Ye et al., 2019, Lepilov et al., 22 May 2026):

• Contour integral expansion: For a kernel $n \times n$4 analytic in $n \times n$5 (or $n \times n$6), an $n \times n$7-point trapezoidal rule on a proxy contour yields a sum $n \times n$8, separating dependencies on $n \times n$9 and $\Psi + \Lambda\Lambda^\top$0 and yielding a rank $\Psi + \Lambda\Lambda^\top$1 factorization.
• Rigorous error bounds: Entrywise and normwise errors decay exponentially in $\Psi + \Lambda\Lambda^\top$2 with constants determined by the analytic domains and separation geometry. One obtains explicit conditions for numerical rank as a function of separation and desired accuracy.
• Hybrid compression: Analytical proxy basis matrices can be further compressed via RRQR or skeletonization, yielding efficient low-rank representations for kernel and Toeplitz matrices and enabling sublinear-time HSS constructions and fast leverage-score computations.

5. Low-Rank Proxies in Regression and Inner-Product Approximation

In high-dimensional multivariate regression, as in clustering-based approximate nearest-neighbor (ANN) search, the use of reduced-rank regression (RRR) provides a proxy for the score matrix mapping queries to candidate inner products (Jääsaari et al., 2024):

• Formulation: Given regression targets $\Psi + \Lambda\Lambda^\top$3 for queries $\Psi + \Lambda\Lambda^\top$4 and cluster data $\Psi + \Lambda\Lambda^\top$5, the constraint $\Psi + \Lambda\Lambda^\top$6 leads to a factorized $\Psi + \Lambda\Lambda^\top$7 where $\Psi + \Lambda\Lambda^\top$8.
• Algorithmic realization: Closed-form solutions via SVD yield minimum-error proxies to the OLS score operator, and batched compressed evaluation reduces time from $\Psi + \Lambda\Lambda^\top$9 to $AB$0 per cluster.
• Empirical implications: RRR-based proxies outperform product quantization in both recall and query-per-second at fixed memory budgets for $AB$1, and permit efficient, quantizable representations suitable for disk/GPU/distributed infrastructures.

6. Score Proxies for Discrete and Non-Euclidean Objectives

In metrics such as $AB$2 (count of entrywise disagreements), which are prominent in binary matrix/tensor approximation, clustering, and discrete data analysis, proxy-based schemes reduce evaluation cost (Fomin et al., 2018, Bringmann et al., 2017):

• Proxy via randomized sampling: Sampling lemmas justify that for binary clustering and GF(2)-rank approximation, clustering centers/proxies chosen by weighted sampling approximately minimize the objective with high probability.
• Proxy function $AB$3: For general candidate $AB$4, a proxy score $AB$5—the cost to the nearest sampled cluster centers—approximates the true objective within $AB$6, yielding near-linear-time meta-heuristics.
• Algorithmic and computational guarantees: Randomized PTAS and deterministic PTAS exploit these proxies for efficient low-rank approximation under non-metric losses, with attention to sublinear sample complexity and optimality bounds.

7. Low-Rank Proxies in Structured and Sparse-Data Inference

In latent variable models and structured inference (e.g. task-by-model evaluation from pairwise LLM comparisons (Li et al., 28 May 2026)), low-rank proxies support both efficient estimation and uncertainty quantification:

• Convex-initialized alternating minimization: Nuclear-norm penalized convex initializers, projected to rank-$AB$7, serve as a score-proxy for initialization before sup-norm–accurate alternating refinement, providing uniform entrywise accuracy over combinatorially incomplete data.
• Debiased one-step estimators: Linear functionals of the low-rank proxy support semiparametrically efficient confidence intervals and simultaneous ranking statements over large index sets.
• Empirical effects: Low-rank regularization delivers sample-efficiency gains and tighter high-dimensional ranking certificates in regimes with severe data sparsity relative to ambient problem dimensionality.

8. Theoretical Guarantees, Tradeoffs, and Limitations

Proxy/low-rank score approximation methods are supported by a range of theoretical results:

• Spectral and Frobenius error bounds: Most deterministic and randomized sketching and low-rank approximation schemes guarantee that the proxy achieves (multiplicative or additive) bounds relative to the best rank-$AB$8 solutions, often matching or beating Johnson–Lindenstrauss or random projection methods in complexity [(Cohen et al., 2015); (Pan et al., 2019); (Bhojanapalli et al., 2014)].
• Exponential convergence rates: Analytical proxy point expansions for kernel matrices achieve exponential error decay and logarithmic numerical rank scaling with target accuracy (Ye et al., 2019, Lepilov et al., 22 May 2026).
• Computational efficiency: In high dimensions, reduction from $AB$9 or $O(d^2)$0 storage/cost to $O(d^2)$1 or sublinear in $O(d^2)$2 is tractable for $O(d^2)$3 and ubiquitous in streaming, distributed, sparse, or matrix-product regimes.
• Limitations: Relative approximation in spectral or entrywise norm is affected by ill-conditioning, coherence, or cluster separation, and the effectiveness of a low-rank proxy reflects the target rank and underlying algebraic or geometric structure. In some deterministic settings, worst-case instances may force error inflation (Pan et al., 2019).
• Extensions: Proxies can be adapted for regularized/robust regression, tensor structures, structured kernels, and non-Euclidean regimes, but high accuracy for off-model queries or data requires careful architecture-dependent tuning (Jääsaari et al., 2024). For discrete proxies, the curse of combinatorial growth in $O(d^2)$4 or cluster complexity induces computational phase transitions (Fomin et al., 2018).

Proxy and low-rank score approximation thus constitute a cross-disciplinary methodological core enabling scalable modeling, optimization, and inference with precise, quantifiable tradeoffs between accuracy, storage, and runtime, with applications in variational inference, kernel approximation, clustering, regression, ranking, and beyond (Modi et al., 2024, Cohen et al., 2015, Pan et al., 2019, Jääsaari et al., 2024, Ye et al., 2019, Lepilov et al., 22 May 2026, Li et al., 28 May 2026).