Sharp analysis of sketched least squares and randomized low-rank approximation

Abstract: Two widely used randomized algorithms are the sketch-and-solve method for least-squares regression and the randomized SVD for low-rank approximation. These algorithms apply a random embedding to compress a target matrix, and they perform computations on the compressed matrix to save computational cost. This paper asks, what is the optimal random embedding in these algorithms? Also, what is the sharpest possible error bound for the optimal embedding? The paper proves that a random orthonormal matrix is minimax optimal for the sketch-and-solve algorithm while any rotation-invariant embedding is minimax optimal for the randomized SVD. Following these results, the paper obtains the best possible error bounds for sketched least-squares and the randomized SVD. Last, empirical experiments provide evidence of universality phenomena, in which several random embeddings lead to similar accuracy to the optimal embeddings in practice.

• The paper proves exact, unimprovable mean-square error bounds and minimax optimality for sketch-and-solve least squares, with random orthonormal embeddings achieving the best worst-case factor and embedding dimension of roughly r/ε rather than r/ε².
• The analysis shows that Gaussian and orthonormal embeddings are nearly equivalent when the sketch dimension is much smaller than n, while rotation-invariant embeddings are minimax optimal for randomized SVD and Nyström approximation.
• The paper establishes a fundamental complexity gap between generalized Nyström and randomized SVD—approximately q/ε² versus q/ε matrix–vector multiplications—and derives an optimal oversampling split for a fixed computational budget.

This paper by Epperly and Webber establishes minimax optimality results and sharp, unimprovable error bounds for two foundational randomized linear algebra algorithms: sketch-and-solve least squares and the randomized SVD (with extensions to Nyström-type approximations). The central question is which random embedding $\boldsymbol{\Omega} \in K^{n \times \ell}$ minimizes worst-case mean square error, and what the exact value of that error is.

Sketch-and-solve: exact analysis and optimality

For the least-squares problem $\min_X \|B - AX\|_{\rm F}^2$ with rank-$r$ matrix $A$, the paper gives an exact formula for the expected residual of the sketch-and-solve solution $(\Omega^*A)^+(\Omega^*B)$. For a Gaussian embedding,

$$\mathbb{E}\,\|B - A\hat{X}\|_{\rm F}^2 = \left(1 + \frac{r}{\ell - r - \alpha_K}\right)\min_X \|B - AX\|_{\rm F}^2,$$

while a random orthonormal embedding achieves the strictly smaller factor $$1 + \frac{n-\ell}{n-r}\cdot\frac{r}{\ell - r - \alpha_K}$$, where $\alpha_K$ is 1 over $\mathbb{R}$ and 0 over $\mathbb{C}$. The Gaussian result extends Bartan and Pilanci; the random orthonormal formulas are new.

Several practical consequences follow directly from this formula. Orthogonality helps only marginally: since practical settings use $\min_X \|B - AX\|_{\rm F}^2$0, Gaussian and orthonormal embeddings are nearly interchangeable. Complexity helps only marginally: a real embedding of dimension $\min_X \|B - AX\|_{\rm F}^2$1 matches a complex one of dimension $\min_X \|B - AX\|_{\rm F}^2$2. Most importantly, the formula corrects the folklore scaling: an embedding dimension $\min_X \|B - AX\|_{\rm F}^2$3 suffices for relative accuracy $\min_X \|B - AX\|_{\rm F}^2$4, whereas standard subspace-embedding analyses yield the incorrect $\min_X \|B - AX\|_{\rm F}^2$5 scaling.

The paper then proves a matching lower bound: for any full-rank data-oblivious embedding, there exists a problem instance with $\min_X \|B - AX\|_{\rm F}^2$6 whose normalized mean square error is at least $\min_X \|B - AX\|_{\rm F}^2$7. Combined with the upper bound, this establishes that the random orthonormal matrix is the unique minimax optimal embedding in the game where an adversary selects $\min_X \|B - AX\|_{\rm F}^2$8 knowing the distribution of $\min_X \|B - AX\|_{\rm F}^2$9. Notably, the lower bound assumes nothing about the structure of $r$0, unlike prior lower bounds restricted to Gaussian sketches.

Randomized SVD and Nyström approximation

For the randomized SVD approximation $r$1, the classical Halko–Martinsson–Tropp bound gives $r$2. The paper refines this by the factor $r$3:

$r$4

valid for any rotation-invariant embedding, and proves a matching lower bound holding for every full-rank embedding. Hence any rotation-invariant embedding—Gaussian or orthonormal—is minimax optimal for the randomized SVD, and the bound cannot be improved without additional assumptions on $r$5.

The proof route is via the Gram correspondence between the randomized SVD and the randomized Nyström approximation of psd matrices: the Schur complement $r$6 governs both errors. The key technical device is a hard instance $r$7 with two distinct eigenvalues, whose limiting Nyström error as $r$8 is computed exactly using expectation formulas for inverse Wishart and Beta (Jacobi) matrices. An averaging argument over random permutations of eigenvalues, combined with concavity and monotonicity of the Schur complement map, converts the hard-instance analysis into a universal upper bound.

Generalized Nyström and the oversampling ratio

The generalized Nyström approximation $r$9 combines the two primitives above, and the paper derives sharp upper and lower bounds for it as well. A consequence resolves an open question of Chen et al.: guaranteeing $A$0-relative error requires at least roughly $A$1 matrix–vector multiplications for generalized Nyström, versus $A$2 for the randomized SVD. Thus generalized Nyström has intrinsic $A$3 complexity while the randomized SVD enjoys $A$4.

The bounds also settle how to split a fixed budget $A$5 of multiplications between the left and right embeddings: minimizing the worst-case bound yields the oversampling ratio

$A$6

This interpolates between the conflicting recommendations in prior literature ($A$7 versus $A$8): aggressive ratios near 1 are appropriate when singular values decay rapidly (large target rank $A$9), while slower decay favors larger ratios.

Empirical universality

Experiments with eight embeddings (Gaussian, random sign, uniform, SparseIID, SparseStack, random orthonormal, Givens, SRTT) support a universality phenomenon: embeddings cluster into a "Gaussian" class (near-iid entries) and a "random orthonormal" class (orthonormal columns), each tracking the corresponding theoretical prediction closely across coherent and incoherent test problems. One exception is documented: sign-based and sparse embeddings with small dimension $(\Omega^*A)^+(\Omega^*B)$0 exhibit elevated error on a "step/coherent" randomized SVD instance, because discrete $(\Omega^*A)^+(\Omega^*B)$1 values can annihilate dominant eigenvector directions; universality is restored for $(\Omega^*A)^+(\Omega^*B)$2. In "poly" spectral-decay problems, the sharp bound overestimates actual error by roughly $(\Omega^*A)^+(\Omega^*B)$3, an overestimation factor the authors note is not theoretically understood since it depends on the detailed eigenvalue distribution.

Limitations and open questions

The theory covers only Gaussian and rotation-invariant embeddings exactly; claims about fast structured embeddings rest on empirical evidence, with rigorous universality theory still incomplete. The authors conjecture that no algorithm whatsoever operating on sketched data can beat sketch-and-solve or the randomized SVD with a random orthonormal embedding (extending optimality beyond the specific estimators analyzed), and conjecture conversely that generalized Nyström is not worst-case optimal. Whether the observed universality classes can be characterized rigorously—and whether the $(\Omega^*A)^+(\Omega^*B)$4 overestimation on polynomial spectra can be explained—remain open.

Conclusion

The paper closes the gap between upper and lower bounds for sketch-and-solve and randomized low-rank approximation, identifying random orthonormal matrices as minimax optimal for sketch-and-solve and all rotation-invariant embeddings as minimax optimal for the randomized SVD and Nyström methods. The resulting bounds are exact rational functions of the rank, embedding dimensions, and target rank, they determine minimal embedding dimensions, and they yield concrete guidance on oversampling ratios and the fundamental $(\Omega^*A)^+(\Omega^*B)$5 versus $(\Omega^*A)^+(\Omega^*B)$6 complexity separation between generalized Nyström and the randomized SVD.