Randomized Sketch-and-Project Algorithms

• Randomized sketch-and-project algorithms are iterative methods that project current estimates onto solution sets defined by random low-dimensional sketches, ensuring global convergence at low per-iteration cost.
• They leverage diverse sketch constructions—such as Gaussian, SRHT, and sparse sketches—to adapt to problem structures and achieve improved convergence rates.
• The framework unifies classical methods like randomized Kaczmarz and coordinate descent while enabling advanced extensions like adaptive sampling, momentum acceleration, and structured sketching.

Randomized sketch-and-project algorithms constitute a foundational paradigm for efficiently solving large-scale linear (and some nonlinear) algebraic and optimization problems by iteratively projecting onto solution sets defined by random low-dimensional sketches of the original system. By repeatedly leveraging cheap projections onto random subspaces—selected according to data-driven, combinatorial, or structured probabilistic rules—these methods combine theoretical guarantees of global convergence with extremely favorable iteration costs and parallelizability. The generality of the framework subsumes a diverse array of classical and contemporary algorithms, including randomized Kaczmarz, coordinate descent, subspace Newton, and their structured and block variants.

1. Core Framework: Iterative Sketch-and-Project

The prototypical sketch-and-project iteration addresses a linear system $Ax = b$, $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^m$. At each iteration, one draws a random sketch matrix $S \in \mathbb{R}^{m \times q}$, projects the current iterate $x_k$ onto the solution set of $S^T A x = S^T b$ in a user-defined $G$-geometry, and computes the update: $$x_{k+1} = x_k + G^{-1}A^T S\bigl(S^T A G^{-1}A^T S\bigr)^{-1}S^T(b - A x_k),$$ where $G \succ 0$ modulates the metric (e.g., $G=I$, $A \in \mathbb{R}^{m \times n}$0, $A \in \mathbb{R}^{m \times n}$1). This form unifies several perspectives: projection (proximal), constrained approximation (dual), algebraic random linear solve, random fixed point, and geometric intersect. Efficient implementations exploit the small $A \in \mathbb{R}^{m \times n}$2-dimensional structure for rapid block, coordinate, or randomized subspace updates (Gower et al., 2015, Gower, 2016).

Key specializations include:

• Randomized Kaczmarz: $A \in \mathbb{R}^{m \times n}$3, coordinate sampling; recovers Strohmer-Vershynin's exponential convergence (Gower et al., 2015).
• Randomized Coordinate Descent: $A \in \mathbb{R}^{m \times n}$4, $A \in \mathbb{R}^{m \times n}$5, $A \in \mathbb{R}^{m \times n}$6; block- or coordinate-level minimization (Gower et al., 2015).
• Block/Structured/Krylov Variants: $A \in \mathbb{R}^{m \times n}$7 as block-rows or constructed from residuals to bridge with CG/GMRES (Brust et al., 2024).
• Sketching for matrix equations $A \in \mathbb{R}^{m \times n}$8: simultaneous left and right projections using $A \in \mathbb{R}^{m \times n}$9, $b \in \mathbb{R}^m$0; see (Bao et al., 2023).

2. Randomized Sketch Construction: Gaussian, Orthogonal, Sparse, Structured

The statistical and computational behavior of sketch-and-project methods is governed by the distribution and structure of $b \in \mathbb{R}^m$1. Representative constructions include:

• Dense Gaussian/standard random projections: Provide rotational invariance; $b \in \mathbb{R}^m$2 (Dereziński et al., 2022).
• Subsampled Randomized Hadamard Transform (SRHT): Enables fast transforms and preserves isometry asymptotically (Lacotte et al., 2020).
• Truncated Haar orthogonal sketches: Asymptotically identical performance to SRHT in least squares (Lacotte et al., 2020).
• Leverage-score sparsified (LESS): Achieves subspace embedding with high probability using sparse $b \in \mathbb{R}^m$3 (Dereziński et al., 2022).
• Count-sketch/sign sketches and sparse blockings: Reduce memory and arithmetic cost in massive systems (Jiang et al., 17 Nov 2025, Brust et al., 2024).
• Structured/block/coordinate sketches: Exploit data or application-specific patterns for improved parallelism or adaptation (Brust et al., 2024, Morshed et al., 2020).

The choice of sketch size $b \in \mathbb{R}^m$4 (or block size, number of coordinates) and distribution of $b \in \mathbb{R}^m$5 fundamentally determines the per-iteration convergence rate, as well as computational cost (Dereziński et al., 2022, Gower et al., 2015).

3. Convergence Guarantees and Sharp Rate Analysis

The expected error at iteration $b \in \mathbb{R}^m$6 decays at a linear rate governed by spectral characteristics of the expected sketched projection: $b \in \mathbb{R}^m$7 with

$b \in \mathbb{R}^m$8

Sharp analysis links this $b \in \mathbb{R}^m$9 explicitly to the sketch size, data spectrum, and sketching law (Dereziński et al., 2022):

• For $S \in \mathbb{R}^{m \times q}$0 and $S \in \mathbb{R}^{m \times q}$1 full column rank,

$S \in \mathbb{R}^{m \times q}$2

where $S \in \mathbb{R}^{m \times q}$3 is the expected residual error of randomized SVD with $S \in \mathbb{R}^{m \times q}$4 sketches (Dereziński et al., 2022).

• Spectral decay in $S \in \mathbb{R}^{m \times q}$5 (polynomial/exponential) results in superlinear acceleration with increasing $S \in \mathbb{R}^{m \times q}$6.
• For SRHT/Haar, closed-form optimal convergence rates and step-sizes were derived via random matrix theory, showing strict improvement over Gaussian sketches (Lacotte et al., 2020).
• Sparse sketches exhibit comparable rates to fully dense sketches under stable-rank conditions (Dereziński et al., 2022).

The methodology applies to overdetermined, underdetermined, and even matrix equation settings, with explicit adaptation for block, coordinate, and dual update settings (Bao et al., 2023, Jarman et al., 2021).

4. Algorithmic Extensions: Bregman Geometry, Adaptive and Greedy Sampling, Momentum

The archetype accepts broad generalizations:

• Bregman projection: The Euclidean norm is replaced by the Bregman divergence $S \in \mathbb{R}^{m \times q}$7, enabling sparsity, positivity, or entropy constraints in the iterates. The iteration becomes an exact minimization of $S \in \mathbb{R}^{m \times q}$8 over the sketched constraint (Yuan et al., 2021).
• Adaptive sampling: The sketch selection probability is updated on-the-fly based on residuals or sketched losses, e.g., max-distance, proportional-to-loss, Kaczmarz-Motzkin with block selection. Adaptive sampling provably yields stronger convergence factors, quantifiable via spectral constants $S \in \mathbb{R}^{m \times q}$9 (Yuan et al., 2021, Morshed et al., 2020).
• Heavy-ball momentum and acceleration: Appropriately parameterized momentum steps can systematically accelerate convergence, particularly when the data are ill-conditioned (Morshed et al., 2020).
• Projected feasibility and dual formulations: Randomized projections generalize to inequalities, matrix equations, and dual problems, enabling feasibility certificates and projection-based solvers (Morshed et al., 2020, Nakatsukasa et al., 2021, Wang et al., 2016).

Empirical results have demonstrated that block and adaptive strategies (especially SKM/Kaczmarz-Motzkin) provide substantial reductions in total arithmetic cost for high-accuracy solutions, often outperforming uniform or leverage-score sampling by a wide factor (Yuan et al., 2021, Morshed et al., 2020).

5. Structured Sketching: Block, Krylov, and Krylov–Like Randomized Methods

Recent developments incorporate Krylov subspaces and residual history as active sketches, yielding structured methods that unify elements of classical (e.g., CG, GMRES) and randomized solvers (Brust et al., 2024):

• Sketches constructed from residual history or basis expansion provide short recurrences and drastically reduce memory overhead compared to GMRES-type orthogonalization.
• Weighting strategies, e.g., $x_k$0 or $x_k$1 (for symmetric positive definite $x_k$2), reduce the per-iteration complexity to $x_k$3 and align with the behavior of Krylov solvers in SPD settings, recovering CG in exact arithmetic (Brust et al., 2024).
• Finite termination in exact arithmetic is assured in at most $x_k$4 steps for full-rank $x_k$5 if the sketch matrix has accumulated $x_k$6 independent directions.

Structured sketching competes favorably in wall-clock time and iteration counts with optimized GMRES, SYMMLQ, and state-of-the-art block Kaczmarz variants across a wide range of large-scale linear systems, including highly ill-conditioned and indefinite problems (Brust et al., 2024).

6. Practical Considerations: Computational Cost, Implementation, and Large-Scale Performance

The computational cost of each iteration is dominated by the cost to apply $x_k$7 and $x_k$8 (and their transposes), solve the small $x_k$9 (or block-sized) system, and update the iterate. Key factors include:

• Sketch dimensionality: Larger blocks/sketches typically improve $S^T A x = S^T b$0, reduce iteration count, but increase per-iteration work.
• Sketch structure: Sparse/structured sketches enable implementation in sublinear time per iteration, suitable for streamed or distributed environments (Jiang et al., 17 Nov 2025).
• Basis updating and memory: Structured and Krylov-based sketches reduce storage from $S^T A x = S^T b$1 (for GMRES with $S^T A x = S^T b$2-step basis) to $S^T A x = S^T b$3 or $S^T A x = S^T b$4 in the block/inverse case (Brust et al., 2024).
• Numerical stability: Condition number of the sketched system and orthogonalization of basis vectors is critical; adaptive basis “whitening” and regular restarting schemes address loss of independence (Nakatsukasa et al., 2021).

Empirical benchmarks, including SuiteSparse systems and large-scale synthetic data, confirm that randomized sketch-and-project algorithms with adaptive, block, or structured sketching routinely outperform both classic iterative methods and one-shot sketching in time to accuracy, especially as $S^T A x = S^T b$5 increase (Jiang et al., 17 Nov 2025, Nakatsukasa et al., 2021, Brust et al., 2024).

7. Theoretical Frontiers: Random Matrix Theory, Subspace Embeddings, and Statistical Limits

The quantitative performance of sketch-and-project algorithms is underpinned by advances in random matrix theory:

• Deterministic and asymptotic spectral characterizations: For popular sketch classes (SRHT/Haar, Gaussian, OSNAP, CW), precise limiting spectra and eigenvalue distributions have been established, enabling analytic convergence rates (Lacotte et al., 2020, Dereziński et al., 2022).
• Tracy-Widom law and high-dimensional embeddings: The asymptotic probability that a sketch preserves subspace geometry up to $S^T A x = S^T b$6-distortion is governed by Tracy-Widom fluctuations, enabling sharp predictions of the behavior and reliability of randomized sketching at scale (Ahfock et al., 2022).
• Optimal step-sizes and rates: Closed-form expressions for (asymptotically) optimal parameters are available for many sketch families, yielding the precise conditions under which Haar/SRHT outperform Gaussian sketches, and the (super)linear dependence of rates on sketch/block size (Lacotte et al., 2020, Dereziński et al., 2022).

A plausible implication is that “orthogonal sketches” (SRHT/Haar) should be preferred over i.i.d. Gaussian for optimal rates and robustness in high-dimensional regimes, and that sparse embedding constructions can offer nearly identical convergence with massively reduced computational cost (Dereziński et al., 2022, Lacotte et al., 2020).

---

References

• (Lacotte et al., 2020) "Optimal Iterative Sketching with the Subsampled Randomized Hadamard Transform"
• (Dereziński et al., 2022) "Sharp Analysis of Sketch-and-Project Methods via a Connection to Randomized Singular Value Decomposition"
• (Gower et al., 2015) "Randomized Iterative Methods for Linear Systems"
• (Gower, 2016) "Sketch and Project: Randomized Iterative Methods for Linear Systems and Inverting Matrices"
• (Wang et al., 2016) "Sketching Meets Random Projection in the Dual: A Provable Recovery Algorithm for Big and High-dimensional Data"
• (Morshed et al., 2020) "Sketch & Project Methods for Linear Feasibility Problems: Greedy Sampling & Momentum"
• (Yuan et al., 2021) "Adaptively Sketched Bregman Projection Methods for Linear Systems"
• (Brust et al., 2024) "Structured Sketching for Linear Systems"
• (Jiang et al., 17 Nov 2025) "Accelerated Kaczmarz methods via randomized sketch techniques for solving consistent linear systems"
• (Bao et al., 2023) "A sketch-and-project method for solving the matrix equation AXB = C"
• (Ahfock et al., 2022) "On randomized sketching algorithms and the Tracy-Widom law"
• (Jarman et al., 2021) "Randomized Extended Kaczmarz is a Limit Point of Sketch-and-Project"
• (Nakatsukasa et al., 2021) "Fast & Accurate Randomized Algorithms for Linear Systems and Eigenvalue Problems"