RSVD with SRHT for Low-Rank Compression

• RSVD is a randomized method that uses the Subsampled Randomized Hadamard Transform to project matrices into a lower-dimensional space for efficient low-rank compression.
• The technique significantly reduces storage and arithmetic complexity by leveraging structured random projections and fast transforms.
• Applications include data compression, signal processing, and machine learning, where maintaining essential matrix geometry is crucial.

Low-rank compression via randomized singular value decomposition (RSVD) is a methodology for approximating large matrices or linear operators by structured lower-rank representations, with a substantial reduction in both storage and arithmetic complexity. RSVD achieves this by projecting the input matrix onto a lower-dimensional random subspace that captures most of its action, then post-processing this “sketch” to extract near-optimal low-rank factorizations. The approach is widely used in scientific computing, data analysis, signal processing, machine learning, and modern neural network compression.

1. Randomized Sketching with the Subsampled Randomized Hadamard Transform

A central mechanism in efficient RSVD is the design of the random projection used to form the sketch. The Subsampled Randomized Hadamard Transform (SRHT) is a key tool for this purpose. Instead of a dense Gaussian or Rademacher matrix, SRHT leverages structured randomness to enable quasi-linear time complexity and better preservation of geometry. The typical form is: $\Omega = S H D,$ with $D$ a random diagonal matrix of $\pm 1$, $H$ the normalized Hadamard matrix (of size $n \times n$, with $n$ a power of two or appropriately padded), and $S$ a subsampling matrix selecting $s$ rows from $HD$. The compressed sketch is computed as: $Y = A \Omega^\top.$ This fast transformation preserves the important subspaces of $A$ and allows the sketch to be computed in $O(n \log n)$ time per vector, which is essential for large-scale problems (Boutsidis, 2011).

2. Algorithmic Frameworks: SRHT-RSVD Algorithms

Two algorithmic flavors emerge for low-rank approximation using SRHT:

Algorithm 1 (Basic RSVD with SRHT):

• Compute $Y = A\Omega^\top$ as above.
• Construct an orthonormal basis $Q$ for the columns of $Y$, e.g. by QR factorization, $Y = QR$.
• Project $A$ into this basis: $B = Q^\top A$.
• Compute a (deterministic) SVD of $B$ to obtain $U_B \Sigma_B V_B^\top$.
• The low-rank approximation to $A$ is then $A \approx Q U_B \Sigma_B V_B^\top$.

Algorithm 2 (Accelerated RSVD with Modified SRHT/Adaptive Sampling):

• Same as above, but further optimizes the application of SRHT and the sampling step.
• For example, the sample size $s$ or the structure of $S$ is chosen to reduce redundancy and to enable the embedding step in $O(m n \log s)$ time rather than $O(m n s)$.
• The net effect is to match or improve the approximation bound of Algorithm 1, while significantly reducing computation (Boutsidis, 2011).

The approximation error is rigorously controlled: for a target rank $k$, with high probability,

$$\|A - Q(Q^\top A)\|_2 \leq (1 + \epsilon) \sigma_{k+1} + \zeta,$$

where $\sigma_{k+1}$ is the $(k+1)^{\text{th}}$ singular value and $\zeta$ decays with increasing $s$, improved via concentration inequalities specific to SRHT (Boutsidis, 2011).

3. Enhanced Approximation Guarantees and Running Time Improvements

The novel analyses inherent to SRHT-based RSVD frameworks provide two main improvements:

• Sharper Error Bounds: Through refined use of measure concentration for SRHT, approximation errors (in both spectral and Frobenius norms) are tied closely to the intrinsic singular spectrum of $A$, and extra error terms $\zeta$ can be made negligible with moderate oversampling.
• Running Time Enhancements: By structuring the sketching operation (either by leveraging the fast Hadamard transform or by adaptive/structured sampling in $S$), the dominant cost of forming $Y = A\Omega^\top$ drops from matrix-matrix multiplication ($O(mns)$) to $O(m n \log s)$. This reduction is particularly impactful for large, sparse, or structured $A$ (Boutsidis, 2011).

The following table summarizes typical complexity and guarantees:

Algorithm	Sketching Cost	Error Bound Type
RSVD w/ dense random $\Omega$	$O(m n k)$	$(1+\epsilon)\sigma_{k+1}$
SRHT-RSVD (Alg. 1)	$O(m n \log n)$	$(1+\epsilon)\sigma_{k+1}+\zeta$
Modified SRHT-RSVD (Alg. 2)	$O(m n \log s)$	Same, with smaller $\zeta$

4. Applications in Compression, Signal Processing, and Data Science

SRHT-based RSVD underpins numerous large-scale data and signal processing pipelines:

• Data/Table Compression: By representing $A$ as a product of low-rank factors, storage costs drop from $O(mn)$ to $O((m+n)k)$. The compressed factors may be efficiently stored, manipulated, or transmitted.
• Machine Learning: Used to accelerate kernel methods, large regression, or dimen-sion-al-ity reduction, where forming the full SVD is infeasible.
• Signal Processing: RSVD enables efficient noise reduction, denoising, and subspace separation in streaming or batch settings.
• Fast Linear Solvers: In methods such as hierarchical solvers (H-matrices) and preconditioners, SRHT-based RSVD is used to approximate off-diagonal blocks or operators.

These methods are particularly suitable where matrices are not only large but also structured or sparse, as SRHT preserves structure while compressing redundant information (Boutsidis, 2011).

5. Design Trade-offs and Practical Considerations

SRHT-based RSVD exposes several practical trade-offs:

• Sketch Size vs. Approximation Quality: Increasing $s$ (sketch dimension) yields better error guarantees at increased cost; the impact of $s$ and the oversampling strategy impacts both error and computation.
• Choice of Transform: SRHT is preferred when $n$ is a power of two or can be padded efficiently; otherwise, alternatives such as subsampled randomized Fourier or cosine transforms may be adopted.
• Numerical Stability: SRHT matrices retain favorable stability properties, but downstream computations (e.g., SVD on $B$) require attention to orthogonality.
• Scalability: The $O(n \log n)$ cost and simple, parallelizable structure of SRHT sketching make these methods viable for distributed and GPU-accelerated environments, especially given the quasi-linear scaling.

6. Theoretical and Empirical Impact

The revised analyses provided for SRHT-based methods, as indicated in the cited work (Boutsidis, 2011), have narrowed the gap between classical deterministic SVD truncation and fast randomized algorithms in both error bounds and wall-time:

• Provable Near-Optimal Bounds: For large $s$ and appropriate $\epsilon$, the approximation approaches that achieved by deterministic truncated SVD.
• Algorithmic Simplicity: The transformations are straightforward to implement, do not require large memory footprints for random matrices, and integrate smoothly with streaming or block processing.
• Dominant Use Cases: SRHT-based RSVD became standard in massive-scale applications (petabyte data tables, genomics, scientific simulation output), providing fast, certifiable low-rank approximations where only approximate preservation of input structure is acceptable.

7. Comparative Perspective

While other structured random projections (Fourier, cosine) and Gaussian random matrices are also viable, the SRHT achieves a compelling balance between speed, accuracy, and storage. Compared to methods that lack fast-multiply structure, SRHT methods remain the standard for low-rank compression where computational efficiency is paramount, especially in memory-limited or real-time applications (Boutsidis, 2011).

These algorithmic advances and their theoretical analysis frame SRHT-based RSVD as a core tool in numerical linear algebra's response to the scale and complexity of contemporary data and scientific workloads.