TR-SVD: Scalable Randomized SVD

• TR-SVD is a family of algorithms that efficiently computes low-rank approximations using thresholded or truncated randomized SVD frameworks.
• It leverages techniques like subspace iteration, Lanczos bidiagonalization, and hybrid block-power methods to balance accuracy, efficiency, and scalability.
• TR-SVD enhances practical applications such as regularized least-squares, correlation screening, and matrix completion through rigorous error bounds and adaptive parameter tuning.

The TR-SVD algorithm (Truncated or Thresholded Randomized Singular Value Decomposition) encompasses a family of techniques and algorithmic frameworks for efficiently computing truncated or thresholded SVD decompositions and associated problem solutions, especially in large-scale or ill-posed settings. These algorithms leverage randomized projection, Lanczos bidiagonalization, subspace iteration, and hybrid block-power methods to obtain low-rank SVD approximations, regularized least-squares solutions, and efficient computations for tasks such as thresholded correlation screening and matrix completion.

1. Mathematical Principles and Problem Formulation

Across its variants, TR-SVD seeks to approximate a given matrix $A \in \mathbb{R}^{m \times n}$ by a low-rank factorization of the form $A \approx U_k \Sigma_k V_k^T$, where $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$ contains the leading singular values, and $k \ll \min(m, n)$ is either a user-chosen target rank, or the number of singular values above a threshold $\tau$. In thresholded variants, the goal is to compute all triplets $(\sigma_i, u_i, v_i)$ with $\sigma_i \ge \tau$ and, often, to maximize the fraction of matrix “energy” retained: $$\mathrm{energy}(k) = \frac{\sum_{i=1}^k \sigma_i^2}{\|A\|_F^2}$$ SVD truncation is widely used for regularization (e.g., in least-squares regression), dimensionality reduction, principal component analysis, and more (Boutsidis et al., 2014, Baglama et al., 2015, Baglama et al., 8 Jul 2024, Jia et al., 2017).

2. Algorithmic Frameworks for TR-SVD Computation

Multiple algorithmic instantiations of TR-SVD exist, each balancing accuracy, efficiency, and scalability.

• Randomized TR-SVD by Subspace Iteration: Constructs a Gaussian test matrix $S \in \mathbb{R}^{n \times k}$, computes an initial sketch $Y_0 = AS$, possibly followed by power iterations $Y_j = A(A^T Y_{j-1})$, and then orthonormalizes via QR to obtain a basis $Q$ for the action of $A$ (Boutsidis et al., 2014).
• Projection and Subspace SVD: Projects $A$ into the low-dimensional space as $M = Q^T A$, computes the SVD of $M$, and transforms the factors back to form $\widetilde{A}_k$. This compressed SVD significantly reduces cost for large, sparse, or structured data (Jia et al., 2017).
• Hybrid SVD with Thresholding: Repeatedly applies a restarted Lanczos bidiagonalization (e.g., IRLBA or thick-restarted GKLB) with explicit deflation for computed singular directions. When convergence or orthogonality deteriorates (by heuristic criteria), a block-power SVD step restores accuracy and bidiagonal structure. The iteration doubles the batch size and repeats until all $\sigma_i \ge \tau$ are extracted, or an energy criterion is met (Baglama et al., 8 Jul 2024).
• Efficient Screening for Correlation Thresholds: Exploits truncated SVDs for pairwise correlation pruning: using the leading right singular vectors, a projected distance bounds the true correlation, so that most pairs below threshold can be discarded without explicit high-dimensional computation (Baglama et al., 2015).

The choice of oversampling parameter $q$ (number of extra random projections beyond $k$) is critical for probabilistic guarantees, typically taken as 5–10 or larger for ill-posed problems (Jia et al., 2017).

3. Formal Error Bounds and Theoretical Guarantees

TR-SVD algorithms are underpinned by rigorous error control and convergence results.

• Randomized SVD error: For a best rank-$k$ approximation $A_k$, the TR-SVD with oversampling $q$ produces $\widetilde{A}_k$ satisfying

$$\|A - \widetilde{A}_k\| \le \widetilde{\sigma}_{k+1} + \|A - QQ^T A\|$$

where $\widetilde{\sigma}_{k+1} \le \sigma_{k+1}$, and sharper bounds exist in ill-posed problem classes (Jia et al., 2017).

• Least-squares solution: If $x_k = A_k^+ b$ is the SVD-truncated regularized solution and $\tilde{x}_k$ is computed via randomized TR-SVD, then with probability at least $1-e^{-n} - 2.35\delta$,

$$\|A \tilde{x}_k - b\|_2 \le \|A x_k - b\|_2 + \epsilon\|b\|_2$$

$$\frac{\|x_k - \tilde{x}_k\|_2}{\|x_k\|_2} \le \frac{4}{3}\epsilon$$

for failure probability parameter $\delta$ and error parameter $\epsilon$ (Boutsidis et al., 2014).

• Correlation screening: For truncated-SVD-projected distance screening, there are no false negatives: any pair with $\operatorname{cor}(a_i,a_j) \ge t$ remains after pruning (Baglama et al., 2015).
• Hybrid thresholded SVD: For each computed singular triplet, residuals satisfy

$$\|A v_i - \sigma_i u_i\| \leq \texttt{tol} \cdot \sigma_1$$

and similar for $A^T u_i$, ensuring each is accurate to user-specified tolerance (Baglama et al., 8 Jul 2024).

4. Computational Complexity and Scalability

TR-SVD variants achieve substantial performance savings relative to classical SVD:

Algorithm	Leading Cost Term	Storage Requirement
Full SVD	$O(m n \min\{m,n\})$	Full $U,V$
TR-SVD (randomized)	$O((p+1)\operatorname{nnz}(A)k + (m+n)k^2)$	$k$-sized factors
TR-SVD (hybrid/Lanczos)	$O(s \cdot T_{\mathrm{mv}})$, per triplet	Retained singular vectors
Correlation TR-SVD	$O(m n p)$ for SVD, $O(n p \ell)$ for pruning	$O(n p + m p)$

Where:

• $p$ is power iterations,
• $\operatorname{nnz}(A)$ is number of nonzero matrix entries,
• $s$ is total number of significant singular values,
• $T_{\mathrm{mv}}$ is matrix-vector multiplication cost,
• $k$ is truncation rank, $\ell=k+q$ (Boutsidis et al., 2014, Baglama et al., 2015, Baglama et al., 8 Jul 2024, Jia et al., 2017).

For large, sparse, or structured $A$, TR-SVD algorithms with randomized sketching and iterative schemes reduce both cost and memory footprint, enabling practical analysis of matrices with $m, n$ in the $10^4$–$10^5$ range.

5. Applications and Implementation Strategies

TR-SVD has been applied to:

• Regularized Least-Squares: Computing the SVD-truncated solution $x_k = A_k^+ b$ for regression and inverse problems (Boutsidis et al., 2014).
• Thresholded Correlation Screening: Efficient discovery of all pairs exceeding a user-specified Pearson correlation threshold, notably in large-scale genomics or finance (Baglama et al., 2015).
• Matrix Completion and Image Compression: Computing partial SVDs to a given threshold or fractional energy, used in imputation, denoising, and blockwise compression (Baglama et al., 8 Jul 2024).
• Regularized Inversion with General-Form Penalties: Combining TR-SVD with LSQR in the MTRSVD framework solves large constrained minimization problems of the form $\min \|Lx\|$ subject to $\min \|Ax-b\|$, where $L$ is arbitrary (Jia et al., 2017).

Implementations leverage:

• IRLBA: For fast, restartable truncated SVDs,
• MATLAB/R: Publicly available hybrid routines using function handles or custom C extensions,
• Oversampling and Block Power: Practical parameter tuning for ill-posedness, heuristics for parallelization, and explicit QR-based reorthogonalization for loss of accuracy.

6. Error Analysis and Parameter Selection

Parameter tuning (truncation rank $k$, oversampling $q$, tolerance, block-power steps) critically affects performance and accuracy:

• Oversampling $q$: Substantially reduces risk of missing significant singular directions, especially crucial in ill-posed problems where leading singular spectra decay rapidly. In practice, $q=5$–$10$ or even $q \approx k$ (Jia et al., 2017).
• Truncation/Threshold $\tau$/relative energy: Provides a sharp-cutoff regime for singular-value significance, allowing adaptive stopping and memory economy (Baglama et al., 8 Jul 2024).
• Stopping in Iterative LSQR: For inner loops in regularization, choosing tolerance $\texttt{tol} \ll \|e\|/\|b_{\rm true}\|$ ensures solution accuracy tracks that of the SVD truncation itself (Jia et al., 2017).
• Error propagation: In ill-posed contexts, error bounds for TR-SVD are sharply characterized, showing that the TRSVD error approaches the best achievable ($\|A-QQ^T A\|$) with respect to the true spectrum (Jia et al., 2017).

7. Implementation Notes and Empirical Performance

Empirical demonstrations highlight substantial savings in time and memory for TR-SVD algorithms. Reported results include:

• In high-dimensional genomics data, TR-SVD-based correlation screening (with $p=10$) reduces auxiliary storage by an order of magnitude compared to brute force, with the same output (Baglama et al., 2015).
• On DNA methylation data ($80 \times 394{,}014$), full enumeration is infeasible, but TR-SVD finds all above-threshold correlations in hours (serial) or minutes (parallel) using moderate memory (Baglama et al., 2015).
• MTRSVD achieves regularization accuracy matching or surpassing deterministic GSVD-based regularization, while scaling to problems at least an order of magnitude larger (Jia et al., 2017).
• MATLAB, Octave, and R implementations provide user-accessible routines with detailed parameter control, especially for threshold/energy, convergence tolerance, and explicit block-power re-orthogonalization (Baglama et al., 8 Jul 2024).

The broader significance is that TR-SVD—across its randomized, thresholded, and hybridized instantiations—enables a class of scalable, robust, and mathematically controlled algorithms for low-rank approximation, regularized inversion, and large-scale data analytics (Boutsidis et al., 2014, Baglama et al., 2015, Baglama et al., 8 Jul 2024, Jia et al., 2017).