TR-SVD: Scalable Randomized SVD

• TR-SVD is a family of algorithms that efficiently computes low-rank approximations using thresholded or truncated randomized SVD frameworks.
• It leverages techniques like subspace iteration, Lanczos bidiagonalization, and hybrid block-power methods to balance accuracy, efficiency, and scalability.
• TR-SVD enhances practical applications such as regularized least-squares, correlation screening, and matrix completion through rigorous error bounds and adaptive parameter tuning.

The TR-SVD algorithm (Truncated or Thresholded Randomized Singular Value Decomposition) encompasses a family of techniques and algorithmic frameworks for efficiently computing truncated or thresholded SVD decompositions and associated problem solutions, especially in large-scale or ill-posed settings. These algorithms leverage randomized projection, Lanczos bidiagonalization, subspace iteration, and hybrid block-power methods to obtain low-rank SVD approximations, regularized least-squares solutions, and efficient computations for tasks such as thresholded correlation screening and matrix completion.

1. Mathematical Principles and Problem Formulation

Across its variants, TR-SVD seeks to approximate a given matrix $A \in \mathbb{R}^{m \times n}$ by a low-rank factorization of the form $A \approx U_k \Sigma_k V_k^T$, where $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$ contains the leading singular values, and $k \ll \min(m, n)$ is either a user-chosen target rank, or the number of singular values above a threshold $\tau$. In thresholded variants, the goal is to compute all triplets $(\sigma_i, u_i, v_i)$ with $\sigma_i \ge \tau$ and, often, to maximize the fraction of matrix “energy” retained: $$\mathrm{energy}(k) = \frac{\sum_{i=1}^k \sigma_i^2}{\|A\|_F^2}$$ SVD truncation is widely used for regularization (e.g., in least-squares regression), dimensionality reduction, principal component analysis, and more (Boutsidis et al., 2014, Baglama et al., 2015, Baglama et al., 2024, Jia et al., 2017).

2. Algorithmic Frameworks for TR-SVD Computation

Multiple algorithmic instantiations of TR-SVD exist, each balancing accuracy, efficiency, and scalability.

• Randomized TR-SVD by Subspace Iteration: Constructs a Gaussian test matrix $S \in \mathbb{R}^{n \times k}$, computes an initial sketch $Y_0 = AS$, possibly followed by power iterations $A \approx U_k \Sigma_k V_k^T$0, and then orthonormalizes via QR to obtain a basis $A \approx U_k \Sigma_k V_k^T$1 for the action of $A \approx U_k \Sigma_k V_k^T$2 (Boutsidis et al., 2014).
• Projection and Subspace SVD: Projects $A \approx U_k \Sigma_k V_k^T$3 into the low-dimensional space as $A \approx U_k \Sigma_k V_k^T$4, computes the SVD of $A \approx U_k \Sigma_k V_k^T$5, and transforms the factors back to form $A \approx U_k \Sigma_k V_k^T$6. This compressed SVD significantly reduces cost for large, sparse, or structured data (Jia et al., 2017).
• Hybrid SVD with Thresholding: Repeatedly applies a restarted Lanczos bidiagonalization (e.g., IRLBA or thick-restarted GKLB) with explicit deflation for computed singular directions. When convergence or orthogonality deteriorates (by heuristic criteria), a block-power SVD step restores accuracy and bidiagonal structure. The iteration doubles the batch size and repeats until all $A \approx U_k \Sigma_k V_k^T$7 are extracted, or an energy criterion is met (Baglama et al., 2024).
• Efficient Screening for Correlation Thresholds: Exploits truncated SVDs for pairwise correlation pruning: using the leading right singular vectors, a projected distance bounds the true correlation, so that most pairs below threshold can be discarded without explicit high-dimensional computation (Baglama et al., 2015).

The choice of oversampling parameter $A \approx U_k \Sigma_k V_k^T$8 (number of extra random projections beyond $A \approx U_k \Sigma_k V_k^T$9) is critical for probabilistic guarantees, typically taken as 5–10 or larger for ill-posed problems (Jia et al., 2017).

3. Formal Error Bounds and Theoretical Guarantees

TR-SVD algorithms are underpinned by rigorous error control and convergence results.

• Randomized SVD error: For a best rank-$$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$0 approximation $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$1, the TR-SVD with oversampling $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$2 produces $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$3 satisfying

$$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$4

where $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$5, and sharper bounds exist in ill-posed problem classes (Jia et al., 2017).

• Least-squares solution: If $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$6 is the SVD-truncated regularized solution and $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$7 is computed via randomized TR-SVD, then with probability at least $$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$8,

$$\Sigma_k = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$$9

$k \ll \min(m, n)$0

for failure probability parameter $k \ll \min(m, n)$1 and error parameter $k \ll \min(m, n)$2 (Boutsidis et al., 2014).

• Correlation screening: For truncated-SVD-projected distance screening, there are no false negatives: any pair with $k \ll \min(m, n)$3 remains after pruning (Baglama et al., 2015).
• Hybrid thresholded SVD: For each computed singular triplet, residuals satisfy

$k \ll \min(m, n)$4

and similar for $k \ll \min(m, n)$5, ensuring each is accurate to user-specified tolerance (Baglama et al., 2024).

4. Computational Complexity and Scalability

TR-SVD variants achieve substantial performance savings relative to classical SVD:

Algorithm	Leading Cost Term	Storage Requirement
Full SVD	$k \ll \min(m, n)$6	Full $k \ll \min(m, n)$7
TR-SVD (randomized)	$k \ll \min(m, n)$8	$k \ll \min(m, n)$9-sized factors
TR-SVD (hybrid/Lanczos)	$\tau$0, per triplet	Retained singular vectors
Correlation TR-SVD	$\tau$1 for SVD, $\tau$2 for pruning	$\tau$3

Where:

• $\tau$4 is power iterations,
• $\tau$5 is number of nonzero matrix entries,
• $\tau$6 is total number of significant singular values,
• $\tau$7 is matrix-vector multiplication cost,
• $\tau$8 is truncation rank, $\tau$9 (Boutsidis et al., 2014, Baglama et al., 2015, Baglama et al., 2024, Jia et al., 2017).

For large, sparse, or structured $(\sigma_i, u_i, v_i)$0, TR-SVD algorithms with randomized sketching and iterative schemes reduce both cost and memory footprint, enabling practical analysis of matrices with $(\sigma_i, u_i, v_i)$1 in the $(\sigma_i, u_i, v_i)$2–$(\sigma_i, u_i, v_i)$3 range.

5. Applications and Implementation Strategies

TR-SVD has been applied to:

• Regularized Least-Squares: Computing the SVD-truncated solution $(\sigma_i, u_i, v_i)$4 for regression and inverse problems (Boutsidis et al., 2014).
• Thresholded Correlation Screening: Efficient discovery of all pairs exceeding a user-specified Pearson correlation threshold, notably in large-scale genomics or finance (Baglama et al., 2015).
• Matrix Completion and Image Compression: Computing partial SVDs to a given threshold or fractional energy, used in imputation, denoising, and blockwise compression (Baglama et al., 2024).
• Regularized Inversion with General-Form Penalties: Combining TR-SVD with LSQR in the MTRSVD framework solves large constrained minimization problems of the form $(\sigma_i, u_i, v_i)$5 subject to $(\sigma_i, u_i, v_i)$6, where $(\sigma_i, u_i, v_i)$7 is arbitrary (Jia et al., 2017).

Implementations leverage:

• IRLBA: For fast, restartable truncated SVDs,
• MATLAB/R: Publicly available hybrid routines using function handles or custom C extensions,
• Oversampling and Block Power: Practical parameter tuning for ill-posedness, heuristics for parallelization, and explicit QR-based reorthogonalization for loss of accuracy.

6. Error Analysis and Parameter Selection

Parameter tuning (truncation rank $(\sigma_i, u_i, v_i)$8, oversampling $(\sigma_i, u_i, v_i)$9, tolerance, block-power steps) critically affects performance and accuracy:

• Oversampling $\sigma_i \ge \tau$0: Substantially reduces risk of missing significant singular directions, especially crucial in ill-posed problems where leading singular spectra decay rapidly. In practice, $\sigma_i \ge \tau$1–$\sigma_i \ge \tau$2 or even $\sigma_i \ge \tau$3 (Jia et al., 2017).
• Truncation/Threshold $\sigma_i \ge \tau$4/relative energy: Provides a sharp-cutoff regime for singular-value significance, allowing adaptive stopping and memory economy (Baglama et al., 2024).
• Stopping in Iterative LSQR: For inner loops in regularization, choosing tolerance $\sigma_i \ge \tau$5 ensures solution accuracy tracks that of the SVD truncation itself (Jia et al., 2017).
• Error propagation: In ill-posed contexts, error bounds for TR-SVD are sharply characterized, showing that the TRSVD error approaches the best achievable ($\sigma_i \ge \tau$6) with respect to the true spectrum (Jia et al., 2017).

7. Implementation Notes and Empirical Performance

Empirical demonstrations highlight substantial savings in time and memory for TR-SVD algorithms. Reported results include:

• In high-dimensional genomics data, TR-SVD-based correlation screening (with $\sigma_i \ge \tau$7) reduces auxiliary storage by an order of magnitude compared to brute force, with the same output (Baglama et al., 2015).
• On DNA methylation data ($\sigma_i \ge \tau$8), full enumeration is infeasible, but TR-SVD finds all above-threshold correlations in hours (serial) or minutes (parallel) using moderate memory (Baglama et al., 2015).
• MTRSVD achieves regularization accuracy matching or surpassing deterministic GSVD-based regularization, while scaling to problems at least an order of magnitude larger (Jia et al., 2017).
• MATLAB, Octave, and R implementations provide user-accessible routines with detailed parameter control, especially for threshold/energy, convergence tolerance, and explicit block-power re-orthogonalization (Baglama et al., 2024).

The broader significance is that TR-SVD—across its randomized, thresholded, and hybridized instantiations—enables a class of scalable, robust, and mathematically controlled algorithms for low-rank approximation, regularized inversion, and large-scale data analytics (Boutsidis et al., 2014, Baglama et al., 2015, Baglama et al., 2024, Jia et al., 2017).