Low-Rank Compressed Representation

• Low-rank compressed representation is a technique that approximates large matrices or tensors by decomposing them into lower-dimensional factors for enhanced efficiency.
• It employs methods like SVD, randomized sketching, and adaptive hierarchical approaches to control approximation error and optimize performance.
• This approach is widely applied in machine learning, signal processing, and scientific computing to achieve significant storage savings and computation speedups.

A low-rank compressed representation refers to the approximation, encoding, or manipulation of high-dimensional data objects—most commonly matrices or tensors—by decomposing them into factors of much lower rank than the ambient dimension, thereby dramatically reducing storage, computation, or transmission costs. This paradigm exploits the inherent redundancy in many applied datasets and operator representations, enabling both theoretical guarantees and practical speedups across fields such as numerical linear algebra, machine learning, signal processing, scientific computing, and electronic structure theory. Low-rank compressed representations are realized in diverse algorithmic forms, including matrix/tensor SVD, nuclear-norm minimization, randomized sketching, structured matrix parametrizations, adaptive hierarchical approaches, and combinations with quantization or sparsity.

1. Mathematical Foundations of Low-Rank Compression

A matrix $A\in\mathbb R^{m\times n}$ is said to be exactly (or approximately) rank-$r$ if it can be written as $A = U V^T$ with $$U\in\mathbb R^{m\times r}, V\in\mathbb R^{n\times r}$$, or closely approximated in Frobenius or spectral norm by such a decomposition. This is the basis for classical SVD-based compression and the Eckart–Young theorem, which states that the best rank-$r$ approximation of $A$ (in the Frobenius norm) is given by truncating its singular value decomposition to the top $r$ singular vectors (Cai et al., 2013).

For higher-order tensors $\mathcal{A}$, canonical decompositions such as the CP (CANDECOMP/PARAFAC), Tucker, and tensor-train (TT) formats generalize this principle, approximating a $d$-way array as multi-linear products of lower-dimensional factors, reducing the parameter count from $O(\prod_{k=1}^d I_k)$ to $r$0 or similar expressions (Hawkins et al., 2021).

In many scientific and engineering applications, the effective or numerical rank required for a given error tolerance $r$1 grows much more slowly than the ambient dimension, yielding compression factors of $r$2 or higher.

2. Principal Algorithmic Approaches

Low-rank compressed representation methods can be categorized by their factorization strategy, error control, adaptivity, and numerical implementation:

• Global truncated SVD/decomposition: Directly computes the top-$r$3 components of $r$4 or $r$5, yielding optimal error for a given rank (Cai et al., 2013), but at $r$6 cost or worse for large matrices.
• Randomized sketching: Rapidly estimates approximate spectral projectors, then computes low-rank factors or "cores" in much reduced dimension. Examples include Compressed Randomized UTV (CoR-UTV), which achieves $r$7 cost for rank-$r$8 compression and is competitive with classical randomized SVD (Kaloorazi et al., 2018, Saha et al., 2023). Bitwise quantization can be combined with low-rank sketching for extreme compression (Saha et al., 2023).
• Structured matrix approaches: Low displacement rank (LDR) parameterizations encode matrices via displacement operators plus a low-rank residual, subsuming Toeplitz-like, Hankel-like, and other structure, allowing fast storage and matrix–vector multiplication (Thomas et al., 2018).
• Tensor compression: Multi-way decompositions (CP, Tucker, TT, TTM) and their hierarchical/multiresolution counterparts capture multi-scale redundancy in high-dimensional arrays with provable local convergence, allowing compression that beats single-scale matrix truncation, especially for multiscale signals (Mickelin et al., 2019, Hawkins et al., 2021).
• Adaptive and hierarchical methods: Blockwise or hierarchical partitioning (as in HODLR or hierarchical adaptive low-rank (HALR) formats) targets local low-rankness, yielding O(n log n) storage for matrices with hierarchical low-rank off-diagonal blocks. Adaptive hierarchical compression is well suited for PDEs with localized features and for large-residual dynamics (Kaye et al., 2020, Massei et al., 2021).

3. Error Analysis, Control, and Theoretical Guarantees

Fundamental to low-rank compressed representations is controlling the approximation error. For matrix case, the Eckart–Young theorem provides optimality in $r$9. For tensors, analogous results are weaker due to the lack of optimal low-rank tensor truncation (NP-hardness in general), but quasi-optimality via greedy subspace projections or alternating minimization is obtained in practice (Mickelin et al., 2019).

Advanced methods for selecting the rank include adaptive thresholding to reach energy cutoff (sum of singular values), global or local residual analysis, or hyperparameter tuning based on the trade-off between compression ratio and clustering error in quantized and clustered models (Zhu et al., 2022, Hamlomo et al., 13 May 2025). Hierarchical and multiresolution formats can exploit scale-local rank adaptation (Mickelin et al., 2019, Massei et al., 2021).

For compressed sensing and matrix completion, information-theoretic limits are explicitly quantified—exact recovery is achieved provided the measurement operator $A = U V^T$0 satisfies a restricted isometry property (RIP) on rank-$A = U V^T$1 matrices, with sharp bounds such as $A = U V^T$2 for $A = U V^T$3 (Cai et al., 2013).

4. Combinations with Quantization, Sparsity, and Nonconvex Penalties

Real-world deployment of low-rank compressed representations often requires integration with quantization (for deployment at low hardware precision), sparsity (to represent local anomalies or background-foreground separation), or both:

• Low-rank + quantization: Randomized factorization followed by low-precision quantization of factor matrices (e.g., 1–4 bits per entry) (Saha et al., 2023), or hybrid schemes as in LR$A = U V^T$4VQ and Palu for model and KV cache compression in neural networks (Zhu et al., 2022, Chang et al., 2024).
• Low-rank + sparsity: Additive or masking combinations of low-rank tensor decompositions with sparse pruning optimize both coarse and fine structure. This achieves Pareto-optimal compression/accuracy on modern networks, though the net benefit over pure pruning can be marginal in already highly factorized architectures (Hawkins et al., 2021, Tanner et al., 2020).
• Nonconvex low-rank regularization: To overcome nuclear-norm (convex) biases such as over-shrinking of singular values, nonconvex surrogates (e.g., $A = U V^T$5, logarithmic, MCP, SCAD penalties) are used within group-sparse and ADMM frameworks for image recovery and compressive sensing (Li et al., 2019).

5. Applications and Practical Workflows

Low-rank compressed representations underpin numerous state-of-the-art techniques:

• Model compression and acceleration: Reparameterizing neural network weight tensors as low-rank factors or low-rank + quantized representations yields substantial memory and latency reductions with minimal accuracy drop (Zhu et al., 2022, Hawkins et al., 2021, Lee et al., 2019, Chang et al., 2024, Liu et al., 2024). Palu achieves up to $A = U V^T$6 speedup on LLM attention modules by compressing the key–value cache over the hidden dimension using low-rank projections and custom operator fusion (Chang et al., 2024).
• Scientific computing and PDEs: Hierarchical low-rank compression reduces memory and compute from $A = U V^T$7 and $A = U V^T$8 to $A = U V^T$9 and $$U\in\mathbb R^{m\times r}, V\in\mathbb R^{n\times r}$$0, respectively, enabling fast simulation of discretized PDEs and density matrix flows with evolving local features (Kaye et al., 2020, Massei et al., 2021).
• Signal/image processing: Adaptive, multiscale, and cluster-wise SVDs are superior to global low-rank decompositions when local variability is high, as in patch-based LoRMA for medical imaging, which yields higher PSNR, SSIM, and edge preservation indices compared to global SVD compression (Hamlomo et al., 13 May 2025, Mickelin et al., 2019). Robust low-rank models combining cosparsity and Schatten-0/nuclear-norm regularization improve compressed-sensing of EEG and hyperspectral signals (Liu et al., 2015, Zhu, 2020).
• Scientific data analysis: In electronic structure, quantum chemistry, and molecular simulation, low-rank compressed 2-electron reduced density matrices (2RDMs) admit large storage savings (quartic to quadratic scaling), facilitating many-body calculations at scale [(Atalar et al., 11 May 2026), abstract].

6. Advanced Topics: Hierarchical, Structured, and Adaptive Formats

Recent developments expand low-rank compression beyond naive factorization:

• Hierarchical matrix/tensor formats: HODLR, hierarchical adaptive low-rank (HALR), and multiresolution tensor decompositions enable adaptive storage and blockwise compression tailored to local structure, leveraging recursion and tree-based partitioning, often with local adaptivity and recompression (Kaye et al., 2020, Massei et al., 2021, Mickelin et al., 2019).
• Structured parameterizations: Low displacement rank (LDR) matrices generalize classical convolutional and Toeplitz formats, permitting both shift-invariant and more general operator forms with learnable displacement generators and efficient Krylov-type reconstructions (Thomas et al., 2018).
• Hybrid subspace–sparsity models: Sparse Power Factorization (SPF) reconstructs simultaneously sparse and low-rank signals from near-optimal numbers of measurements, outperforming convex mixed-norm relaxations for matrices that are both row-sparse and low-rank (Lee et al., 2013).

7. Theoretical and Empirical Trade-offs

The effectiveness of low-rank compressed representation is governed by fundamental and practical trade-offs:

• Compression ratio vs. accuracy: The parameter choices—target rank(s), patch size, number of clusters, degree of quantization—directly determine compressed size and error, with precise error bounds given for various settings (e.g., $$U\in\mathbb R^{m\times r}, V\in\mathbb R^{n\times r}$$1 in LPLR (Saha et al., 2023), adaptive LoRMA compression factor scaling with patch size (Hamlomo et al., 13 May 2025)).
• Computation vs. storage: While SVD-based methods are storage-optimal, their $$U\in\mathbb R^{m\times r}, V\in\mathbb R^{n\times r}$$2 cost motivates randomized or sketching-based approaches at modest additional error (Kaloorazi et al., 2018, Saha et al., 2023).
• Global vs. adaptive compression: Uniform global SVD can introduce block artifacts in images with local heterogeneity; patchwise or clusterwise adaptation allows preservation of sharp/important structures while permitting strong compression on smoother regions (Hamlomo et al., 13 May 2025, Mickelin et al., 2019).
• Limitations and failure cases: The benefit from low-rank compression diminishes for data lacking strong spectral decay or for neural layers already engineered for factorization (e.g., depthwise separable convs in MobileNet/EfficientNet) (Hawkins et al., 2021). Overregularization via convex proxies (nuclear norm) can degrade perceptual quality by over-shrinking dominant singular values (Li et al., 2019).

---

In summary, low-rank compressed representation is a foundational technology for efficient storage, transmission, and computation on high-dimensional data. It grounds diverse algorithmic strategies—ranging from direct SVD/tensor-train decomposition to randomized sketching and hierarchical adaptive factorization. These approaches are theoretically justified by optimality and sample complexity results, adapt flexibly to quantization and sparsity layers, and underpin a wide range of practical applications across machine learning, scientific computing, and signal processing (Cai et al., 2013, Kaloorazi et al., 2018, Zhu et al., 2022, Kaye et al., 2020, Mickelin et al., 2019, Hawkins et al., 2021, Hamlomo et al., 13 May 2025, Massei et al., 2021, Chang et al., 2024, Liu et al., 2024, Saha et al., 2023, Lee et al., 2013, Li et al., 2019, Liu et al., 2015, Lee et al., 2019, Mickelin et al., 2019, Rogers et al., 1 Oct 2025, Zhu, 2020).