Tensor Decomposition & Low-Rank Factorization

• Tensor decomposition is a method to express high-dimensional arrays as combinations of low-dimensional factors under low‐rank constraints, facilitating dimensionality reduction and interpretability.
• Techniques such as CP, Tucker, and Tensor-Train decompositions use alternating optimization and structured regularization to ensure accuracy and uniqueness in model extraction.
• These methods enable practical applications in clustering, tensor completion, and neural network compression by providing scalable and efficient solutions to high-dimensional problems.

A tensor decomposition expresses a high-order tensor as a combination of lower-dimensional factors, often under explicit low-rank constraints. These decompositions and associated low-rank factorizations are foundational tools in computational mathematics, statistical signal processing, machine learning, and scientific computing. They provide a framework for dimensionality reduction, latent variable discovery, model compression, matrix and tensor completion, and structural interpretation of multi-way data. The central challenge is to extract parsimonious and interpretable multilinear models from high-dimensional arrays, subject to the combinatorial complexity of tensor rank and varied notions of multilinear structural constraint.

1. Formal Definitions and Canonical Decomposition Models

Low-rank factorization refers to expressing a high-dimensional array (matrix or tensor) as a sum or product of factors of much smaller dimension, minimizing the number of parameters needed to represent or approximate the data.

• Matrix factorization:
  • Any $X \in \mathbb{R}^{m \times n}$ has rank-$r$ factorization $$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$.
  • The minimal such $r$ is the rank.
• Canonical Polyadic (CP) decomposition:
  • For $d$-way tensor $$\mathcal{T} \in \mathbb{R}^{I_1 \times \cdots \times I_d}$$,

  $$\mathcal{T}_{i_1,\dots,i_d} = \sum_{r=1}^R a^{(1)}_{i_1 r} a^{(2)}_{i_2 r}\cdots a^{(d)}_{i_d r}$$

  with $a^{(k)}_{i_k r} \in \mathbb{R}^{I_k}$. - Minimal $R$ is the CP rank (Borsoi et al., 25 Aug 2025).

• Tucker decomposition:

  • $$\mathcal{T} = \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_d U^{(d)}$$, core $r$0, matrices $r$1.
  • The vector $r$2 defines the multilinear (“Tucker”) rank (Borsoi et al., 25 Aug 2025).
• Tensor-Train (TT) and Tensor Ring (TR):
  • TT: Represent $r$3 by $r$4 core tensors $r$5, $r$6; chain contraction yields entries (Borsoi et al., 25 Aug 2025, Röhrig-Zöllner et al., 2021).
  • TR: Similar to TT, but the train is closed in a ring, encoding cyclic dependencies (Yuan et al., 2018, Yuan et al., 2018, Zhao et al., 2021).

Low-rank approximation minimizes a loss (typically $r$7-norm/frobenius) subject to such structures (e.g., $r$8).

2. Theoretical Foundations and Identifiability

Tensor rank is generally NP-hard to determine; no general finite algorithm exists for tensors of order $r$9 (Turchetti, 2023, Borsoi et al., 25 Aug 2025). Uniqueness properties are markedly stronger for tensors than for matrices:

• CP uniqueness: Up to scaling and permutation, the CP decomposition is unique under Kruskal's condition: $$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$0, where $$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$1, etc. (Borsoi et al., 25 Aug 2025).
• Tucker/TT/HT identifiability: Tucker decomposition is not unique without orthonormal constraints; TT unique up to internal “gauge” transformations if ranks are minimal and cores have full column rank (Borsoi et al., 25 Aug 2025).
• Exact decomposition: For self-adjoint, positive semi-definite tensor operators, a constructive spectral-type (eigen-) decomposition yields an exact finite expansion (Turchetti, 2023).
• Symmetric tensor decomposition: Symmetric rank is tied to the low-rank completion of structured (Hankel) matrices, with unique solutions generically for symmetric tensors (Ishteva et al., 2013).

3. Algorithmic Paradigms: Classical and Modern Approaches

3.1 Alternating Optimization and Matrix/Tensor Factorization

ALS (Alternating Least Squares):

• CP and Tucker decompositions are commonly computed by block coordinate descent: fix all but one factor and solve a swept least-squares problem (Minster et al., 2021, Pearce et al., 4 Dec 2025).
• For CP, this reduces to updating each factor $$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$2 from mode-$$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$3 unfolding and Khatri-Rao products (Minster et al., 2021).
• Convergence is guaranteed to stationary points for fixed regularization/penalty, but can be slow under non-convexity or collinearity.

Structured low-rank approximation:

• Affinely constrained matrices (e.g., Hankel, Sylvester, Toeplitz) are approximated via penalized factorization approaches. Structure is enforced by projection penalties alongside low-rank (factorization) constraints:

$$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$4

(Ishteva et al., 2013).

Symmetric/structured tensor decomposition:

• Quasi-Hankel constructions “unfold” symmetric tensors into matrices, reducing symmetric tensor rank decomposition to structured low-rank matrix completion (Ishteva et al., 2013).

3.2 Regularization and Surrogates

• Nuclear norm relaxation: The sum of singular values is the tightest convex lower bound of rank over the unit ball. Useful in both matrix and tensor settings.
• Schatten $$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$5-norms / latent space nuclear norm: Imposed on unfolded cores or factors (e.g., in TR decomposition) to induce low-rankness at the latent rather than global data level, greatly reducing computational scale (Yuan et al., 2018, Yuan et al., 2018).
• Sparsity-inducing regularization: Sparse cores (e.g., $$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$6 penalty on Tucker or CP core) promote models that are simultaneously low-rank and sparse, further improving data-adaptation and compression (Pan et al., 2020).
• Manifold and hypergraph regularization: Explicit Laplacian or hyper-Laplacian terms extract nonlinear local structure in embeddings, e.g., in multi-view clustering (Yu et al., 2023, Zhao et al., 2021).

4. Randomized, Sampling-based, and Scalable Algorithms

The computational burden of classical decompositions motivated a wave of randomized algorithms:

• Random projections and sketching: Gaussian/SRHT/CountSketch-based projections reduce problem dimension for large-scale SVD or ALS subproblems, with rigorous error bounds (Pearce et al., 4 Dec 2025).
• Leverage-score and ridge-leverage sampling: Used to sample informative tensor/khatri-rao product rows for fast least-squares in CP/Tucker ALS (Fahrbach et al., 2021, Pearce et al., 4 Dec 2025). With Kronecker structure, multi-way leverage scores factor and can be efficiently sampled.
• TensorSketch for TT/CP/Tucker: FFT-accelerated randomized embeddings efficiently approximate subproblem least-squares in TT/CPALS and HOOI, with per-iteration cost nearly independent of ambient dimension (Chen et al., 2023, Pearce et al., 4 Dec 2025).
• Randomized SVD/ID/CUR for matrices and unfolded tensors: Range sketching, interpolative and CUR decompositions enable sublinear scaling in both storage and runtime, matching or nearly matching best deterministic error rates (Pearce et al., 4 Dec 2025).
• Probabilistic low-rank factorization for tensor networks: Randomized SVD outperforms deterministic TSVD in truncating bond dimensions in MPS/TTN simulations, yielding substantial speedups without loss of physical accuracy (Kohn et al., 2017).

Randomization Strategy	Application	Complexity Gain
Rangefinder SVD/ID	Matrix SVD, mode-$$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$7 unfoldings (Tucker/CP)	$$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$8 for rank-$$X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$$9
TensorSketch	TT/CP/Tucker ALS (CPALS, HOOI)	$r$0–$r$1 per core
Ridge-leverage sampling	Tucker ALS core updates	$r$2 samples

Randomized methods thus yield order-of-magnitude speedups for high order, large scale, and high dimensional problems, with theoretical and practical accuracy trade-offs precisely characterized (Pearce et al., 4 Dec 2025).

5. Applications: Clustering, Completion, Compression, and Learning

• Multi-view and high-order clustering: Low-rank tensor representations aggregate high-order correlation across data views, outperforming matrix-based methods on clustering accuracy due to high expressiveness in capturing nonlinear structures (Yu et al., 2023, Zhao et al., 2021).
• Tensor completion and denoising: Low-rank models, with or without sparse regularization, are effective for imputation under severe missingness (90%–95%), outperforming comparable nuclear-norm or matrix-factorization approaches (Pan et al., 2020, Yuan et al., 2018, Yuan et al., 2018, Yu et al., 2022). Double tubal rank (t-SVD + mode-3) captures additional structure and accelerates convergence (Yu et al., 2022).
• Model and neural network compression: SVD, TT, and Tucker decompositions are deployed in layerwise compression for CNNs, LSTMs, and transformers. Sequential freezing and quantization make low-rank decomposition attractive for both training and inference latency/throughput, at ≤0.5% accuracy drop for significant parameter reduction (Hajimolahoseini et al., 2024, Borsoi et al., 25 Aug 2025).
• Signal and image processing: Spectral CT, MRI, and video compressive sensing benefit from local cube-based tensor factorization (KBR) and deep network–inspired low-rank surrogates for superior SNR and structure preservation (Wu et al., 2018, Saragadam et al., 2022).
• Theory of deep learning: Tensor decompositions underpin theoretical advances in the expressivity, learnability, and generalization analysis of deep architectures, including the exponential expressivity gap between deep (TT/Tensor Hierarchies) and shallow (CP) architectures (Borsoi et al., 25 Aug 2025).

6. Optimization, Robustness, and Numerical Issues

• ADMM and Split-Bregman frameworks: Modular block optimization with closed/proximal-form updates are widely used, exploiting separability of nuclear and sparsity penalties. These schemes guarantee monotonic objective decrease, robustness to initialization, and efficient convergence (Yuan et al., 2018, Yuan et al., 2018, Pan et al., 2020).
• Automatic rank selection: Nuclear norms (on latent factors or data unfoldings) and automatic relevance determination (ARD) priors in Bayesian factorizations enable adaptive, data-driven rank selection, essential for high-dimensional and high-order settings (Yuan et al., 2018, Liu et al., 2023).
• Numerical stability: Ill-conditioning endemic in ALS subproblems (especially in CP) is mitigated by QR/SVD-based solvers, with provable backward error improvements and superior recovery under high collinearity (Minster et al., 2021).
• Scaling and software: Memory-bound performance models (Roofline) dictate that advanced kernels (Q-less TSQR, fused reshape/matmul) are essential for high throughput—computation can be as cheap as a few passes through data (Röhrig-Zöllner et al., 2021, Pearce et al., 4 Dec 2025).

7. Extensions and Emerging Directions

• Probabilistic and Bayesian frameworks: LMH-BRTF and similar methods integrate explicit sparse+Gaussian noise priors with ARD for multi-rank determination, extending robust PCA innovation to higher-order tensors (Liu et al., 2023).
• Deep generative network priors: DeepTensor replaces classical tensor factors by outputs of untrained deep generative networks, achieving implicitly regularized, nonlinear low-rank factorization with strong robustness to non-Gaussian noise and high compression (Saragadam et al., 2022).
• Fast nonnegative and structured decompositions: Multiplicative updates under nonnegativity, hypergraph/graph Laplacian regularization, and low-rank “tricks” (e.g., Tucker precomputation) yield scalable clustering and manifold learning for high-dimensional nonnegative tensors (Zhao et al., 2021).
• Exact algebraic and polynomial-constraint methods: Generating polynomial methods recover CP decompositions and near-optimal low-rank approximations for nearly low-rank tensors by solving structured linear systems and eigenproblems (Nie et al., 2022, Turchetti, 2023).

---

The theory and computational methodology of tensor decomposition and low-rank factorization constitute a mature but rapidly evolving field, combining algebraic, analytic, numerical, and statistical insights. Future advances are expected in adaptive rank/model selection, parallel and high-throughput computation, robust multimodal modeling, and deeper integration of domain-specific structure (e.g., graph, hypergraph, neural priors) into scalable low-rank frameworks.