---
title: Tensor Decomposition & Low-Rank Factorization
url: https://www.emergentmind.com/topics/tensor-decomposition-and-low-rank-factorization
type: topic
---

# Tensor Decomposition & Low-Rank Factorization

A tensor decomposition expresses a high-order tensor as a combination of lower-dimensional factors, often under explicit low-rank constraints. These decompositions and associated low-rank factorizations are foundational tools in computational mathematics, statistical signal processing, machine learning, and scientific computing. They provide a framework for dimensionality reduction, latent variable discovery, model compression, matrix and tensor completion, and structural interpretation of multi-way data. The central challenge is to extract parsimonious and interpretable multilinear models from high-dimensional arrays, subject to the combinatorial complexity of tensor rank and varied notions of multilinear structural constraint.

## 1. Formal Definitions and Canonical Decomposition Models

**Low-rank factorization** refers to expressing a high-dimensional array (matrix or tensor) as a sum or product of factors of much smaller dimension, minimizing the number of parameters needed to represent or approximate the data.

- **Matrix factorization**:
  - Any $X \in \mathbb{R}^{m \times n}$ has rank-$r$ factorization $X = UV^\top,\;\; U \in \mathbb{R}^{m \times r},\; V \in \mathbb{R}^{n \times r}$.
  - The minimal such $r$ is the rank.
- **Canonical Polyadic (CP) decomposition**:
  - For $d$-way tensor $\mathcal{T} \in \mathbb{R}^{I_1 \times \cdots \times I_d}$,
    $$
    \mathcal{T}_{i_1,\dots,i_d} = \sum_{r=1}^R a^{(1)}_{i_1 r} a^{(2)}_{i_2 r}\cdots a^{(d)}_{i_d r}
    $$
    with $a^{(k)}_{i_k r} \in \mathbb{R}^{I_k}$.
  - Minimal $R$ is the CP rank [2508.18408].
- **Tucker decomposition**:
  - $\mathcal{T} = \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_d U^{(d)}$, core $\mathcal{G}\in\mathbb{R}^{R_1\times\cdots\times R_d}$, matrices $U^{(k)}\in\mathbb{R}^{I_k\times R_k}$.
  - The vector $(R_1,\dots,R_d)$ defines the multilinear (“Tucker”) rank [2508.18408].
- **Tensor-Train (TT) and Tensor Ring (TR)**:
  - TT: Represent $\mathcal{T}$ by $d$ core tensors $G^{(k)}\in\mathbb{R}^{r_{k-1}\times I_k \times r_k}$, $r_0=r_d=1$; chain contraction yields entries [2508.18408, 2102.00104].
  - TR: Similar to TT, but the train is closed in a ring, encoding cyclic dependencies [1809.02288, 1805.08468, 2109.02314].

Low-rank approximation minimizes a loss (typically $\ell_2$-norm/frobenius) subject to such structures (e.g., $\min_{\operatorname{rank}(X) \le r} \|D-X\|$).

## 2. Theoretical Foundations and Identifiability

**Tensor rank** is generally NP-hard to determine; no general finite algorithm exists for tensors of order $d\geq3$ [2309.07819, 2508.18408]. Uniqueness properties are markedly stronger for tensors than for matrices:

- **CP uniqueness**: Up to scaling and permutation, the CP decomposition is unique under Kruskal's condition: $k_A + k_B + k_C \geq 2R+2$, where $k_A = \text{krank}(A)$, etc. [2508.18408].
- **Tucker/TT/HT identifiability**: Tucker decomposition is not unique without orthonormal constraints; TT unique up to internal “gauge” transformations if ranks are minimal and cores have full column rank [2508.18408].
- **Exact decomposition**: For self-adjoint, positive semi-definite tensor operators, a constructive spectral-type (eigen-) decomposition yields an exact finite expansion [2309.07819].
- **Symmetric tensor decomposition**: Symmetric rank is tied to the low-rank completion of structured (Hankel) matrices, with unique solutions generically for symmetric tensors [1308.1827].

## 3. Algorithmic Paradigms: Classical and Modern Approaches

### 3.1 Alternating Optimization and Matrix/Tensor Factorization

**ALS (Alternating Least Squares):**
- CP and Tucker decompositions are commonly computed by block coordinate descent: fix all but one factor and solve a swept least-squares problem [2112.10855, 2512.05286].
- For CP, this reduces to updating each factor $A_n$ from mode-$n$ unfolding and Khatri-Rao products [2112.10855].
- Convergence is guaranteed to stationary points for fixed regularization/penalty, but can be slow under non-convexity or collinearity.

**Structured low-rank approximation**:
- Affinely constrained matrices (e.g., Hankel, Sylvester, Toeplitz) are approximated via penalized factorization approaches. Structure is enforced by projection penalties alongside low-rank (factorization) constraints:
  $$
  \min_{U,V}\;\|D-UV^\top\|_W^2 + \lambda\|UV^\top-\mathcal{P}(UV^\top)\|_F^2
  $$
  [1308.1827].

**Symmetric/structured tensor decomposition**:
- Quasi-Hankel constructions “unfold” symmetric tensors into matrices, reducing symmetric tensor rank decomposition to structured low-rank matrix completion [1308.1827].

### 3.2 Regularization and Surrogates

- **Nuclear norm relaxation**: The sum of singular values is the tightest convex lower bound of rank over the unit ball. Useful in both matrix and tensor settings.
- **Schatten $p$-norms / latent space nuclear norm**: Imposed on unfolded cores or factors (e.g., in TR decomposition) to induce low-rankness at the latent rather than global data level, greatly reducing computational scale [1809.02288, 1805.08468].
- **Sparsity-inducing regularization**: Sparse cores (e.g., $\ell_1$ penalty on Tucker or CP core) promote models that are simultaneously low-rank and sparse, further improving data-adaptation and compression [2010.00359].
- **Manifold and hypergraph regularization**: Explicit Laplacian or hyper-Laplacian terms extract nonlinear local structure in embeddings, e.g., in multi-view clustering [2304.11435, 2109.02314].

## 4. Randomized, Sampling-based, and Scalable Algorithms

The computational burden of classical decompositions motivated a wave of randomized algorithms:

- **Random projections and sketching**: Gaussian/SRHT/CountSketch-based projections reduce problem dimension for large-scale SVD or ALS subproblems, with rigorous error bounds [2512.05286].
- **Leverage-score and ridge-leverage sampling**: Used to sample informative tensor/khatri-rao product rows for fast least-squares in CP/Tucker ALS [2107.10654, 2512.05286]. With Kronecker structure, multi-way leverage scores factor and can be efficiently sampled.
- **TensorSketch for TT/CP/Tucker**: FFT-accelerated randomized embeddings efficiently approximate subproblem least-squares in TT/CPALS and HOOI, with per-iteration cost nearly independent of ambient dimension [2309.08093, 2512.05286].
- **Randomized SVD/ID/CUR for matrices and unfolded tensors**: Range sketching, interpolative and CUR decompositions enable sublinear scaling in both storage and runtime, matching or nearly matching best deterministic error rates [2512.05286].
- **Probabilistic low-rank factorization for tensor networks**: Randomized SVD outperforms deterministic TSVD in truncating bond dimensions in MPS/TTN simulations, yielding substantial speedups without loss of physical accuracy [1710.01463].

| Randomization Strategy   | Application                                  | Complexity Gain            |
|-------------------------|----------------------------------------------|----------------------------|
| Rangefinder SVD/ID      | Matrix SVD, mode-$n$ unfoldings (Tucker/CP) | $O(mnk)$ for rank-$k$      |
| TensorSketch            | TT/CP/Tucker ALS (CPALS, HOOI)               | $O(\text{rank}^2)$–$O(\text{rank}^3)$ per core |
| Ridge-leverage sampling | Tucker ALS core updates                      | $O(\text{rank} \log \text{rank})$ samples      |

Randomized methods thus yield order-of-magnitude speedups for high order, large scale, and high dimensional problems, with theoretical and practical accuracy trade-offs precisely characterized [2512.05286].

## 5. Applications: Clustering, Completion, Compression, and Learning

- **Multi-view and high-order clustering**: Low-rank tensor representations aggregate high-order correlation across data views, outperforming matrix-based methods on clustering accuracy due to high expressiveness in capturing nonlinear structures [2304.11435, 2109.02314].
- **Tensor completion and denoising**: Low-rank models, with or without sparse regularization, are effective for imputation under severe missingness (90%–95%), outperforming comparable nuclear-norm or matrix-factorization approaches [2010.00359, 1809.02288, 1805.08468, 2201.09197]. Double tubal rank (t-SVD + mode-3) captures additional structure and accelerates convergence [2201.09197].
- **Model and neural network compression**: SVD, TT, and Tucker decompositions are deployed in layerwise compression for CNNs, LSTMs, and transformers. Sequential freezing and quantization make low-rank decomposition attractive for both training and inference latency/throughput, at ≤0.5% accuracy drop for significant parameter reduction [2407.20266, 2508.18408].
- **Signal and image processing**: Spectral CT, MRI, and video compressive sensing benefit from local cube-based tensor factorization (KBR) and deep network–inspired low-rank surrogates for superior SNR and structure preservation [1807.10610, 2204.03145].
- **Theory of deep learning**: Tensor decompositions underpin theoretical advances in the expressivity, learnability, and generalization analysis of deep architectures, including the exponential expressivity gap between deep (TT/Tensor Hierarchies) and shallow (CP) architectures [2508.18408].

## 6. Optimization, Robustness, and Numerical Issues

- **ADMM and Split-Bregman frameworks**: Modular block optimization with closed/proximal-form updates are widely used, exploiting separability of nuclear and sparsity penalties. These schemes guarantee monotonic objective decrease, robustness to initialization, and efficient convergence [1809.02288, 1805.08468, 2010.00359].
- **Automatic rank selection**: Nuclear norms (on latent factors or data unfoldings) and automatic relevance determination (ARD) priors in Bayesian factorizations enable adaptive, data-driven rank selection, essential for high-dimensional and high-order settings [1809.02288, 2311.05888].
- **Numerical stability**: Ill-conditioning endemic in ALS subproblems (especially in CP) is mitigated by QR/SVD-based solvers, with provable backward error improvements and superior recovery under high collinearity [2112.10855].
- **Scaling and software**: Memory-bound performance models (Roofline) dictate that advanced kernels (Q-less TSQR, fused reshape/matmul) are essential for high throughput—computation can be as cheap as a few passes through data [2102.00104, 2512.05286].

## 7. Extensions and Emerging Directions

- **Probabilistic and Bayesian frameworks**: LMH-BRTF and similar methods integrate explicit sparse+Gaussian noise priors with ARD for multi-rank determination, extending robust PCA innovation to higher-order tensors [2311.05888].
- **Deep generative network priors**: DeepTensor replaces classical tensor factors by outputs of untrained deep generative networks, achieving implicitly regularized, nonlinear low-rank factorization with strong robustness to non-Gaussian noise and high compression [2204.03145].
- **Fast nonnegative and structured decompositions**: Multiplicative updates under nonnegativity, hypergraph/graph Laplacian regularization, and low-rank “tricks” (e.g., Tucker precomputation) yield scalable clustering and manifold learning for high-dimensional nonnegative tensors [2109.02314].
- **Exact algebraic and polynomial-constraint methods**: Generating polynomial methods recover CP decompositions and near-optimal low-rank approximations for nearly low-rank tensors by solving structured linear systems and eigenproblems [2208.07477, 2309.07819].

---

The theory and computational methodology of tensor decomposition and low-rank factorization constitute a mature but rapidly evolving field, combining algebraic, analytic, numerical, and statistical insights. Future advances are expected in adaptive rank/model selection, parallel and high-throughput computation, robust multimodal modeling, and deeper integration of domain-specific structure (e.g., graph, hypergraph, neural priors) into scalable low-rank frameworks.

Source: https://www.emergentmind.com/topics/tensor-decomposition-and-low-rank-factorization