Nonnegative Tensor Factorization (NTF)

• Nonnegative Tensor Factorization (NTF) is a method that decomposes nonnegative tensors into additive, low-rank components while preserving the intrinsic nonnegativity of data.
• NTF employs optimization objectives such as the Frobenius norm or Bregman divergences, ensuring the existence of well-posed and interpretable factorization solutions.
• NTF facilitates practical, parts-based representations with applications in chemometrics, signal processing, and bioinformatics for latent-variable modeling.

Nonnegative Tensor Factorization (NTF) is the approximation of a nonnegative tensor by a sum or multilinear transformation of nonnegative low-rank components, optimizing over a nonnegative constraint set and typically employing objectives such as the Frobenius norm or Bregman divergences. NTF generalizes the concept of Nonnegative Matrix Factorization (NMF) to higher-dimensional arrays, offering uniquely interpretable, parts-based decompositions that respect the intrinsic nonnegativity of data encountered in chemometrics, signal processing, pattern mining, and latent-variable modeling. Theoretical results establish the universal existence of best nonnegative low-rank tensor approximations for broad classes of losses, in contrast to the potentially ill-posed unconstrained tensor approximation problem.

1. Problem Formulation and Mathematical Structures

Let $$T \in \mathbb{R}_+^{I_1 \times I_2 \times \dots \times I_N}$$ be an $N$-way nonnegative tensor. NTF seeks a decomposition of $T$ as a sum of $R$ nonnegative rank-1 terms: $$T \approx X = [\![\lambda; A^{(1)}, \dots, A^{(N)}]\!] = \sum_{r=1}^R \lambda_r\, a^{(1)}_r \otimes a^{(2)}_r \otimes \dots \otimes a^{(N)}_r,$$ with nonnegative factors $A^{(n)} \in \mathbb{R}_+^{I_n \times R}$ and weights $\lambda \in \mathbb{R}_+^R$. This corresponds to the nonnegative CP (CANDECOMP/PARAFAC) decomposition.

The NTF objective is to minimize a dissimilarity measure $d(T, X)$ over the nonnegative orthant: $$\min_{\lambda \ge 0,\; A^{(n)} \ge 0}~ d(T,\;[\![\lambda;A^{(1)},\dots,A^{(N)}]\!]).$$

Important choices of $d$ include:

• Frobenius norm: $$d_F(T,X) = \|T - X\|_F^2 = \sum_{i_1 \dots i_N} (T_{i_1 \dots i_N} - X_{i_1 \dots i_N})^2$$.
• Bregman divergences, e.g., Generalized Kullback-Leibler: $D_\phi(T\|X)$ with $$\phi(T) = \sum T_{i_1\dots i_N} \log T_{i_1\dots i_N}$$ (0903.4530).

The Bregman divergence generalizes the loss in probabilistic models where NTF is interpreted as maximum likelihood estimation for naive Bayes or mixture models.

Variational Formulations

• Frobenius norm NTF:

$$f_R(T) = \min_{\lambda\ge0,\,A^{(n)}\ge0} \| T - [\![\lambda;A^{(1)},\dots,A^{(N)}]\!] \|_F^2$$

• Bregman divergence NTF:

$$\min_{\lambda\ge0,\,A^{(n)}\ge0} D_\phi \left(T\,\Vert\,[\![\lambda;A^{(1)},\dots,A^{(N)}]\!]\right).$$

2. Existence, Properties, and Theoretical Results

A foundational result establishes that the set of nonnegative tensors of nonnegative rank at most $R$ is closed in the standard topology, and for any continuous norm or suitably regular Bregman divergence, a best nonnegative rank-$R$ approximation exists (0903.4530). This sharply contrasts with general tensor approximation, where best low-rank approximations may not exist due to PARAFAC degeneracy.

• Existence theorem (Lim–Comon, 2009): For any norm or continuous dissimilarity $d$, the nonnegative rank-$R$ approximation problem has an optimal solution.
• No border rank gap: Unlike in the unconstrained setting, a tensor of nonnegative rank $>R$ cannot be arbitrarily well approximated by nonnegative rank-$R$ tensors.

The result is robust under the addition of regularization terms or when the feasible set is further constrained (e.g., orthogonality, sparsity) as long as the feasible region remains compact.

3. Algorithmic Methods and Computational Aspects

Most practical NTF algorithms are iterative and structured to respect nonnegativity. The choice of objective and structure determines the update rules and convergence behavior.

• Multiplicative Update (MU) Rules: Extension of Lee–Seung's updates, applicable to both CP and Tucker NTF under either Frobenius or KL/Bregman loss (0903.4530). Guarantee nonincreasing objective and, given the existence theorem, have well-posed convergence to stationary points.
• Alternating Nonnegative Least Squares (ANLS): Block-coordinate descent on individual factors, solving nonnegative least-squares at each block update.
• Hierarchical Elimination Algorithms: For symmetric tensors—especially completely positive (CP) tensors—algorithms exploit strong symmetry and hierarchical dominance to obtain explicit symmetric nonnegative decompositions (Qi et al., 2013).

Algorithmic progress, continuity, and boundedness of level sets together guarantee that limit points of such iterative algorithms are stationary points, and that iterates do not diverge (i.e., no "components blowing up to infinity" as seen with unconstrained CP/PARAFAC).

4. Extensions: Structure, Symmetry, and Divergence Classes

NTF theory naturally accommodates:

• Symmetric NTF and CP tensors: A symmetric nonnegative tensor with a symmetric nonnegative rank decomposition is called completely positive (CP). CP tensors admit nonnegative decompositions with spectral properties: all H-eigenvalues and, for even order, Z-eigenvalues are nonnegative. CP and copositive tensor cones form duals (Qi et al., 2013). For strongly symmetric and hierarchically dominated tensors, efficient algorithms can construct such decompositions.
• Structured constraints: Sparsity, smoothness, and orthogonality can be integrated, with theoretical existence preserved as long as the feasible region remains closed and bounded (0903.4530).
• Divergence measures beyond Bregman: The framework accommodates general Bregman divergences; open questions remain regarding more general divergences (e.g., $\alpha$-divergences).

A recurring open question is the uniqueness and identifiability of the nonnegative tensor decomposition. While existence is guaranteed, uniqueness is not: distinct parameterizations can yield the same nonnegative rank-$R$ tensor, especially in the absence of additional constraints or genericity conditions.

5. Impact on Applications and Interpretability

NTF is extensively used in chemometrics, bioinformatics, signal processing, document and topic modeling, among others, due to its guaranteed existence properties and the interpretability of nonnegative factors:

• In chemometrics and bioinformatics, NTF's parts-based representations map onto physically meaningful sources or pathways.
• In text and data mining, NTF models yield interpretable topics or latent factors with direct relevance to observed behavior.
• The closure and non-degeneracy properties underpin the empirical reliability of NTF applications (0903.4530).

The positive spectral character (all nonnegative H-eigenvalues) of CP tensors provides an additional layer of physical interpretability and theoretical assurance in applications requiring symmetric factorizations (Qi et al., 2013).

6. Open Problems and Future Research Directions

Several significant theoretical and computational challenges remain:

• Uniqueness and identifiability: Characterizing precise generic or structural conditions under which NTF admits unique decompositions.
• Algorithmic convergence rates: Quantitative convergence analysis under different losses and structural constraints.
• Extension to new divergences: Impact of alternative divergence measures beyond Bregman, and their implications for theory and computation.
• Structured and regularized factorizations: Integration of through-mode constraints (sparsity, smoothness, orthogonality) while retaining existence and convergence guarantees (0903.4530).
• Efficient large-scale solvers: Scalable, parallel, and structure-exploiting algorithms for modern big data settings.

Empirical success motivates deeper exploration of identifiability, scalability, and structured regularization to further expand the applicability and theoretical foundation of nonnegative tensor factorization.

---

References:

• Nonnegative approximations of nonnegative tensors (0903.4530)
• Nonnegative Tensor Factorization, Completely Positive Tensors and an Hierarchical Elimination Algorithm (Qi et al., 2013)