---
title: Einsum-Based Multiplicative Update Framework
url: https://www.emergentmind.com/topics/einsum-based-multiplicative-update-framework
type: topic
---

# Einsum-Based Multiplicative Update Framework

The einsum-based multiplicative update framework is a unifying approach for fitting a broad class of nonnegative tensor factorizations using majorization–minimization (MM) and accelerated by efficient tensor contraction (“einsum”) operations. This framework generalizes standard CP and Tucker multiplicative updates to any model expressible as a tensor contraction, accommodates a spectrum of loss functions including the full $(\alpha, \beta)$-divergence family, and scales to extremely large, sparse, or incomplete tensors. It provides a modeling interface that allows users to specify arbitrary contraction patterns via simple model strings, facilitating flexible, domain-tailored decompositions without manual derivation of update rules [2602.02759], [2602.14683].

## 1. Mathematical Foundations and Model Generality

Let $Y \in \mathbb{R}_{\ge0}^{I_1 \times \cdots \times I_M}$ denote an $M$-mode observed nonnegative tensor. The objective is to find a nonnegative approximation $\widehat Y$ minimizing an entrywise sum of divergence losses:
\[
L(Y, \widehat Y) = \sum_{i_1,\dots,i_M} d(y_{i_1\cdots i_M}, \widehat y_{i_1\cdots i_M}),
\]
subject to $\widehat y_{i_1\cdots i_M} \ge 0$.

The factorization is described by a set of nonnegative tensors $\Theta^{(1)},\ldots,\Theta^{(L)}$, each affiliated with observed and latent index sets. The generative model for tensor entries is formulated as
\[
\widehat y_{\text{obs}} = \sum_{\text{lat}} \prod_{\ell=1}^L \theta^{(\ell)}_{\text{obs}_\ell, \text{lat}_\ell},
\]
where $\text{obs}$ are observed indices, $\text{lat}$ are latent (summed) indices, and $\text{obs}_\ell,\text{lat}_\ell$ are the index sets for $\Theta^{(\ell)}$. This structure subsumes standard decompositions (CP, Tucker, tensor-train) and custom contraction models.

The einsum framework specifies this via a string, e.g., `"i r, j r, k r -> i j k"` for a 3-way CP factorization, allowing automated construction of contraction and update routines [2602.02759].

## 2. Loss Functions and $(\alpha, \beta)$-Divergences

A key feature is support for a wide class of loss functions:
\[
L_{\alpha, \beta}(Y, \widehat Y) = \sum_{i_1,\dots,i_M} d_{\alpha,\beta}(y_{i_1\cdots i_M}, \widehat y_{i_1\cdots i_M}),
\]
with the $(\alpha, \beta)$-divergence defined as:
\[
d_{\alpha, \beta}(x, y) =
\begin{cases}
\frac{1}{\alpha\beta}\left[\frac{\alpha}{\alpha+\beta}x^{\alpha+\beta} + \frac{\beta}{\alpha+\beta}y^{\alpha+\beta} - x^\alpha y^\beta\right], & \alpha, \beta, \alpha+\beta \neq 0 \\
\frac{1}{\beta^2}[\beta y^\beta \log\frac{y}{x} - y^\beta + x^\beta], & \alpha=0, \beta\neq0 \\
\frac12(\log x - \log y)^2, & \alpha=\beta=0
\end{cases}
\]
This formulation unifies Frobenius (squared-Euclid), Kullback–Leibler, Itakura–Saito, and other divergences, enabling choice according to the generative assumptions or application [2602.02759], [2602.14683].

## 3. Multiplicative Update Derivation via Majorization–Minimization

The MM methodology constructs a tight surrogate for the loss over current factors, then analytically minimizes it for each block update:
- The divergence is decomposed into convex and concave parts with respect to $y$.
- Using the essential “decomposability property,” the resulting update for each $\Theta^{(\ell)}$ takes the multiplicative form:
\[
\Theta^{(\ell)} \leftarrow \Theta^{(\ell)} \odot g^{-1}\left( \frac{A}{B} \right),
\]
where $A, B$ are contractions (einsum over factors and auxiliary tensors $a(Y,\widehat Y)$, $b(Y,\widehat Y)$ derived from the loss), $g$ is invertible and loss-dependent, and einsum strings are constructed by swapping the current factor’s indices with model outputs [2602.02759].

For CP and Tucker models under $\beta$-divergence, these updates reduce to repeated einsum operations with powers and masks, as formalized in the block-MM and joint-MM pseudocode [2602.14683].

| Model             | Update Formulation                        | Einsum Example (CP, 3-way)                    |
|-------------------|-------------------------------------------|-----------------------------------------------|
| Block-MM          | $A^{(n)} \leftarrow A^{(n)} \odot (Num/Den)^{\gamma(\beta)}$ | `einsum('ijk,jr,kr->ir', P, A2, A3)`          |
| Joint-MM          | Reference-cached surrogates, inner updates | Uses $\chi_{1,\beta}, \chi_{2,\beta}$          |

## 4. Algorithmic Implementation and Computational Aspects

The implementation involves the following steps:
1. User specifies $Y$, a model string (`model_str`), $\alpha,\beta$ (divergence), and initial $\Theta^{(1:L)}$.
2. Precompute einsum strings for each factor.
3. Iterate until convergence:
    a. Compute reconstruction $\widehat Y = \texttt{einsum}(model\_str, \Theta^{(1)}, \ldots, \Theta^{(L)})$.
    b. For each factor, compute numerators and denominators using auxiliary functions $a(\cdot), b(\cdot)$ and einsum contractions.
    c. Apply the block update formula.
4. On completion, factors are returned; loss, iterations, and wall time can be monitored [2602.02759].

Efficient implementation leverages hardware-optimized einsum routines (e.g., via BLAS or GPU backends), yielding per-iteration costs linear in the number of nonzeros plus model parameters. In practice, the method scales to tensors with hundreds of millions of entries in seconds.

Empirical complexity benchmarks demonstrate that einsum-based updates typically outperform classical unfolding-based routines (MTTKRP/Khatri-Rao), with speedups up to $2.5\times$ in single-thread benchmarks and even greater acceleration when parallelized [2602.14683].

## 5. Treatment of Missing Data

Handling of missing data is implemented by introducing a binary mask $\mathcal M \in \{0,1\}^{I_1 \times \cdots \times I_M}$:
\[
Y \mapsto \mathcal M \odot Y, \quad \widehat Y \mapsto \mathcal M \odot \widehat Y
\]
in all loss and update computations. Masked-out entries are omitted from all surrogate contractions, efficiently excluding them from the optimization [2602.02759].

This mechanism easily integrates with the einsum-based contraction pattern, requiring only one additional masking operation with negligible computational overhead.

## 6. Interface, Model Specification, and Extendability

The framework provides a user interface where models are specified by einsum-compatible strings. The user selects or defines the loss (e.g., via $(\alpha,\beta)$), and optionally custom-initializes factor tensors. Batch runs, convergence settings, restarts, and validation-based early stopping are supported.

Beyond standard CP and Tucker factorizations, this enables custom or hybrid models—e.g., non-halting, block-structured, domain-aware contractions—without solver modification. Loss functions can be extended, provided that the convex–concave decomposability and invertibility conditions are met. Extensions include:
- Discrete and count losses (Bernoulli, negative binomial)
- Structured sparsity via explicit $\ell_0$ or $\ell_1$ penalties
- Probabilistic or Bayesian surrogates (Polya-urn MM updates)
- Hybridization with automatic differentiation for differentiable submodules [2602.02759]

## 7. Empirical Performance and Limitations

On large real-world tensors such as spatio-temporal Uber pickups, ICEWS event counts, and WITS trade flows, einsum-based MU (notably, the NNEinFact package) consistently converged up to $90\times$ faster than Adam and standard gradient routines, achieved over $37\%$ reductions in held-out divergence relative to off-the-shelf CP/Tucker models, and produced more interpretable latent components (e.g., morning commutes, nightlife). Qualitative patterns are more sharply resolved with custom domain-encoded contractions [2602.02759].

Limitations include:
- Requirement that loss admits a suitable convex–concave, decomposable surrogate.
- Convergence only to stationary points or blockwise local minima, with sensitivity to initialization; multiple restarts are recommended.
- Failure or divergence if auxiliary ratios $A/B$ escape the invertibility domain of $g$.
- For the joint-MM method, while each inner step decreases the surrogate, asymptotic convergence of the joint sequence is not guaranteed in all cases [2602.14683].

A plausible implication is that the einsum-based multiplicative update paradigm will subsume many bespoke methods in nonnegative multiway modeling, provided loss decomposability and contraction structures are carefully verified.

## References

- "Near-Universal Multiplicative Updates for Nonnegative Einsum Factorization" [2602.02759]
- "Joint Majorization-Minimization for Nonnegative CP and Tucker Decompositions under $β$-Divergences: Unfolding-Free Updates" [2602.14683]

Source: https://www.emergentmind.com/topics/einsum-based-multiplicative-update-framework