Divergence Matrix Framework Overview

• Divergence Matrix Framework is a family of tools that automatically selects and optimizes divergence functions for enhanced matrix and tensor factorization.
• It formalizes divergence selection as a maximum likelihood problem, unifying β-, α-, γ-, and Rényi-divergences to improve modeling accuracy.
• MEDAL employs grid search and multiplicative updates to jointly learn matrix factors and optimal divergence parameters, validated on synthetic and real data.

The divergence matrix framework encompasses a family of mathematical and algorithmic tools designed for the selection, optimization, and automatic adaptation of information divergence functions between matrices or tensors in statistical modeling and machine learning. Its core is a disciplined procedure for learning the optimal divergence—typically from a parametric family—in tasks such as nonnegative matrix/tensor factorization, embedding, or probabilistic modeling, thus obviating arbitrary or purely heuristic selection of the divergence parameter and enabling transfer to non-separable and generalized divergences (Dikmen et al., 2014).

1. Parametric Divergence Families and Formal Structure

The framework structures divergence selection as a formal maximum-likelihood (ML) problem by leveraging the parametric forms of β-, α-, γ-, and Rényi-divergence families. For nonnegative matrices or tensors $X$ and their model approximant %%%%1%%%%:

• The pointwise β-divergence ($\beta \in \mathbb{R}$):

$$d_\beta(x\Vert\mu) = \begin{cases} \frac{x^{\beta+1} + \beta\mu^{\beta+1} - (\beta+1)x\mu^\beta}{\beta(\beta+1)} & \beta \neq 0, -1 \ x \log(x/\mu) - x + \mu & \beta = 0 \ x/\mu - \log(x/\mu) - 1 & \beta = -1 \end{cases}$$

• The matrix divergence: $$D_\beta(X\Vert Y) = \sum_{i,j} d_\beta(x_{ij}\Vert y_{ij})$$

This unifies the Euclidean, Kullback-Leibler (KL), and Itakura–Saito divergences as β = 2, 1, 0 specializations, with β continuously interpolating among generative noise models linked to the Tweedie exponential dispersion family.

2. Maximum Likelihood Divergence Selection and Augmented Tweedie Densities

Divergence selection is cast as finding the β* (and, by extension, other parameters) that maximize the likelihood of the observed data under an effective density parameterized by (μ, β, φ):

• The Tweedie density links directly to the β-divergence:

$$p_{Tw}(x; \mu, \phi, \beta) \propto f(x, \phi, \beta) \exp\left( -\frac{D_\beta(x\Vert\mu)}{\phi} \right)$$

Direct maximization of the Tweedie log-likelihood is restricted by computational intractability and nonexistence for certain β (i.e., $-1 < \beta < 0$). The exponential-divergence-with-augmentation (EDA) density circumvents these issues by introducing an augmentation term $R(x,\beta) = \frac{\beta-1}{2}\ln x$ that ensures normalizability for $\beta \in \mathbb{R}$:

$$p_{EDA}(x_i;\mu_i,\beta,\phi) = \frac{1}{Z_i} \exp\left\{R(x_i,\beta) - \frac{1}{\phi} d_\beta(x_i\Vert\mu_i)\right\}$$

Resulting in the (log-)likelihood:

$$\log L(\beta,\phi) = \sum_{i,j} \left[\frac{\beta-1}{2} \ln x_{ij} - \frac{1}{\phi} d_\beta(x_{ij}\Vert\mu_{ij}) - \ln Z(y_{ij}, \beta, \phi)\right]$$

The optimal divergence parameter is then

$$\beta^* = \arg\max_\beta \max_\phi \log L(\beta, \phi)$$

3. Extensions: α-, γ-, and Rényi Divergences

The framework generalizes to α-divergence—defined for α ≠ 0, 1 as

$$D_\alpha(x\Vert\mu) = \frac{ \sum_i x_i^\alpha \mu_i^{1-\alpha} - \alpha\sum_i x_i + (\alpha-1)\sum_i \mu_i }{ \alpha(\alpha-1) }$$

and shows an explicit entrywise reparametrization reduces α-divergence selection to the β-divergence case using

$$y_i = x_i^\alpha / \alpha^{2\alpha},\quad m_i = \mu_i^\alpha/\alpha^{2\alpha},\quad \beta = 1/\alpha - 1$$

allowing identical optimization machinery for β and thus for α.

Non-separable divergences (γ, Rényi) are handled through further reductions: minimizing D_γ or D_ρ is shown to be equivalent to optimizing a scaled or transformed β- or α-divergence, after introducing an intermediate scale parameter and alternating minimization.

4. Joint Learning Algorithm (MEDAL) and Optimization

A prototypical application is nonnegative matrix factorization (NMF), where both model (e.g., $W, H$ in $V \approx WH$) and divergence parameters (β, α, γ) are optimized in a two-stage block-coordinate framework (referred to as MEDAL):

1. Grid Search: Fix β (or α, γ) across a grid.
2. Model Fitting: For each candidate, fit $W$, $H$ by multiplicative updates:

• $$H \leftarrow H\odot\frac{W^\top(V\odot(W H)^{\beta-1})}{W^\top(W H)^\beta}$$
• $$W \leftarrow W\odot\frac{(V\odot(W H)^{\beta-1})H^\top}{(W H)^\beta H^\top}$$

1. Likelihood Evaluation: With $μ_{ij} = (W H)_{ij}$, evaluate $\log L(\beta)$.
2. Selection/Refinement: Select β*, refine via local line search if needed.
3. α, γ, Rényi: Wrap β-selection as appropriate using derived reparametrizations.

Score matching on the (possibly non-normalized) EDA density provides a likelihood-free alternative, yielding similar selection accuracy.

5. Empirical Performance and Validation

Comprehensive experiments illustrate continuous and accurate recovery of true β or α across synthetic Tweedie (β ∈ [−2,1]) and Poisson (α → 1) data, outperforming or matching earlier methods (MTL, ED) and extending beyond their domain of applicability. In real-world NMF, e.g., music spectrograms and financial time series, divergence parameters chosen by this framework agree with theoretical or empirical cross-validation results but require neither reserved testing data nor manual heuristics.

For non-separable divergences, e.g., in multinomial, projective NMF, or SNE, automatic divergence selection yields principled, interpretable optima (γ* aligning with normalized KL, achieving perfect clustering, or clear community structures).

A summary of case studies:

Setting	Selected Divergence	Remark
Synthetic Tweedie (Poisson, β=0)	β* = 0	True model recovered
Projective NMF on block data	γ* ≈ –0.76	Perfect clustering
Symmetric SNE (dolphin network)	γ* ≈ –0.6	Two-community layout
Short piano spectrogram	β* = –1, α* ≈ 0.5	IS and Hellinger, resp.
Dow-Jones prices (50% missing)	β* ≈ 0.4	Matches prior cross-validation

MEDAL (with EDA) achieves accurate, likelihood-based divergence selection across the β ∈ ℝ continuum and extends automatically to non-separable and composite divergence families (Dikmen et al., 2014).

6. Theoretical and Practical Significance

The divergence matrix framework establishes:

• Unified likelihood-based selection: Divergence parameter learning becomes a formal ML optimization problem rather than ad hoc tuning.
• Extensibility: The machinery provides reductions to treat separable/α/γ/Rényi divergences systematically, including for model classes without closed-form densities.
• Generalization: Covers the entire real line of β, enabling robust modeling in non-Gaussian, non-Poisson, non-Gamma scenarios.
• Empirical discipline: Avoids overfitting via cross-validation entirely for divergence selection, leveraging only the data likelihood.

These properties afford practitioners the ability to adaptively and justifiably tune information divergences to individual learning tasks—yielding improved model fit, interpretability, and transferability.

7. Limitations and Alternative Criteria

• Standard Tweedie maximum-likelihood approaches are undefined for –1 < β < 0 and can be numerically unstable for β > 1. The EDA resolves these through augmentation.
• For certain tasks, score matching may be computationally more tractable than ML and gives equivalent qualitative behavior.
• While this framework provides continuous selection across divergence families, modeling assumptions (independence, exponential family) and the fidelity of the EDA approximation may set practical limits.

• S. Dikmen, A.-H. Phan, N. Gillis, L. Lim, S. Vembu, & A. Cichocki, "Learning the Information Divergence" (Dikmen et al., 2014)