---
title: Multimodal Attention-based Normalizing Flow (MANGO)
url: https://www.emergentmind.com/topics/multimodal-attention-based-normalizing-flow-mango
type: topic
---

# Multimodal Attention-based Normalizing Flow (MANGO)

The Multimodal Attention-based Normalizing Flow (MANGO) framework is a multimodal fusion learning approach designed to provide explicit, interpretable, and tractable modeling of joint distributions across high-dimensional multimodal data. MANGO achieves this by replacing standard attention modules with invertible cross-attention (ICA) flow layers, enabling theoretical guarantees of bijection, likelihood tractability, and interpretable cross-modal representations. The framework demonstrates state-of-the-art performance on tasks such as semantic segmentation, image-to-image translation, and movie genre classification [2508.10133].

## 1. Model Architecture and Data Flow

MANGO processes a set of $N$ input tokens $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$ from multiple modalities. For modalities $A$ and $B$ with $M$ and $K$ tokens,
\[
\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]
\]
The raw, high-dimensional multimodal data is first compressed using a perceptual encoder $\mathcal{E}$ (e.g., MAE for images, CLIP-text for captions) to obtain semantic latent features $\mathbf{F} = \mathcal{E}(\mathbf{X})$. This latent is then processed through a bijective normalizing flow backbone $G$, structured as a stack of $L$ flow blocks, each comprising eight ICA layers (with varied cross-modal partitioning) and one affine coupling layer:
- ICA breakdown per block: 2 MMCA, 4 IMCA, 2 LICA
The final latent $\mathbf{Z}=G(\mathbf{F})$ (with $\mathbf{Z}\sim\mathcal{N}(0,I)$) forms the basis for the task-specific head $\mathrm{TaskHead}$, outputting predictions $\hat Y$. Likelihood terms and invertibility ensure that the joint density and latent traversals are explicitly tractable.

The overall workflow:
- Raw multimodal inputs $\mathbf{X}$
- $\xrightarrow[]{\mathcal{E}}$ Compressed features $\mathbf{F}$
- $\xrightarrow[]{G = [\mathrm{ICA}_1 \to ... \to \mathrm{ICA}_8 \to \text{Coupling}] \times L}$ Latent $\mathbf{Z}$
- $\xrightarrow[]{\mathrm{TaskHead}}$ Predicted output $\hat Y$

## 2. Mathematical Formulation and Flow Model

Let $\mathbf{F}\in\mathbb{R}^{N\times d}$ be the compressed tokens. The overall bijection is built from $L$ invertible blocks:
\[
G = f_L \circ f_{L-1} \circ \cdots \circ f_1
\]
With $\mathbf{Z} = G(\mathbf{F})$ and $\pi(\mathbf{Z}) = \mathcal{N}(\mathbf{0},\mathbf{I})$. Via the change of variables,
\[
p_{\mathbf{F}}(\mathbf{F}) = \pi(G(\mathbf{F}))\, 
  \prod_{i=1}^L \left|\det\left(\frac{\partial f_i}{\partial h_{i-1}}\right)\right|
\]
where $h_0=\mathbf{F}$, $h_L=\mathbf{Z}$. For a data point $\mathbf{f}$,
\[
\log p_{\mathbf{F}}(\mathbf{f}) = \log\pi(G(\mathbf{f})) + \sum_{i=1}^L \log\left|\det\left(\frac{\partial f_i}{\partial h_{i-1}}\right)\right|
\]
This explicit density enables exact likelihood estimation and interpretable latent variable traversals.

## 3. Invertible Cross-Attention (ICA) Layers

The ICA replaces standard coupling layers with a cross-attention operation that remains bijective and ensures Jacobian tractability:

**Forward mapping**:
- Partition $H\in\mathbb{R}^{N\times d}$ into $(H_1, H_2)$
- Compute, for $Q = \mathrm{LN}(W_Q H_1),\ K = \mathrm{LN}(W_K H_1),\ V = H_2$,
\[
A = \mathrm{softmax}\left(\frac{Q K^\top}{\sqrt{d}} \odot M\right)
\]
with $M$ as an upper-triangular mask (autoregressive). Outputs are $Y_1=H_1$, $Y_2=AV$; merged as $Y$.

**Inverse mapping**:
- Since $A$ is upper-triangular with positive diagonal, $A$ is invertible:
\[
H_1 = Y_1,\quad H_2 = A^{-1}Y_2
\]

**Jacobian**:
\[
\det\left(\frac{\partial Y}{\partial H}\right) = \det(A)^{n_2} = \prod_{i=1}^{n_2} A_{ii}
\]
Computation of $\log\det(A)$ is $O(N)$. This guarantees tractability for exact likelihood and gradient computation.

## 4. Cross-Modal Attention Partitioning Mechanisms

To efficiently capture diverse inter- and intra-modal dependencies, MANGO employs three partitioning styles within the ICA:

- **Modality-to-Modality Cross-Attention (MMCA):** $H_1 = \mathbf{x}^A_{1:M}$, $H_2 = \mathbf{x}^B_{1:K}$ (and vice versa)
- **Inter-Modality Cross-Attention (IMCA):** Each modality is split in half; e.g., $H_1=[\mathbf{X}_A^1, \mathbf{X}_B^1]$, $H_2=[\mathbf{X}_A^2, \mathbf{X}_B^2]$
- **Learnable Inter-Modality Cross-Attention (LICA):** A learnable permutation $W_{per} = P L (U+\mathrm{diag}(s))$ is applied on $\mathbf{X}$, followed by bipartition and ICA; inverse permutation merges the outputs. The log-Jacobian of permutation is $\sum_i\log|s_i|$.

All three mechanisms use shared $W_Q, W_K$ projections; only LICA introduces trainable permutation parameters.

## 5. Training Objectives and Optimization

The objective comprises an explicit negative log-likelihood from the flow and a task-specific prediction loss:
\[
\mathcal{L}(\theta) = -\log\pi(G(\mathcal{E}(\mathbf{X}))) - \sum_{i=1}^L \log\left|\det\frac{\partial f_i}{\partial h_{i-1}}\right| + \lambda\,\mathcal{L}_{task}(\hat Y, Y)
\]
where $\mathcal{L}_{task}$ is cross-entropy (categorical) or $\ell_1/\ell_2$ (regression/image translation), and $\lambda$ is a balancing factor.

Optimization uses AdamW with weight decay $1\mathrm{e}{-4}$, learning rate warm-up for 5 epochs, and cosine decay scheduling. Spectral normalization is applied on $W_Q$ and $W_K$ to stabilize invertible transformations.

## 6. Scalability and Efficiency Considerations

The framework scales to high-dimensional data via a two-stage approach:
- **Perceptual encoder** compresses, e.g., $256\times 256$ images into $16\times 16 \times d$ tokens (reducing $65{,}536d$ to $256d$).
- **Latent flow $G$** is computed on $N=256$ tokens of $d=512$ (on modern GPU hardware).
- Each ICA layer has $O(N^2d)$ complexity; $L=12$ flow blocks suffice for modeling complex multimodal joint distributions.

## 7. Experimental Results and Ablation Analysis

State-of-the-art results are achieved across diverse multimodal tasks:

**Semantic Segmentation (NYUDv2/SUN RGB-D):**

| Method                 | Inputs | NYUDv2 Pixel (%) | NYUDv2 mIoU (%) | SUN Pixel (%) | SUN mIoU (%) |
|------------------------|--------|-----------------|----------------|--------------|--------------|
| TokenFusion (Small)    | RGB+D  | 79.0            | 54.2           | 84.7         | 53.0         |
| GeminiFusion (MiT-B5)  | RGB+D  | 80.3            | 57.7           | 83.8         | 53.3         |
| MANGO                  | RGB+D  | 81.5            | 59.2           | 83.9         | 54.1         |

**Image-to-Image Translation (Taskonomy):**

| Task                    | GeminiFusion (FID) | MANGO (FID or MAE/MSE) |
|-------------------------|-------------------|------------------------|
| Shade+Texture$\to$RGB   | 41.32             | 39.61                  |
| Depth+Normal$\to$RGB    | 96.98             | 67.61                  |
| RGB+Shade$\to$Normal    | 0.65 (MAE)        | 0.52                   |
| RGB+Edge$\to$Depth      | 0.20 (MSE)        | 0.17                   |

**MM-IMDB Movie Genre Classification:**

| Method         | Micro-F1 (%) | Macro-F1 (%) |
|----------------|--------------|--------------|
| BridgeTower    | 68.2         | 63.3         |
| MANGO          | 71.7         | 68.2         |

**Ablation:**

| Configuration                     | NYUDv2 mIoU | SUN mIoU |
|------------------------------------|-------------|----------|
| Replace ICA with coupling layer    | 50.8        | 48.5     |
| + Glow                            | 53.0        | 49.1     |
| + Flow++                          | 54.2        | 50.5     |
| + AttnFlow                        | 56.5        | 52.2     |
| Full MANGO (ICA+MMCA+IMCA+LICA)    | 59.2        | 54.1     |

**Partitioning Analysis:**

| Partitioning         | NYUDv2 mIoU | SUN mIoU |
|----------------------|-------------|----------|
| MMCA only            | 56.4        | 51.3     |
| MMCA + IMCA          | 58.0        | 53.7     |
| MMCA + IMCA + LICA   | 59.2        | 54.1     |

## 8. Interpretability and Visualization Capabilities

MANGO’s explicit invertibility and attention structure facilitate interpretability:
- The flow backbone $G$ enables tracking of individual token shifts in latent space, providing insight into cross-modal and intra-modal fusion at each ICA level.
- Triangular ICA attention matrices $A$ serve as interpretable attention maps, illustrating routing (e.g., depth-to-RGB) in early layers.
- Latent traversals in $\mathbf{Z}$ space lead to predictable, smooth transformations in output $\hat Y$, controlled by explicit likelihood constraints.
A plausible implication is increased transparency of multimodal decision processes compared with black-box transformer-based approaches.

## Summary Table: MANGO vs. Baselines

| Aspect                    | Baseline Transformers        | MANGO                                  |
|---------------------------|-----------------------------|----------------------------------------|
| Joint distribution        | Implicit                    | Explicit normalizing flow              |
| Flow construction         | Non-invertible attention    | Invertible cross-attention (ICA)       |
| Tractability              | No                          | Exact likelihood, tractable Jacobian   |
| Cross-modal mechanisms    | Implicit/self-attention     | MMCA, IMCA, LICA partitioned flows     |
| State-of-the-art (SoTA)   | Variable                    | Achieved on all benchmarked tasks      |

MANGO advances multimodal fusion by delivering explicit, invertible, and interpretable cross-modal modeling, with empirical and theoretical improvements over prior approaches in both predictive performance and analysis capabilities [2508.10133].

Source: https://www.emergentmind.com/topics/multimodal-attention-based-normalizing-flow-mango