Multimodal Attention-based Normalizing Flow (MANGO)

• MANGO is a multimodal fusion framework integrating invertible cross-attention flow layers to enable tractable and interpretable joint distribution modeling.
• It utilizes a stack of ICA layers and affine coupling blocks to ensure bijective mappings and precise latent variable traversal across modalities.
• State-of-the-art performance is achieved on tasks such as semantic segmentation, image-to-image translation, and movie genre classification through explicit likelihood estimation.

The Multimodal Attention-based Normalizing Flow (MANGO) framework is a multimodal fusion learning approach designed to provide explicit, interpretable, and tractable modeling of joint distributions across high-dimensional multimodal data. MANGO achieves this by replacing standard attention modules with invertible cross-attention (ICA) flow layers, enabling theoretical guarantees of bijection, likelihood tractability, and interpretable cross-modal representations. The framework demonstrates state-of-the-art performance on tasks such as semantic segmentation, image-to-image translation, and movie genre classification (Truong et al., 13 Aug 2025).

1. Model Architecture and Data Flow

MANGO processes a set of $N$ input tokens $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$ from multiple modalities. For modalities $A$ and $B$ with $M$ and $K$ tokens,

$$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$

The raw, high-dimensional multimodal data is first compressed using a perceptual encoder $\mathcal{E}$ (e.g., MAE for images, CLIP-text for captions) to obtain semantic latent features $\mathbf{F} = \mathcal{E}(\mathbf{X})$. This latent is then processed through a bijective normalizing flow backbone $G$, structured as a stack of $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$0 flow blocks, each comprising eight ICA layers (with varied cross-modal partitioning) and one affine coupling layer:

• ICA breakdown per block: 2 MMCA, 4 IMCA, 2 LICA The final latent $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$1 (with $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$2) forms the basis for the task-specific head $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$3, outputting predictions $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$4. Likelihood terms and invertibility ensure that the joint density and latent traversals are explicitly tractable.

The overall workflow:

• Raw multimodal inputs $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$5
• $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$6 Compressed features $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$7
• $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$8 Latent $\mathbf{X} = [\mathbf{x}_1,\dots,\mathbf{x}_N]$9
• $A$0 Predicted output $A$1

2. Mathematical Formulation and Flow Model

Let $A$2 be the compressed tokens. The overall bijection is built from $A$3 invertible blocks: $A$4 With $A$5 and $A$6. Via the change of variables,

$A$7

where $A$8, $A$9. For a data point $B$0,

$B$1

This explicit density enables exact likelihood estimation and interpretable latent variable traversals.

3. Invertible Cross-Attention (ICA) Layers

The ICA replaces standard coupling layers with a cross-attention operation that remains bijective and ensures Jacobian tractability:

Forward mapping:

• Partition $B$2 into $B$3
• Compute, for $B$4,

$B$5

with $B$6 as an upper-triangular mask (autoregressive). Outputs are $B$7, $B$8; merged as $B$9.

Inverse mapping:

• Since $M$0 is upper-triangular with positive diagonal, $M$1 is invertible: $M$2

Jacobian: $M$3 Computation of $M$4 is $M$5. This guarantees tractability for exact likelihood and gradient computation.

4. Cross-Modal Attention Partitioning Mechanisms

To efficiently capture diverse inter- and intra-modal dependencies, MANGO employs three partitioning styles within the ICA:

• Modality-to-Modality Cross-Attention (MMCA): $M$6, $M$7 (and vice versa)
• Inter-Modality Cross-Attention (IMCA): Each modality is split in half; e.g., $M$8, $M$9
• Learnable Inter-Modality Cross-Attention (LICA): A learnable permutation $K$0 is applied on $K$1, followed by bipartition and ICA; inverse permutation merges the outputs. The log-Jacobian of permutation is $K$2.

All three mechanisms use shared $K$3 projections; only LICA introduces trainable permutation parameters.

5. Training Objectives and Optimization

The objective comprises an explicit negative log-likelihood from the flow and a task-specific prediction loss: $K$4 where $K$5 is cross-entropy (categorical) or $K$6 (regression/image translation), and $K$7 is a balancing factor.

Optimization uses AdamW with weight decay $K$8, learning rate warm-up for 5 epochs, and cosine decay scheduling. Spectral normalization is applied on $K$9 and $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$0 to stabilize invertible transformations.

6. Scalability and Efficiency Considerations

The framework scales to high-dimensional data via a two-stage approach:

• Perceptual encoder compresses, e.g., $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$1 images into $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$2 tokens (reducing $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$3 to $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$4).
• Latent flow $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$5 is computed on $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$6 tokens of $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$7 (on modern GPU hardware).
• Each ICA layer has $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$8 complexity; $$\mathbf{X} = [\,\underbrace{\mathbf{x}^A_1,\dots,\mathbf{x}^A_M}_{A},\, \underbrace{\mathbf{x}^B_1,\dots,\mathbf{x}^B_K}_{B}]$$9 flow blocks suffice for modeling complex multimodal joint distributions.

7. Experimental Results and Ablation Analysis

State-of-the-art results are achieved across diverse multimodal tasks:

Semantic Segmentation (NYUDv2/SUN RGB-D):

Method	Inputs	NYUDv2 Pixel (%)	NYUDv2 mIoU (%)	SUN Pixel (%)	SUN mIoU (%)
TokenFusion (Small)	RGB+D	79.0	54.2	84.7	53.0
GeminiFusion (MiT-B5)	RGB+D	80.3	57.7	83.8	53.3
MANGO	RGB+D	81.5	59.2	83.9	54.1

Image-to-Image Translation (Taskonomy):

Task	GeminiFusion (FID)	MANGO (FID or MAE/MSE)
Shade+Texture$\mathcal{E}$0RGB	41.32	39.61
Depth+Normal$\mathcal{E}$1RGB	96.98	67.61
RGB+Shade$\mathcal{E}$2Normal	0.65 (MAE)	0.52
RGB+Edge$\mathcal{E}$3Depth	0.20 (MSE)	0.17

MM-IMDB Movie Genre Classification:

Method	Micro-F1 (%)	Macro-F1 (%)
BridgeTower	68.2	63.3
MANGO	71.7	68.2

Ablation:

Configuration	NYUDv2 mIoU	SUN mIoU
Replace ICA with coupling layer	50.8	48.5
+ Glow	53.0	49.1
+ Flow++	54.2	50.5
+ AttnFlow	56.5	52.2
Full MANGO (ICA+MMCA+IMCA+LICA)	59.2	54.1

Partitioning Analysis:

Partitioning	NYUDv2 mIoU	SUN mIoU
MMCA only	56.4	51.3
MMCA + IMCA	58.0	53.7
MMCA + IMCA + LICA	59.2	54.1

8. Interpretability and Visualization Capabilities

MANGO’s explicit invertibility and attention structure facilitate interpretability:

• The flow backbone $\mathcal{E}$4 enables tracking of individual token shifts in latent space, providing insight into cross-modal and intra-modal fusion at each ICA level.
• Triangular ICA attention matrices $\mathcal{E}$5 serve as interpretable attention maps, illustrating routing (e.g., depth-to-RGB) in early layers.
• Latent traversals in $\mathcal{E}$6 space lead to predictable, smooth transformations in output $\mathcal{E}$7, controlled by explicit likelihood constraints. A plausible implication is increased transparency of multimodal decision processes compared with black-box transformer-based approaches.

Summary Table: MANGO vs. Baselines

Aspect	Baseline Transformers	MANGO
Joint distribution	Implicit	Explicit normalizing flow
Flow construction	Non-invertible attention	Invertible cross-attention (ICA)
Tractability	No	Exact likelihood, tractable Jacobian
Cross-modal mechanisms	Implicit/self-attention	MMCA, IMCA, LICA partitioned flows
State-of-the-art (SoTA)	Variable	Achieved on all benchmarked tasks

MANGO advances multimodal fusion by delivering explicit, invertible, and interpretable cross-modal modeling, with empirical and theoretical improvements over prior approaches in both predictive performance and analysis capabilities (Truong et al., 13 Aug 2025).