Transformer-Based Multimodal Flow

• Transformer-based multimodal flow is a design space where cross-modal interactions are executed via attention-mediated, spatial, or probabilistic flows rather than simple concatenation.
• These models leverage structured token exchanges and spatial alignment to integrate heterogeneous modalities, enabling applications such as sentiment analysis, image registration, and unified text–image generation.
• Despite advances in efficiency and interpretability, challenges like quadratic attention costs, sensitivity to geometric distortions, and modality gaps remain to guide future research.

Transformer-based multimodal flow denotes a family of models in which cross-modal interaction is implemented as a structured flow of information, features, tokens, or probability mass through transformer operators rather than through only direct concatenation or late ensembling. In the cited literature, the term “flow” is used for at least three related but non-identical ideas: attention-mediated information transfer between modalities, dense spatial flow or occupancy flow fields estimated from multimodal observations, and normalizing-flow or flow-matching formulations for unified multimodal generation (Wang et al., 2020, Liu et al., 7 Apr 2026, Shen et al., 8 May 2026, Truong et al., 13 Aug 2025). A plausible implication is that the topic is not a single architecture class, but a design space organized around how transformers route cross-modal dependency.

1. Meanings of “flow” in transformer multimodality

A central ambiguity of the term is conceptual. In multimodal sentiment analysis and text–vision fusion, flow usually denotes movement of information across modalities inside attention or encoder–decoder translation. TransModality explicitly defines “multimodal flow” as the movement and transformation of information across modalities via attention, specifically decoder cross-attention that uses encoder outputs from modality $s$ as keys and values and decoder states tied to modality $t$ as queries (Wang et al., 2020). MuSE uses the same broad intuition, but instantiates it as token-level exchange inside a shared-parameter CrossTransformer, where low-importance tokens in one modality are replaced by a residual average from the other modality (Zhu et al., 2023).

In other work, flow is literal spatial correspondence. CRFT estimates a dense feature flow field $u(x)$ for cross-modal image registration, and STrajNet predicts occupancy and backward flow fields in bird’s-eye-view autonomous-driving grids (Liu et al., 7 Apr 2026, Liu et al., 2022). In these systems, multimodality refers to heterogeneous sensing inputs, while flow denotes geometric displacement.

A third usage is probabilistic. STARFlow2 argues that autoregressive normalizing flows are autoregressive Transformers, sharing the same causal mask, KV-cache mechanism, and left-to-right structure as LLMs, and therefore provide a natural basis for unified interleaved text–image generation (Shen et al., 8 May 2026). MANGO likewise uses invertible cross-attention inside a normalizing flow so that multimodal fusion is explicit, interpretable, and tractable, with exact log-likelihoods via change-of-variables (Truong et al., 13 Aug 2025).

This variety dispels a common misconception: transformer-based multimodal flow is not synonymous with optical flow, nor with any single fusion heuristic. It includes translation, exchange, routing, warping, and density transport, depending on the task formulation.

2. Attention-mediated information transfer

The canonical transformer mechanism remains scaled dot-product attention,

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$

but the literature differs sharply in how $Q$, $K$, and $V$ are assigned across modalities (Wang et al., 2020).

TransModality treats fusion as cross-modal translation. Each video is segmented into $N$ utterances, with utterance-level textual, visual, and acoustic features. A BiGRU first contextualizes each modality across the utterance sequence, and dense projections produce $D_i^\lambda$. A Modality Fusion Cell then performs forward $\alpha \rightarrow \beta$ and backward $t$0 translation with separate encoder–decoder Transformers. With text as the main modality, two parallel MFCs are used: $t$1 for text–visual and $t$2 for text–acoustic. The final representation is

$t$3

followed by softmax classification. The encoded states are therefore not raw unimodal embeddings, but translation-informed joint features (Wang et al., 2020).

MuSE adopts a different strategy. Two separate encoders first map text and image inputs to low-dimensional sequences, and two decoders regularize them into a shared space through image captioning and text-to-image generation. CrossTransformer then applies two Transformer encoders with shared parameters. Early layers learn intra-modal contextual information; exchange layers select a proportion $t$4 of tokens with the smallest attention from $t$5, and replace them with a residual sum involving the average embedding of the other modality. In the reported configuration, the default depth is 6 layers, with $t$6, $t$7, and $t$8 (Zhu et al., 2023).

Factorization and masking give two further answers to the same problem. FMT decomposes multimodal self-attention into seven factors,

$t$9

so that intra-modal and inter-modal dynamics are modeled in parallel with full time-domain receptive fields. Two summarization networks, $u(x)$0 and $u(x)$1, compress the many attention outputs back to fixed dimensionality, allowing more attentions without uncontrolled parameter growth (Zadeh et al., 2019). Zorro instead keeps some representations deliberately modality-pure: audio queries attend only to audio, video queries only to video, and fusion queries to audio, video, and fusion tokens. This block-sparse routing is the key mechanism by which the model avoids contrastive collapse and preserves unimodal inference capability (Recasens et al., 2023).

Taken together, these models show that transformer-based multimodal flow is often less about adding more attention, and more about specifying which information is allowed to move, when, and through which structural bottleneck.

3. Spatial and temporal correspondence as flow fields

In cross-modal registration, transformer-based multimodal flow is instantiated as dense geometric alignment. CRFT uses a shared ResNet-based encoder to extract modality-independent features at resolutions $u(x)$2, then applies a coarse-to-fine transformer. At $u(x)$3 resolution, global correspondences are estimated by projected, normalized features and a correlation volume,

$u(x)$4

yielding coarse flow $u(x)$5. Fine-scale refinement uses window-based self-attention and cross-attention, a Fine-Scale Feature Transformation MLP, discrepancy-guided attention, residual updates, confidence estimation, and a differentiable Spatial Geometric Transform for recurrent warping (Liu et al., 7 Apr 2026).

STrajNet addresses a different motion problem: future occupancy and backward flow in autonomous driving. Historical occupancy, dense road maps, historical backward flow, and vectorized agent trajectories are encoded separately. A multi-stage Swin-based visual encoder produces hierarchical features, and a vector encoder models interactions across agents. Its distinctive operation is Flow-Guided Multi-Head Self-Attention, in which learned flow offsets warp keys and values before attention so that occupancy aggregation becomes motion-aware. Late fusion then uses temporal pixel-wise cross-attention from flow-guided occupancy features to vector features (Liu et al., 2022).

These architectures use transformers not just to correlate modalities, but to reconcile heterogeneous sensors with explicit spatial dynamics. CRFT states that it learns a modality-independent feature flow representation and jointly performs feature alignment and flow estimation, while STrajNet jointly predicts occupancy and flow and models the mathematical relations between them (Liu et al., 7 Apr 2026, Liu et al., 2022). A plausible implication is that transformer-based multimodal flow becomes especially useful when modality gaps and geometric variability invalidate photometric constancy or single-sensor priors.

4. Flow-based generative and probabilistic unification

The generative branch of the literature reframes multimodal flow as exact or approximate transport in latent space. STARFlow2 is explicit: it uses autoregressive normalizing flows parameterized by causal Transformers so that both text tokens and continuous visual latents are generated under one causal mask and one KV-cache. Its Pretzel architecture vertically interleaves a frozen pretrained VLM stream with a TarFlow stream through zero-initialized residual skip connections. For a visual token $u(x)$6, the per-token affine flow is

$u(x)$7

with exact log-determinant accumulation through autoregressive triangular Jacobians (Shen et al., 8 May 2026).

NextFlow takes a discrete-token route but retains the same principle of a unified decoder-only sequence model. Text uses standard next-token prediction,

$u(x)$8

whereas images use next-scale prediction,

$u(x)$9

so that coarse visual grids are generated first and finer scales are sampled later, with all positions at a given scale sampled in parallel. This departs from raster-scan autoregression and supports 1024$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$01024 generation in just 5 seconds (Zhang et al., 5 Jan 2026).

LaTtE-Flow retains flow matching but improves inference by partitioning transformer layers into timestep-specialized experts. With $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$1 layers and $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$2 groups, only $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$3 layers are activated at each sampling step. The layerwise loss is defined on the relevant timestep interval,

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$4

and Timestep-Conditioned Residual Attention reuses attention maps across adjacent layers with a timestep gate $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$5 (Shen et al., 8 Jun 2025).

MANGO brings invertibility to multimodal fusion itself. Its Invertible Cross-Attention layer partitions tokens into $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$6, forms

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$7

with a triangular mask $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$8, and sets $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$9, $Q$0. Because the Jacobian is block-triangular, $Q$1, and the inverse is available through $Q$2. MMCA, IMCA, and LICA then provide different partitioning and permutation schemes for explicit cross-modal interaction (Truong et al., 13 Aug 2025).

These models share a strong claim: multimodal generation need not be bolted onto a transformer by an external diffusion module alone. It can instead be encoded as a flow process whose autoregressive, invertible, or flow-matching structure is itself transformer-native.

5. Routing, efficiency, and controllability

Transformer-based multimodal flow is often presented as a remedy for brute-force entanglement. Zorro is the clearest case: unrestricted early fusion is reported to collapse under contrastive audio–visual self-supervision, whereas masked routing preserves audio-pure, video-pure, and fusion streams inside one backbone (Recasens et al., 2023). This directly challenges the idea that maximal cross-modal connectivity is always beneficial.

Efficiency-oriented work reaches a similar conclusion through token economics rather than masking. The wireless-network framework of “Transformer Architecture with Minimal Inference Latency for Multi-Modal Wireless Networks” introduces modality-specific tokenizers, a token router that predicts per-layer importance scores, and trainable keep ratios $Q$3 under a target FLOPs budget. The relaxed objective is

$Q$4

On DeepSense 6G beamforming, the reported reductions reach 86.2% latency, 35% GPU memory, and 80% FLOPs with negligible accuracy loss; on the real handover dataset, latency is reduced by 83.2% with minimal accuracy degradation (Kim et al., 21 Apr 2026).

CRFT and LaTtE-Flow provide two other efficiency templates. CRFT pushes global attention only to $Q$5 resolution and uses windowed attention at finer scales, while LaTtE-Flow reduces per-step active layers from 28 to 7 and reports activated parameters per timestep of about 500M rather than about 2.0B in its vanilla baselines (Liu et al., 7 Apr 2026, Shen et al., 8 Jun 2025). In both cases, multimodal flow is preserved by restricting where expensive interaction occurs.

Interpretability is likewise tied to controllability. TransFlower, although focused on commuting flow prediction, shows that transformer attention over origin–destination flow tokens can be made interpretable through anisotropy-aware relative location encodings and destination-by-destination attention matrices that highlight influential regions such as CBDs and train stations (Luo et al., 2024). MANGO makes the same point probabilistically: if the multimodal interaction itself is invertible and its Jacobian tractable, the correlation structure is no longer only implicit (Truong et al., 13 Aug 2025).

6. Empirical picture, limitations, and open problems

Reported results are heterogeneous because tasks, modalities, and metrics differ sharply. The table summarizes representative values exactly as reported.

Model	Task	Reported result
TransModality	CMU-MOSI	82.71 weighted accuracy
CRFT	OSdataset	AEPE 0.65; CMR@1px 95.1%
Zorro-Swin	AudioSet-2M	46.5 mAP
STARFlow2	GenEval / MMMU (val)	0.82 / 44.7
NextFlow	Text-to-image	1024×1024 images in ~5 seconds
LaTtE-Flow Couple	ImageNet-50K	FID 5.79; 0.052 s/img
MANGO	MM-IMDB	Micro-F1 71.7; Macro-F1 68.2

The empirical picture supports several recurring patterns. Bidirectional translation helps in sentiment analysis: TransModality reports CMU-MOSI tri-modal performance of 82.71 with backward translation versus 80.18 without, and similar gains on MELD and IEMOCAP (Wang et al., 2020). Structured token exchange helps in text–vision tasks: MuSE reports Twitter15 F1 76.81, Twitter17 F1 88.62, MVSA-Single 75.80 / 75.58, and MVSA-Multiple 74.10 / 73.93, with degradation when exchange or auxiliary decoders are removed (Zhu et al., 2023). In cross-modal registration, CRFT reports AEPE 0.65 on OSdataset and 2.37 on RoadScene, and an ablation on OSdataset improves from AEPE 3.76 for baseline XoFTR to 0.40 for the full CRFT configuration (Liu et al., 7 Apr 2026). In unified generation, STARFlow2 reports MME-p 1528.8, GenEval 0.82, and DPG-Bench 84.94, while NextFlow reports GenEval 0.83, DPG 86.00, WISE 0.59, and image editing gains after RL prefix-tuning (Shen et al., 8 May 2026, Zhang et al., 5 Jan 2026).

Limitations are equally recurrent. Attention remains quadratic in sequence length unless additional routing or hierarchy is introduced. TransModality depends on reliable pre-extracted utterance-level features and reports no optimizer or hyperparameter details beyond the use of BiGRU contextualization and Transformer-based encoder–decoder modules (Wang et al., 2020). CRFT reports performance drops under extreme geometric distortions such as 90° rotations with large scale changes, and potential failure in textureless regions, severe noise, or extreme modality gaps (Liu et al., 7 Apr 2026). STARFlow2 notes that 256×256 resolution and fine-grained fidelity are constrained by the FAE encoder (Shen et al., 8 May 2026). NextFlow explicitly identifies the discrete bottleneck introduced by vector quantization, and LaTtE-Flow notes that timestep partitioning is currently fixed rather than learned (Zhang et al., 5 Jan 2026, Shen et al., 8 Jun 2025). MANGO, by design, trades additional structural constraints for exact invertibility and tractable Jacobians (Truong et al., 13 Aug 2025).

Two misconceptions recur in the literature. First, unrestricted fusion is not necessarily a virtue: Zorro shows that full entanglement can be actively harmful for contrastive learning and unimodal inference (Recasens et al., 2023). Second, flow is not a unitary construct: in one paper it may denote feature warping, in another exact density transport, and in another controlled information routing. This suggests that future work will likely continue not toward one canonical transformer-based multimodal flow model, but toward sharper correspondences between task structure and the specific kind of flow a transformer is asked to realize.