Multi-modal Transformer Encoder Overview

• Multi-modal Transformer Encoder is a design space that integrates heterogeneous modalities through unified tokenization, varied attention strategies, and flexible parameter sharing.
• Architectural approaches include shared encoders, parallel modality-specific encoders with synchronization, and non-standard encoder variants that leverage cross-modal token exchange.
• Empirical results across vision-language, autonomous driving, and remote sensing demonstrate that MTEs can significantly improve task performance with tailored fusion strategies.

Searching arXiv for recent and foundational uses of “Multi-modal Transformer Encoder” and related multimodal transformer architectures. Multi-modal Transformer Encoder (MTE) denotes an encoder-centered mechanism for jointly processing heterogeneous modalities, but the term is used with marked heterogeneity across the arXiv literature. In some works it refers to a standard shared Transformer encoder over unified token sequences; in others it denotes parallel modality-specific encoders with explicit synchronization, hierarchical cross-modal attention, token exchange, or progressive token selection; and in one case it denotes a parameter-free encoder built from frozen random Gaussian compression matrices rather than learned self-attention. This suggests that MTE is best understood as a family of multimodal encoding strategies rather than a single canonical architecture (Zhu et al., 2023, Kim et al., 2024, Zhang et al., 2023, Liang et al., 2022).

1. Terminological scope and representative usages

The phrase “Multi-modal Transformer Encoder” appears in vision-language fusion, autonomous driving, remote sensing, recommendation, video retrieval, image captioning, human-state recognition, multimodal entity alignment, event classification, finance, and general multimodal representation learning. Earlier multimodal Transformer systems for image captioning and caption-to-video retrieval already used encoder-decoder structures in which self-attention modeled intra-modal relations and cross-attention or joint attention modeled inter-modal relations, even when the term MTE was not yet standardized (Yu et al., 2019, Gabeur et al., 2020).

Usage pattern	Defining characteristic	Representative papers
Exchanging-based encoder	Selected tokens are mixed with the average of the other modality’s embeddings	MuSE (Zhu et al., 2023)
Multi-modal attention encoder	Each predicted trajectory is conditioned on an independent attention mode	Motion prediction (Huang et al., 2021)
Shared standard encoder	Different modalities are processed as different token streams by one shared encoder	TMT (Kim et al., 2024), Meta-Transformer (Zhang et al., 2023)
Synchronized parallel encoders	Modality-specific encoders exchange only special class tokens after each block	SCT Fusion (Hoffmann et al., 2023)
Progressive tri-modal encoder	Cross-modal attention compresses long visual and acoustic token streams to length $K$	ME2ET (Wu et al., 2022)
Non-standard “transformer” encoder	Frozen random projections over column stripes or time segments	PC-VAE (Liang et al., 2022)

A central consequence of this breadth is that “encoder” may denote very different loci of fusion. In MuSE, the MTE is the CrossTransformer backbone; in TMT and Meta-Transformer it is the shared modality-agnostic backbone; in SCT Fusion it is the synchronized collection of per-modality ViT blocks; and in MoAlign it is a hierarchical modifiable self-attention stack over graph-structural, textual, and visual tokens (Zhu et al., 2023, Kim et al., 2024, Hoffmann et al., 2023, Li et al., 2023).

2. Recurrent architectural organizations

One recurrent organization is the shared-parameter parallel stack. MuSE uses two parallel Transformer-encoder stacks, each with $L=6$ layers, hidden size $d$, intermediate size $d_{ff}=4d$, and $h$ heads, while all self-attention weights, FFN weights, and layer-norm parameters are shared between the text and image stacks. Layers $1\ldots\mu$ with $\mu=2$ by default are plain Transformer layers for modality-internal contextualization; inter-modal exchange is then applied on layers $\mu+1\ldots\eta$ with $\eta=4$ by default (Zhu et al., 2023).

A second organization is the single shared encoder over tokenized modalities. TMT tokenizes speech with a HuBERT encoder and a k-means codebook, tokenizes images with a VQ-VAE style quantizer, tokenizes text with a standard subword tokenizer, forms a single vocabulary $V=V_i\cup V_s\cup V_t$, and feeds the resulting tokens into a 12-layer encoder with $L=6$0, $L=6$1, and $L=6$2. There are no separate modality-specific layers inside the encoder; modality identity is injected through a modality-type embedding $L=6$3 added at the input (Kim et al., 2024). Meta-Transformer adopts a related but more general pattern: a unified data tokenizer maps each modality into $L=6$4-dimensional tokens, after which a frozen ViT-B/16 encoder with $L=6$5, $L=6$6, MLP hidden size $L=6$7, and $L=6$8 processes all modalities, while only the modality tokenizers and downstream heads are optimized per task (Zhang et al., 2023).

A third organization is the parallel modality-specific encoder with block-wise synchronization. SCT Fusion uses one ViT-style encoder per modality, with patch embedding, a learnable class token, positional embeddings, and $L=6$9 encoder blocks at embedding dimension $d$0 and $d$1 heads. After every block, the updated class tokens are concatenated, fused by a trainable linear transformation, and the synchronized token is broadcast back into every modality stream (Hoffmann et al., 2023). Husformer follows a different branch-specific design: modality-specific Conv1D projections first produce low-level unimodal features, cross-modal Transformer layers reinforce each modality from the concatenated fusion sequence, and a self-attention Transformer then fuses the reinforced features into a high-level representation (Wang et al., 2022).

A fourth organization is the task-agnostic shared backbone with structured inter-modal blocks. UniTR replaces modality-specific CNN or sparse-convolution backbones with a single Transformer stack whose weights are shared across images and LiDAR. Four DSVT blocks are used: Block 1 is intra-modal, Blocks 2–3 are inter-modal in 2D image space, and Block 4 is inter-modal in 3D space (Wang et al., 2023). A closely related idea appears in the multi-scale cross-attention encoder for HL-LHC event classification, where jet-substructure streams and a kinematics stream are first encoded separately and then fused by two consecutive cross-attention encoder blocks (Hammad et al., 2023).

3. Mechanisms of cross-modal interaction

The most distinctive interaction rule in the surveyed literature is MuSE’s inter-modal token exchange. At each exchanging layer, the model computes the multi-head self-attention of cls to every token, selects the bottom $d$2 proportion of tokens with the smallest cls-to-token scores, and for each selected token $d$3 in modality $d$4 replaces its embedding by mixing in the average of the other modality’s token embeddings:

$d$5

with default $d$6. No extra projection or normalization beyond the built-in Transformer layer-norm is applied (Zhu et al., 2023).

A different interaction mechanism is mode-separating attention. In transformer-based motion prediction for autonomous driving, the second layer of the agent–map encoder replaces standard MultiHead attention by a “MultiModal” attention that outputs each head separately rather than concatenating them:

$d$7

with $d$8. Each $d$9 is then combined with the target-agent history and interaction feature to form a mode-specific driving context for a trajectory decoder and a score decoder (Huang et al., 2021).

A third interaction rule is class-token synchronization. SCT Fusion collects the per-modality class tokens after each block, concatenates them,

$d_{ff}=4d$0

and produces a synchronized class token

$d_{ff}=4d$1

which is copied back into the $d_{ff}=4d$2-th position of each modality sequence for the next block (Hoffmann et al., 2023).

ME2ET implements progressive tri-modal attention rather than full-length tri-modal self-attention. Text first guides selection of $d_{ff}=4d$3 visual tokens, then text plus visual guide selection of $d_{ff}=4d$4 acoustic tokens, and finally text plus acoustic refine the visual selection. This reduces self-attention complexity from $d_{ff}=4d$5 to $d_{ff}=4d$6, with $d_{ff}=4d$7 in the reported implementation (Wu et al., 2022). Husformer uses cross-modal transformers in which each modality attends to the concatenated low-level fusion sequence and is then followed by a self-attention transformer over the concatenated reinforced features (Wang et al., 2022).

MoAlign provides a graph-centered variant of multimodal interaction. Its hierarchical modifiable self-attention block performs three successive attentions within each layer—neighbors, then text attributes, then image attributes—under a learned mask that suppresses incompatible token interactions, while entity-type prefix vectors are prepended to keys and values in every attention and injected analogously into the feed-forward layers (Li et al., 2023).

4. Tokenization, alignment, and positional structure

Token construction is a primary design axis. TMT explicitly interprets different modalities as different languages. Speech is converted into discrete units by a pre-trained HuBERT encoder followed by nearest-centroid assignment in a learned k-means codebook; image data are quantized into vision tokens by a VQ-VAE style quantizer; and text is tokenized by a standard subword tokenizer. The encoder input is

$d_{ff}=4d$8

where $d_{ff}=4d$9 is a shared token embedding matrix, $h$0 is positional embedding, and $h$1 is a modality-type embedding for image, speech, or text (Kim et al., 2024).

Meta-Transformer also begins with modality-specialist tokenizers, but its emphasis is on mapping 12 modalities into a shared token space before passing them through a frozen modality-shared encoder. The tokenizer differs by modality—WordPiece for text, ViT-style patch flattening for images, FPS+KNN plus MLP for point clouds, convolutional patching for audio spectrograms, and analogous grouping-and-projection pipelines for video, hyperspectral, tabular, graph, and time-series data—yet all outputs are $h$2-dimensional tokens (Zhang et al., 2023).

Temporal and structural embeddings often mediate multimodal alignment. In video retrieval, the multimodal sequence is

$h$3

where $h$4 contains per-expert aggregated and temporal tokens, $h$5 is an expert-type embedding, and $h$6 is a learned temporal embedding including special tokens $h$7 and $h$8 (Gabeur et al., 2020). UniTR instead uses spatial coordinates passed through a small MLP as positional encoding and alternates 2D projection-based and 3D unprojection-based inter-modal integration so that image and LiDAR tokens join in common structural spaces (Wang et al., 2023). MMF-Trans introduces a three-stage time encoding,

$h$9

to align macro data, technical indicators, financial text, and event knowledge across mixed frequencies (AI et al., 28 Jan 2025).

Alignment can also be enforced by auxiliary generation or by nonstandard token geometry. MuSE uses two decoders for embedding regularization: a TextDecoder with image-to-text cross-entropy loss $1\ldots\mu$0 and an ImageDecoder with PixelCNN++ text-to-image loss $1\ldots\mu$1, and these losses pull $1\ldots\mu$2 and $1\ldots\mu$3 into a shared latent space (Zhu et al., 2023). PC-VAE abandons patch tokens in favor of column stripes for images and contiguous time segments for audio; each stripe or segment is compressed by a frozen random Gaussian matrix, and the per-modality code is the sum of the compressed parts (Liang et al., 2022).

5. Objectives, tasks, and empirical behavior

Training objectives range from pure task supervision to explicitly multimodal auxiliary losses. MuSE optimizes

$1\ldots\mu$4

with $1\ldots\mu$5 set to $1\ldots\mu$6 in the default configuration. On multimodal named entity recognition, it reports $1\ldots\mu$7 F1 on Twitter15 versus a best prior $1\ldots\mu$8, $1\ldots\mu$9 on Twitter17 versus $\mu=2$0, and $\mu=2$1 on MT-Product versus a best prior $\mu=2$2. On multimodal sentiment analysis, it reports $\mu=2$3 Acc/F1 on MVSA-Single versus $\mu=2$4, and $\mu=2$5 on MVSA-Multiple versus $\mu=2$6. Its ablations show F1 drops of $\mu=2$7 without CrossTransformer, $\mu=2$8 without the decoding tasks, $\mu=2$9 without image captioning loss, and $\mu+1\ldots\eta$0 without text-to-image generation loss (Zhu et al., 2023).

In autonomous driving, the motion-prediction MTE is trained end-to-end with

$\mu+1\ldots\eta$1

where $\mu+1\ldots\eta$2 is a variety-style smooth-L1 loss applied only to the best mode and $\mu+1\ldots\eta$3 is a cross-entropy loss against a soft ground-truth distribution over modes. On the Argoverse test set with $\mu+1\ldots\eta$4, the model reports $\mu+1\ldots\eta$5, $\mu+1\ldots\eta$6, $\mu+1\ldots\eta$7, $\mu+1\ldots\eta$8, and a total parameter count of $\mu+1\ldots\eta$9 M (Huang et al., 2021).

Task-agnostic or unified backbones show a different empirical profile. A single 270M-parameter TMT model handling all six translation directions reportedly outperforms or matches six separately trained uni-modal models; on COCO and Flickr8k it yields $\eta=4$0–$\eta=4$1 BLEU in image captioning, $\eta=4$2–$\eta=4$3 CLIP score in text-to-image, and substantial gains in speech-to-image, while ASR beats a standalone Wav2Vec2.0 and TTS quality rivals VITS (Kim et al., 2024). UniTR reports val NDS $\eta=4$4 and mAP $\eta=4$5, test NDS $\eta=4$6 and mAP $\eta=4$7 for 3D detection on nuScenes, as well as BEV segmentation mIoU $\eta=4$8 for camera-plus-LiDAR and $\eta=4$9 with LSS fusion; the reported latency is $V=V_i\cup V_s\cup V_t$0 on A100 and $V=V_i\cup V_s\cup V_t$1 with TensorRT (Wang et al., 2023). Meta-Transformer reports, among other results, ImageNet-1K zero-shot top-1 $V=V_i\cup V_s\cup V_t$2, ModelNet40 OA $V=V_i\cup V_s\cup V_t$3 with only $V=V_i\cup V_s\cup V_t$4 M trainable parameters in the frozen setting, Speech Commands v2 accuracy $V=V_i\cup V_s\cup V_t$5 frozen and $V=V_i\cup V_s\cup V_t$6 finetuned, UCF101 $V=V_i\cup V_s\cup V_t$7 frozen, and Chest X-Ray $V=V_i\cup V_s\cup V_t$8 accuracy frozen (Zhang et al., 2023).

Domain-specific MTEs also report strong results. SCT Fusion on BigEarthNet-MM achieves macro AP / $V=V_i\cup V_s\cup V_t$9 of $L=6$00, compared with $L=6$01 for early fusion, $L=6$02 for Sentinel-2 ViT, and $L=6$03 for Sentinel-1 ViT (Hoffmann et al., 2023). ME2ET reports IEMOCAP AvgAcc $L=6$04 and AvgF1 $L=6$05, and CMU-MOSEI AvgWAcc $L=6$06 and AvgF1 $L=6$07, together with $L=6$08–$L=6$09 speedup and $L=6$10–$L=6$11 GPU memory usage relative to a naive three-stream transformer without token reduction (Wu et al., 2022). In the HL-LHC event-classification setting, multimodal cross-attention reaches AUC $L=6$12, versus $L=6$13 for multimodal simple concatenation, $L=6$14 for kinematics-only self-attention, and $L=6$15 for jet-only self-attention (Hammad et al., 2023).

6. Conceptual issues, misconceptions, and research trajectory

A common misconception is that an MTE is necessarily a standard Transformer encoder in the Vaswani sense. PC-VAE directly contradicts this assumption: its so-called MTE has no learned self-attention or multi-head blocks, uses frozen random Gaussian matrices $L=6$16 and $L=6$17, has zero trainable parameters in the encoder, and aggregates modality codes by summation across column stripes or time segments (Liang et al., 2022). Another misconception is that “poster attention” or analogous phrases always imply a dedicated fusion module. In the movie recommendation system with poster attention, the poster attention is handled implicitly by ViT’s self-attention over image patches, and no additional mask or weighted gating on poster tokens is introduced beyond ViT’s own [CLS] pooling (Xia et al., 2024).

The literature also separates sharply between shared-backbone and modality-branch designs. TMT states that there are no separate modality-specific layers inside the encoder, and Meta-Transformer freezes the shared encoder after contrastive pretraining while training only the tokenizers and downstream heads (Kim et al., 2024, Zhang et al., 2023). By contrast, MuSE keeps separate text and image encoders before regularization and exchange, SCT Fusion maintains one encoder per modality with repeated synchronization, Husformer uses modality-specific Conv1D front ends followed by cross-modal and self-attention fusion, and ME2ET retains modality-specific textual, visual, and acoustic encoders together with progressive tri-modal token selection (Zhu et al., 2023, Hoffmann et al., 2023, Wang et al., 2022, Wu et al., 2022).

This suggests several stable design axes for future MTE work. One axis is how modalities are made commensurate: discrete tokenization and shared vocabulary in TMT, unified token spaces in Meta-Transformer, latent-space regularization in MuSE, or structured coordinate embeddings in UniTR and MMF-Trans (Kim et al., 2024, Zhang et al., 2023, Zhu et al., 2023, Wang et al., 2023, AI et al., 28 Jan 2025). A second axis is where cross-modal interaction occurs: full self-attention over mixed token sets, explicit token replacement, synchronized class-token exchange, hierarchical attention over modality blocks, or task-conditioned mode separation (Zhu et al., 2023, Hoffmann et al., 2023, Li et al., 2023, Huang et al., 2021). A third axis is whether the encoder is task-agnostic or tightly task-coupled: UniTR and Meta-Transformer are presented as general backbones, whereas motion forecasting, entity alignment, human-state recognition, event classification, and stock prediction tie the MTE to specialized losses, priors, and output heads (Wang et al., 2023, Zhang et al., 2023, Li et al., 2023, Wang et al., 2022, Hammad et al., 2023, AI et al., 28 Jan 2025).

Taken together, these works show that “Multi-modal Transformer Encoder” names not a single module but a design space in which token construction, parameter sharing, positional structure, fusion operator, and supervision regime are all variable. The unifying theme is encoder-centric integration of heterogeneous evidence; the decisive differences lie in how that integration is staged, constrained, and optimized.