---
title: Multi-modal Transformer Encoder Overview
url: https://www.emergentmind.com/topics/multi-modal-transformer-encoder-mte
type: topic
---

# Multi-modal Transformer Encoder Overview

Searching arXiv for recent and foundational uses of “Multi-modal Transformer Encoder” and related multimodal transformer architectures.
Multi-modal Transformer Encoder (MTE) denotes an encoder-centered mechanism for jointly processing heterogeneous modalities, but the term is used with marked heterogeneity across the arXiv literature. In some works it refers to a standard shared Transformer encoder over unified token sequences; in others it denotes parallel modality-specific encoders with explicit synchronization, hierarchical cross-modal attention, token exchange, or progressive token selection; and in one case it denotes a parameter-free encoder built from frozen random Gaussian compression matrices rather than learned self-attention. This suggests that MTE is best understood as a family of multimodal encoding strategies rather than a single canonical architecture [2309.02190] [2402.16021] [2307.10802] [2210.16174].

## 1. Terminological scope and representative usages

The phrase “Multi-modal Transformer Encoder” appears in vision-language fusion, autonomous driving, remote sensing, recommendation, video retrieval, image captioning, human-state recognition, multimodal entity alignment, event classification, finance, and general multimodal representation learning. Earlier multimodal Transformer systems for image captioning and caption-to-video retrieval already used encoder-decoder structures in which self-attention modeled intra-modal relations and cross-attention or joint attention modeled inter-modal relations, even when the term MTE was not yet standardized [1905.07841] [2007.10639].

| Usage pattern | Defining characteristic | Representative papers |
|---|---|---|
| Exchanging-based encoder | Selected tokens are mixed with the average of the other modality’s embeddings | MuSE [2309.02190] |
| Multi-modal attention encoder | Each predicted trajectory is conditioned on an independent attention mode | Motion prediction [2109.06446] |
| Shared standard encoder | Different modalities are processed as different token streams by one shared encoder | TMT [2402.16021], Meta-Transformer [2307.10802] |
| Synchronized parallel encoders | Modality-specific encoders exchange only special class tokens after each block | SCT Fusion [2306.01523] |
| Progressive tri-modal encoder | Cross-modal attention compresses long visual and acoustic token streams to length $K$ | ME2ET [2209.09768] |
| Non-standard “transformer” encoder | Frozen random projections over column stripes or time segments | PC-VAE [2210.16174] |

A central consequence of this breadth is that “encoder” may denote very different loci of fusion. In MuSE, the MTE is the CrossTransformer backbone; in TMT and Meta-Transformer it is the shared modality-agnostic backbone; in SCT Fusion it is the synchronized collection of per-modality ViT blocks; and in MoAlign it is a hierarchical modifiable self-attention stack over graph-structural, textual, and visual tokens [2309.02190] [2402.16021] [2306.01523] [2310.06365].

## 2. Recurrent architectural organizations

One recurrent organization is the **shared-parameter parallel stack**. MuSE uses two parallel Transformer-encoder stacks, each with $L=6$ layers, hidden size $d$, intermediate size $d_{ff}=4d$, and $h$ heads, while all self-attention weights, FFN weights, and layer-norm parameters are shared between the text and image stacks. Layers $1\ldots\mu$ with $\mu=2$ by default are plain Transformer layers for modality-internal contextualization; inter-modal exchange is then applied on layers $\mu+1\ldots\eta$ with $\eta=4$ by default [2309.02190].

A second organization is the **single shared encoder over tokenized modalities**. TMT tokenizes speech with a HuBERT encoder and a k-means codebook, tokenizes images with a VQ-VAE style quantizer, tokenizes text with a standard subword tokenizer, forms a single vocabulary $V=V_i\cup V_s\cup V_t$, and feeds the resulting tokens into a 12-layer encoder with $d=1024$, $d_{ff}=4096$, and $H=16$. There are no separate modality-specific layers inside the encoder; modality identity is injected through a modality-type embedding $M_{\text{type}}$ added at the input [2402.16021]. Meta-Transformer adopts a related but more general pattern: a unified data tokenizer maps each modality into $D$-dimensional tokens, after which a frozen ViT-B/16 encoder with $L=12$, $D=768$, MLP hidden size $3072$, and $h=12$ processes all modalities, while only the modality tokenizers and downstream heads are optimized per task [2307.10802].

A third organization is the **parallel modality-specific encoder with block-wise synchronization**. SCT Fusion uses one ViT-style encoder per modality, with patch embedding, a learnable class token, positional embeddings, and $r=8$ encoder blocks at embedding dimension $d_e=256$ and $h=8$ heads. After every block, the updated class tokens are concatenated, fused by a trainable linear transformation, and the synchronized token is broadcast back into every modality stream [2306.01523]. Husformer follows a different branch-specific design: modality-specific Conv1D projections first produce low-level unimodal features, cross-modal Transformer layers reinforce each modality from the concatenated fusion sequence, and a self-attention Transformer then fuses the reinforced features into a high-level representation [2209.15182].

A fourth organization is the **task-agnostic shared backbone with structured inter-modal blocks**. UniTR replaces modality-specific CNN or sparse-convolution backbones with a single Transformer stack whose weights are shared across images and LiDAR. Four DSVT blocks are used: Block 1 is intra-modal, Blocks 2–3 are inter-modal in 2D image space, and Block 4 is inter-modal in 3D space [2308.07732]. A closely related idea appears in the multi-scale cross-attention encoder for HL-LHC event classification, where jet-substructure streams and a kinematics stream are first encoded separately and then fused by two consecutive cross-attention encoder blocks [2401.00452].

## 3. Mechanisms of cross-modal interaction

The most distinctive interaction rule in the surveyed literature is MuSE’s **inter-modal token exchange**. At each exchanging layer, the model computes the multi-head self-attention of `cls` to every token, selects the bottom $\theta$ proportion of tokens with the smallest `cls`-to-token scores, and for each selected token $t$ in modality $i$ replaces its embedding by mixing in the average of the other modality’s token embeddings:
$$
E_t'=
\begin{cases}
E_t, & t\notin S \\
(1-p)\cdot E_t + p\cdot \frac{1}{|E_j|}\sum_{k=1}^{|E_j|} E_{j,k}, & t\in S
\end{cases}
$$
with default $p=0.1$. No extra projection or normalization beyond the built-in Transformer layer-norm is applied [2309.02190].

A different interaction mechanism is **mode-separating attention**. In transformer-based motion prediction for autonomous driving, the second layer of the agent–map encoder replaces standard MultiHead attention by a “MultiModal” attention that outputs each head separately rather than concatenating them:
$$
\mathrm{MultiModal}(Q,K,V)=(\mathrm{head}_1,\mathrm{head}_2,\ldots,\mathrm{head}_M)\in\mathbb{R}^{M\times d},
$$
with $M=6$. Each $Z^{(m)}$ is then combined with the target-agent history and interaction feature to form a mode-specific driving context for a trajectory decoder and a score decoder [2109.06446].

A third interaction rule is **class-token synchronization**. SCT Fusion collects the per-modality class tokens after each block, concatenates them,
$$
c_{\text{concat}}=[c_1;c_2;\dots;c_N]\in\mathbb{R}^{N\cdot d_e},
$$
and produces a synchronized class token
$$
z_{\text{sync}} = W_{\text{cls}}\,c_{\text{concat}} + b_{\text{cls}}\in\mathbb{R}^{d_e},
$$
which is copied back into the $0$-th position of each modality sequence for the next block [2306.01523].

ME2ET implements **progressive tri-modal attention** rather than full-length tri-modal self-attention. Text first guides selection of $K$ visual tokens, then text plus visual guide selection of $K$ acoustic tokens, and finally text plus acoustic refine the visual selection. This reduces self-attention complexity from $O(N^2)$ to $O(K^2)$, with $K=256$ in the reported implementation [2209.09768]. Husformer uses cross-modal transformers in which each modality attends to the concatenated low-level fusion sequence and is then followed by a self-attention transformer over the concatenated reinforced features [2209.15182].

MoAlign provides a graph-centered variant of multimodal interaction. Its hierarchical modifiable self-attention block performs three successive attentions within each layer—neighbors, then text attributes, then image attributes—under a learned mask that suppresses incompatible token interactions, while entity-type prefix vectors are prepended to keys and values in every attention and injected analogously into the feed-forward layers [2310.06365].

## 4. Tokenization, alignment, and positional structure

Token construction is a primary design axis. TMT explicitly interprets different modalities as different languages. Speech is converted into discrete units by a pre-trained HuBERT encoder followed by nearest-centroid assignment in a learned k-means codebook; image data are quantized into vision tokens by a VQ-VAE style quantizer; and text is tokenized by a standard subword tokenizer. The encoder input is
$$
x_i = E[z_i] + P[i] + M_{\text{type}}[k],
$$
where $E$ is a shared token embedding matrix, $P$ is positional embedding, and $M_{\text{type}}$ is a modality-type embedding for image, speech, or text [2402.16021].

Meta-Transformer also begins with modality-specialist tokenizers, but its emphasis is on mapping 12 modalities into a shared token space before passing them through a frozen modality-shared encoder. The tokenizer differs by modality—WordPiece for text, ViT-style patch flattening for images, FPS+KNN plus MLP for point clouds, convolutional patching for audio spectrograms, and analogous grouping-and-projection pipelines for video, hyperspectral, tabular, graph, and time-series data—yet all outputs are $D$-dimensional tokens [2307.10802].

Temporal and structural embeddings often mediate multimodal alignment. In video retrieval, the multimodal sequence is
$$
\Omega(v)=F(v)+E(v)+T(v),
$$
where $F(v)$ contains per-expert aggregated and temporal tokens, $E(v)$ is an expert-type embedding, and $T(v)$ is a learned temporal embedding including special tokens $T_{\text{agg}}$ and $T_{\text{unk}}$ [2007.10639]. UniTR instead uses spatial coordinates passed through a small MLP as positional encoding and alternates 2D projection-based and 3D unprojection-based inter-modal integration so that image and LiDAR tokens join in common structural spaces [2308.07732]. MMF-Trans introduces a three-stage time encoding,
$$
\mathrm{PosEnc}(t)=\gamma_1\,\mathrm{CalEnc}(t)+\gamma_2\,\mathrm{EvtEnc}(t)+\gamma_3\,\mathrm{DecEnc}(t),
$$
to align macro data, technical indicators, financial text, and event knowledge across mixed frequencies [2501.16621].

Alignment can also be enforced by auxiliary generation or by nonstandard token geometry. MuSE uses two decoders for embedding regularization: a TextDecoder with image-to-text cross-entropy loss $\mathcal{L}_{it}$ and an ImageDecoder with PixelCNN++ text-to-image loss $\mathcal{L}_{ti}$, and these losses pull $T_e$ and $I_e$ into a shared latent space [2309.02190]. PC-VAE abandons patch tokens in favor of column stripes for images and contiguous time segments for audio; each stripe or segment is compressed by a frozen random Gaussian matrix, and the per-modality code is the sum of the compressed parts [2210.16174].

## 5. Objectives, tasks, and empirical behavior

Training objectives range from pure task supervision to explicitly multimodal auxiliary losses. MuSE optimizes
$$
\mathcal{L}=\mathcal{L}_{task}+\alpha\,\mathcal{L}_{it}+\beta\,\mathcal{L}_{ti},
$$
with $\alpha,\beta$ set to $1$ in the default configuration. On multimodal named entity recognition, it reports $76.81$ F1 on Twitter15 versus a best prior $75.52$, $88.62$ on Twitter17 versus $87.40$, and $44.32$ on MT-Product versus a best prior $\sim 42.58$. On multimodal sentiment analysis, it reports $75.80/75.58$ Acc/F1 on MVSA-Single versus $75.19/74.97$, and $74.10/73.93$ on MVSA-Multiple versus $73.52/73.49$. Its ablations show F1 drops of $3.55$ without CrossTransformer, $2.42$ without the decoding tasks, $1.38$ without image captioning loss, and $1.54$ without text-to-image generation loss [2309.02190].

In autonomous driving, the motion-prediction MTE is trained end-to-end with
$$
\mathcal{L}=\mathcal{L}_{score}+\alpha\,\mathcal{L}_{traj},\qquad \alpha=0.5,
$$
where $\mathcal{L}_{traj}$ is a variety-style smooth-L1 loss applied only to the best mode and $\mathcal{L}_{score}$ is a cross-entropy loss against a soft ground-truth distribution over modes. On the Argoverse test set with $K=6$, the model reports $\mathrm{minADE}=0.8372\ \mathrm{m}$, $\mathrm{minFDE}=1.2905\ \mathrm{m}$, $\mathrm{brier\text{-}minFDE}=1.9393$, $\mathrm{MR}=0.1429$, and a total parameter count of $6.33$ M [2109.06446].

Task-agnostic or unified backbones show a different empirical profile. A single 270M-parameter TMT model handling all six translation directions reportedly outperforms or matches six separately trained uni-modal models; on COCO and Flickr8k it yields $+2$–$4$ BLEU in image captioning, $+3$–$5$ CLIP score in text-to-image, and substantial gains in speech-to-image, while ASR beats a standalone Wav2Vec2.0 and TTS quality rivals VITS [2402.16021]. UniTR reports val NDS $73.1$ and mAP $70.0$, test NDS $74.1$ and mAP $70.5$ for 3D detection on nuScenes, as well as BEV segmentation mIoU $73.2$ for camera-plus-LiDAR and $74.7$ with LSS fusion; the reported latency is $88.7\,\mathrm{ms}$ on A100 and $50.2\,\mathrm{ms}$ with TensorRT [2308.07732]. Meta-Transformer reports, among other results, ImageNet-1K zero-shot top-1 $69.3\%$, ModelNet40 OA $93.6\%$ with only $0.6$ M trainable parameters in the frozen setting, Speech Commands v2 accuracy $78.3\%$ frozen and $97.0\%$ finetuned, UCF101 $46.6\%$ frozen, and Chest X-Ray $94.1\%$ accuracy frozen [2307.10802].

Domain-specific MTEs also report strong results. SCT Fusion on BigEarthNet-MM achieves macro AP / $F_2$ of $0.8343 / 0.7369$, compared with $0.8227 / 0.7253$ for early fusion, $0.8159 / 0.7125$ for Sentinel-2 ViT, and $0.6796 / 0.5581$ for Sentinel-1 ViT [2306.01523]. ME2ET reports IEMOCAP AvgAcc $86.5$ and AvgF1 $61.3$, and CMU-MOSEI AvgWAcc $68.3$ and AvgF1 $48.0$, together with $1.2\times$–$1.6\times$ speedup and $48\%$–$77\%$ GPU memory usage relative to a naive three-stream transformer without token reduction [2209.09768]. In the HL-LHC event-classification setting, multimodal cross-attention reaches AUC $\approx 98.8\%$, versus $\approx 94.8\%$ for multimodal simple concatenation, $\approx 91.0\%$ for kinematics-only self-attention, and $\approx 84.4\%$ for jet-only self-attention [2401.00452].

## 6. Conceptual issues, misconceptions, and research trajectory

A common misconception is that an MTE is necessarily a standard Transformer encoder in the Vaswani sense. PC-VAE directly contradicts this assumption: its so-called MTE has no learned self-attention or multi-head blocks, uses frozen random Gaussian matrices $\Phi_i$ and $\Psi_j$, has zero trainable parameters in the encoder, and aggregates modality codes by summation across column stripes or time segments [2210.16174]. Another misconception is that “poster attention” or analogous phrases always imply a dedicated fusion module. In the movie recommendation system with poster attention, the poster attention is handled implicitly by ViT’s self-attention over image patches, and no additional mask or weighted gating on poster tokens is introduced beyond ViT’s own `[CLS]` pooling [2407.09157].

The literature also separates sharply between **shared-backbone** and **modality-branch** designs. TMT states that there are no separate modality-specific layers inside the encoder, and Meta-Transformer freezes the shared encoder after contrastive pretraining while training only the tokenizers and downstream heads [2402.16021] [2307.10802]. By contrast, MuSE keeps separate text and image encoders before regularization and exchange, SCT Fusion maintains one encoder per modality with repeated synchronization, Husformer uses modality-specific Conv1D front ends followed by cross-modal and self-attention fusion, and ME2ET retains modality-specific textual, visual, and acoustic encoders together with progressive tri-modal token selection [2309.02190] [2306.01523] [2209.15182] [2209.09768].

This suggests several stable design axes for future MTE work. One axis is **how modalities are made commensurate**: discrete tokenization and shared vocabulary in TMT, unified token spaces in Meta-Transformer, latent-space regularization in MuSE, or structured coordinate embeddings in UniTR and MMF-Trans [2402.16021] [2307.10802] [2309.02190] [2308.07732] [2501.16621]. A second axis is **where cross-modal interaction occurs**: full self-attention over mixed token sets, explicit token replacement, synchronized class-token exchange, hierarchical attention over modality blocks, or task-conditioned mode separation [2309.02190] [2306.01523] [2310.06365] [2109.06446]. A third axis is **whether the encoder is task-agnostic or tightly task-coupled**: UniTR and Meta-Transformer are presented as general backbones, whereas motion forecasting, entity alignment, human-state recognition, event classification, and stock prediction tie the MTE to specialized losses, priors, and output heads [2308.07732] [2307.10802] [2310.06365] [2209.15182] [2401.00452] [2501.16621].

Taken together, these works show that “Multi-modal Transformer Encoder” names not a single module but a design space in which token construction, parameter sharing, positional structure, fusion operator, and supervision regime are all variable. The unifying theme is encoder-centric integration of heterogeneous evidence; the decisive differences lie in how that integration is staged, constrained, and optimized.

Source: https://www.emergentmind.com/topics/multi-modal-transformer-encoder-mte