Cross-Modal Transformer (CMT)

• Cross-Modal Transformer is a deep learning paradigm that uses attention to fuse heterogeneous modalities for enhanced tasks such as emotion recognition and 3D detection.
• It employs architectures ranging from single-stream to two-stream designs, enabling explicit inter-modal interactions through self- and cross-attention mechanisms.
• CMT applications span vision, audio, and language, significantly improving robustness and information fusion compared to unimodal methods.

Searching arXiv for recent and foundational papers on Cross-Modal Transformer architectures and applications. Cross-Modal Transformer (CMT) denotes a transformer-based model that takes embeddings from multiple modalities and uses attention mechanisms to model interactions across these modalities, producing a fused representation optimized for a downstream task such as emotion recognition (Zhong et al., 21 Nov 2025). In the language–vision literature, cross-modal transformers are transformer-based cross-modal architectures that jointly process language and vision through self-attention and cross-attention, typically in either single-stream or two-stream form (Shin et al., 2021). Subsequent work applies the same general idea to visual, auditory, textual, geometric, and symbolic inputs, including emotion recognition, 3D object detection, speech correction, multimodal classification, pansharpening, collaborative perception, and music translation (Yan et al., 2023).

1. Definition and architectural scope

A Cross-Modal Transformer is a transformer architecture designed to jointly process and fuse information coming from multiple modalities so that each modality can attend to the others and the model can exploit their interactions (Pei et al., 18 Apr 2025). In one explicit formulation, the multimodal input at time $t$ is

$X_t = \{x^v_t, x^a_t, x^t_t\},$

and the emotion recognition mapping is

$f_e: X_t \rightarrow y_t \in \mathbb{R}^C,$

where $C$ is the number of emotion classes (Zhong et al., 21 Nov 2025).

The survey literature identifies two main structural patterns. In a single-stream CMT, image and text tokens are concatenated into a single sequence and processed by a standard transformer encoder, so cross-modal interactions are learned implicitly through self-attention (Shin et al., 2021). In a two-stream CMT, visual and textual streams are maintained separately, and explicit co-attention or cross-attention layers let queries from one modality attend to keys and values from the other (Shin et al., 2021). Audio–language work also introduces an asymmetric two-stream design in which text is encoded first and audio explicitly attends to the final text representations (Li et al., 2021).

This diversity suggests that CMT is better understood as an architectural paradigm than as a single canonical block. The common invariant is not a fixed topology, but the use of attention to make inter-modal dependencies explicit and trainable.

2. Core attention mechanism and fusion logic

A recurring formulation begins by encoding each modality into a shared embedding dimension $D$. In the cloud-based emotion-recognition model, the visual, audio, and text streams are encoded as

$$z_v \in \mathbb{R}^D,\qquad z_a \in \mathbb{R}^D,\qquad z_t \in \mathbb{R}^D,$$

then combined into the sequence

$[z_v; z_a; z_t] \in \mathbb{R}^{3 \times D}.$

Queries, keys, and values are computed from the concatenated modality embeddings,

$$Q = W_Q [z_v; z_a; z_t],\quad K = W_K [z_v; z_a; z_t],\quad V = W_V [z_v; z_a; z_t],$$

and attention is

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.$$

This makes cross-modal dependencies explicit because every modality token can attend to every other modality token (Zhong et al., 21 Nov 2025).

In audio–language pre-training, the same scaled dot-product attention is used inside a text-referred audio stream. There, audio performs self-attention over audio frames and then cross-attention over the final text representation, so the model learns both intra-audio and inter-modal structure (Li et al., 2021). In speech-correction work, cross-modality is implemented even more directly: speech frames and ASR hypothesis tokens are concatenated into a single sequence separated by a special token, and self-attention is applied over the entire combined sequence (Tanaka et al., 2021). In that formulation, cross-modal interaction is not a separate module; it emerges from unified self-attention over speech and text.

The functional motivation is consistent across domains. Unimodal systems are fragile under noise, occlusion, or missing information, while late fusion loses fine-grained inter-modal dependencies. CMTs instead fuse at the representation level, allowing one modality to re-weight, contextualize, or compensate for another (Zhong et al., 21 Nov 2025).

3. Major architectural variants

The literature contains several recurring CMT variants, each tied to a particular alignment assumption or task structure.

Variant	Modalities	Distinctive mechanism
Symmetric joint fusion	vision, audio, text	Concatenate modality embeddings and apply attention over all modality embeddings
Cascaded fusion	audio, multilingual text	Fuse text modalities first, then fuse audio with multilingual text
Asymmetric two-stream	audio, language	Text encoder plus text-referred audio stream
Geometry-aware fusion	LiDAR, RGB	Point–pixel aligned symmetric cross-attention
Token-exchange fusion	text, vision	Shared-parameter transformers with token exchange
Modulation-based fusion	PAN, MS	Modulator reshapes the carrier’s value matrix

A cascaded cross-modal transformer first combines two text modalities and then combines acoustic features with the multilingual features previously learned by the first transformer block (Ristea et al., 2023, Ristea et al., 2024). In the first stage, one language provides queries and the other provides keys and values; in the second stage, audio queries attend to the multilingual text representation. This architecture is explicitly stage-wise rather than flat, and the papers argue that it allows progressive refinement of multilingual and cross-modal structure.

A geometry-aware CMT for 3D object detection uses aligned point features and image features at projected pixel locations. Its cross-modal transformer performs symmetric, bidirectional attention between LiDAR and image tokens and concatenates raw single-modal features with cross-modal augmented features (Zhang et al., 2022). A related end-to-end 3D detector uses image tokens and point-cloud tokens without explicit view transformation, performing alignment by encoding the 3D points into multi-modal features and letting object queries attend to the concatenated multi-modal token set (Yan et al., 2023).

An exchanging-based multimodal fusion model uses two transformer encoders with shared parameters and performs inter-modal exchange by selecting a proportion of tokens in one modality and replacing their embeddings with the average of embeddings in the other modality (Zhu et al., 2023). This is not cross-attention in the usual sense; it is token exchange inside a shared transformer backbone.

A cross modulation transformer for pansharpening replaces standard value aggregation with modulation. If $V_i$ is the carrier value matrix and $X_t = \{x^v_t, x^a_t, x^t_t\},$0 is the modulator, the model computes

$X_t = \{x^v_t, x^a_t, x^t_t\},$1

then performs modulation-aware attention with the modulated value matrix (Shu et al., 2024). This is still a cross-modal transformer, but its interaction rule is multiplicative modulation rather than standard cross-attention.

4. Representation, tokenization, and alignment

CMT performance depends heavily on how modalities are tokenized and aligned before fusion. In the emotion-recognition model, the visual stream uses ViT on facial images or video frames, the acoustic stream uses Wav2Vec2, and the textual stream uses BERT, with one synchronized embedding per utterance before fusion (Zhong et al., 21 Nov 2025). Temporal alignment is utterance-level rather than frame-level: for IEMOCAP and MELD, one embedding per modality is assumed synchronized at that granularity (Zhong et al., 21 Nov 2025).

In audio-textual classification, token counts are normalized across modalities by sampling or duplicating tokens to obtain a fixed $X_t = \{x^v_t, x^a_t, x^t_t\},$2 tokens per modality, preserving a class token and enabling cross-attention between equal-length sequences (Ristea et al., 2024). This choice is explicitly motivated by computational tractability and uniform transformer input shape.

In multi-agent collaborative perception, the communicated units are not dense feature maps but object queries ranked by classification score. Each query carries an embedding vector, a 3D center, and class scores; alignment is performed by transforming remote queries into the ego frame and applying masked self-attention over the aligned query set (Wang et al., 13 Mar 2025). This suggests a broader interpretation of “modality”: in that work, different agents’ query streams and sensor configurations are treated as the entities to be fused.

In unified music translation, the modalities themselves are tokenized into sequences: score images through an RQVAE tokenizer, audio through DAC, symbolic notation through Linearized MusicXML, and MIDI through an event vocabulary. A unified tokenization framework discretizes score images, audio, MIDI, and MusicXML into a sequence of tokens, enabling a single encoder-decoder Transformer to tackle multiple cross-modal translation tasks as one coherent sequence-to-sequence task (Jung et al., 19 May 2025). This is a particularly strong example of CMT as a modality-agnostic sequence model.

5. Objectives, supervision, and optimization

CMT training objectives vary with task structure. In multimodal emotion recognition, the model is trained end-to-end with cross-entropy loss,

$X_t = \{x^v_t, x^a_t, x^t_t\},$3

using pretrained ViT, BERT, and Wav2Vec2 encoders, AdamW, a learning rate of $X_t = \{x^v_t, x^a_t, x^t_t\},$4, batch size 64, dropout 0.1, and early stopping (Zhong et al., 21 Nov 2025).

Audio–language pre-training introduces two self-supervised proxy tasks: masked language modeling and masked cross-modal acoustic modeling, with total loss

$X_t = \{x^v_t, x^a_t, x^t_t\},$5

The audio stream reconstructs masked acoustic segments while attending to text, and fine-tuning adds a fusion mechanism with max pooling, attention pooling, and orthogonal regularization between audio and text embeddings (Li et al., 2021). This demonstrates that CMTs need not rely exclusively on supervised downstream labels; pre-training can itself be cross-modal.

In 3D detection, CMT-style models are trained with task-specific losses. CAT-Det combines proposal and refinement losses with point-level and object-level contrastive losses,

$X_t = \{x^v_t, x^a_t, x^t_t\},$6

and reports that the contrastive objectives stabilize cross-modal alignment under one-way augmentation (Zhang et al., 2022). End-to-end 3D multi-modal detection instead uses Hungarian bipartite matching, focal loss for classification, and $X_t = \{x^v_t, x^a_t, x^t_t\},$7 loss for box regression (Yan et al., 2023).

Other domains replace cross-attention losses with task-specific constraints. Pansharpening uses a hybrid loss

$X_t = \{x^v_t, x^a_t, x^t_t\},$8

combining spatial-domain $X_t = \{x^v_t, x^a_t, x^t_t\},$9, Fourier-domain loss, and wavelet-domain loss (Shu et al., 2024). Financial forecasting fuses modalities through a temporally aligned tensor, then applies group LASSO and a transformer encoder over time (Pei et al., 18 Apr 2025). This suggests that “cross-modal transformer” does not uniquely determine the loss design; the interaction mechanism and the supervision protocol can vary substantially.

6. Applications, limitations, and terminological ambiguities

CMTs are used in multimodal emotion recognition and adaptive human–computer interaction, request and complaint detection, audio–language classification, automatic speech recognition correction, 3D object detection, collaborative perception, pansharpening, financial market forecasting, pedestrian crossing intention prediction, and unified translation across score images, symbolic music, MIDI, and audio (Zhong et al., 21 Nov 2025, Ristea et al., 2024, Tanaka et al., 2021, Zhang et al., 2022, Wang et al., 13 Mar 2025, Shu et al., 2024, Pei et al., 18 Apr 2025, Li et al., 25 Nov 2025, Jung et al., 19 May 2025). The breadth of these applications indicates that CMT is a general fusion principle rather than a domain-specific architecture.

Several limitations recur. One line of work notes supervised-only learning, limited adaptability in dynamic, open-world scenarios, and cloud privacy and computational overhead when sensitive video and audio are processed remotely (Zhong et al., 21 Nov 2025). Audio-text models depend on ASR and NMT quality, so errors in transcripts or translations propagate into fusion (Ristea et al., 2024). Geometry-aware models require reliable calibration and can incur substantial compute when operating over high-resolution or long token sequences (Zhang et al., 2022, Yan et al., 2023). The audio–language literature also highlights the quadratic cost of self-attention over long audio sequences and the possibility that asymmetric fusion may not suit every task (Li et al., 2021). These observations suggest that robustness, efficiency, and alignment quality remain central design constraints.

A common misconception is that “CMT” always denotes the same model family. In fact, the acronym is overloaded. It can refer to a cloud-based Cross-Modal Transformer for emotion recognition (Zhong et al., 21 Nov 2025), a Cross Modal Transformer for end-to-end 3D multi-modal detection (Yan et al., 2023), or a Cross Modulation Transformer for pansharpening (Shu et al., 2024). A further terminological exception is “CMT” as Cross-Modal Trigger in hateful meme backdoor attacks, where the term denotes a trigger rather than a transformer architecture (Wang et al., 2024). This ambiguity makes task context essential when interpreting the acronym.

Taken together, the literature defines Cross-Modal Transformer not as a single block but as a family of transformer-based systems in which attention is used to encode intra-modal structure and inter-modal alignment. The exact realization may be single-stream, two-stream, cascaded, geometry-aware, exchange-based, modulation-based, or encoder–decoder, but the central objective remains the same: to learn fused representations in which complementary information from heterogeneous modalities can be jointly exploited.