---
title: Cross-modal Transformer (CM-T) in Multimodal Analysis
url: https://www.emergentmind.com/topics/cross-modal-transformer-cm-t
type: topic
---

# Cross-modal Transformer (CM-T) in Multimodal Analysis

Cross-modal Transformer (CM-T), in the context of multimodal sentiment analysis in videos, denotes a Transformer-based mechanism for modeling directional interactions among visual, acoustic, and textual streams through cross-attention. In "Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos," the exact acronym “CM-T” does not appear; instead, the authors use **CT** for a single-scale crossmodal transformer layer and **MACT** for a multi-scale attentive crossmodal transformer, both embedded in the broader **MCMulT** architecture [2206.07981]. Within that formulation, CM-T is not a standalone model name but a family of cross-modal attention operations that learn soft alignments between asynchronous modalities, progressively construct feature hierarchies, and support both aligned and unaligned multimodal sequences.

## 1. Task definition and modal representations

The target problem is multimodal sentiment analysis in videos, where prediction requires integrating **visual** behaviors such as facial expressions and head gestures, **acoustic** behaviors such as paralinguistic cues, and **textual/verbal** content. The three modalities are denoted by $m \in \{V, A, T\}$, with raw sequences
$$
X_m \in \mathbb{R}^{L_m \times d_m}.
$$
The paper also notes an implementation convention in which text may be denoted by $L$, with raw data $X_L, X_V, X_A$ and lengths $T_{L,V,A}$ [2206.07981].

A central motivation for cross-modal transformers in this setting is that the three streams are often **unaligned**. Audio, video, and text operate at different sampling rates, so trivial synchronization is not generally available. The architecture is therefore designed to learn cross-modal interactions directly from unaligned sequences without manual alignment, while also performing strongly in aligned settings.

Before cross-modal interaction, each modality is projected into a common embedding dimension $d$ through temporal $1$D convolution, after which sinusoidal positional embeddings are added:
$$
\hat{X}_{m} = \mathrm{Conv1D}(X_m, k_m) \in \mathbb{R}^{L_m\times d},
$$
$$
Z^{[0]}_{m} = \hat{X}_m + PE(L_m).
$$
This establishes the low-level modality-specific sequence representation on which the cross-modal transformer operates.

## 2. Cross-modal attention and the multi-scale formulation

The core CM-T operation is directional cross-attention between a **target** modality and a **source** modality. For a target modality $m$ and source modality $n$, the paper defines the cross-modal attention layer as
$$
CM_{n\to m}(Z_m, Z_n) =
\mathrm{softmax}\!\left(
\frac{Z_m W_{Q_m} W_{K_n}^T Z_n^T}{\sqrt{d_k}}
\right) Z_n W_{V_n},
$$
where queries are derived from the target and keys/values from the source [2206.07981].

The formulation follows standard multi-head attention:
$$
\mathrm{MHA}(Q,K,V) = [h_1; \dots; h_H] W_O,
$$
with
$$
h_i = \mathrm{Attn}(Q W_Q^{(i)}, K W_K^{(i)}, V W_V^{(i)}),
$$
and
$$
\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V.
$$
Residual connections and layer normalization are applied around attention and feed-forward sublayers.

What distinguishes MCMulT from single-scale cross-modal transformers such as MulT is the introduction of **multi-scale representations**. Instead of attending only to one level of source features, MCMulT exploits progressively enhanced hidden states across layers as distinct scales. The multi-scale features for modality $m$ are written as
$$
\{X_m^{(\ell)}\}_{\ell=1}^{S_m}.
$$
Higher-level hidden states encode progressively more abstract and context-rich semantics, while earlier states retain lower-level temporal patterns. This suggests that the CM-T mechanism is not merely a modality-to-modality aligner; it is also a hierarchy constructor that reuses semantic content from multiple representational depths.

## 3. Cooperative hierarchy building

The multi-scale cross-modal transformer in MCMulT is explicitly **cooperative**. When a modality $\alpha$ builds its $i$-th level representation from source modality $\beta$, it aggregates multiple preceding scales from the source side through
$$
H^{[i]} =
\left\{
CM_{\beta\to\alpha}(Z_{\beta\to\alpha}^{[i-1]}, Z_{\alpha\to\beta}^{[j]})
\right\}
\quad \text{for } j=0,\dots,i-1,
$$
with initialization
$$
Z_{\alpha\to\beta}^{[0]} = Z_{\beta}^{[0]}.
$$
A multi-scale attention then combines these interaction terms into an enhanced feature $A_{\alpha\to\beta}^{[i]}$ [2206.07981].

The subsequent update uses a position-wise feed-forward network with residual structure:
$$
P^{[i]}_{\beta\to\alpha}
=
f_\theta\!\left(
LN\!\left(
A_{\alpha\to\beta}^{[i]} + LN(Z_{\beta\to\alpha}^{[i-1]})
\right)
\right),
$$
$$
Z_{\beta\to\alpha}^{[i]}
=
\left\{
A_{\alpha\to\beta}^{[i]} + LN(Z_{\beta\to\alpha}^{[i-1]})
\right\}
+ P^{[i]}_{\beta\to\alpha}.
$$

The architecture is bi-directional and iterative. After one modality’s enhanced representation improves the other modality at a deeper level, that improved state is fed back to refine subsequent scales of the first modality. The paper characterizes this as progressive feature-hierarchy construction “in a cooperative manner,” and further states that it helps prevent error signals from fading across long paths. A plausible implication is that CM-T in this formulation is less a one-shot fusion layer than a recurrently interleaved hierarchy-building process across modality pairs.

## 4. Unaligned sequence modeling and block-scale fusion

A major technical role of CM-T in MCMulT is the handling of **unaligned multimodal sequences**. Because cross-attention is computed between
$$
Q_m \in \mathbb{R}^{L_m \times d_k}
\quad \text{and} \quad
K_n \in \mathbb{R}^{L_n \times d_k},
$$
with $L_m \neq L_n$, the resulting attention map has shape $L_m \times L_n$. This provides soft, dynamic alignment scores between asynchronous streams without requiring manual word-level synchronization [2206.07981].

Positional encodings preserve temporal order within each modality despite differing sampling rates. The architecture further distinguishes between **local** and **global** interactions. Local interactions correspond to same-scale, single-source relations, while global interactions aggregate across multiple scales and source levels. The paper states that this balance improves robustness on unaligned data.

At the architectural level, these interactions are organized into **multi-scale cross-modal Transformer blocks (MCTBs)**. Each block contains one **multi-scale attentive cross-modal transformer (MACT)** and several single-scale **CT** layers. MACT handles global multi-scale interactions by using outputs from multiple preceding blocks of the source modality, while CT handles local interactions using same-scale source features.

For final prediction, source-specific pathways are concatenated for each target modality:
$$
[ Z_{V\to T}^{[D]}, Z_{A\to T}^{[D]} ], \quad
[ Z_{T\to V}^{[D]}, Z_{A\to V}^{[D]} ], \quad
[ Z_{T\to A}^{[D]}, Z_{V\to A}^{[D]} ].
$$
Each concatenated sequence is then passed through an additional Transformer for temporal aggregation and into a fully connected prediction layer. In compact form,
$$
\hat{y}
=
f_{cls}\!\left(
[
Agg(\{Z_{V\to T}^{[D]}, Z_{A\to T}^{[D]}\}),
Agg(\{Z_{T\to V}^{[D]}, Z_{A\to V}^{[D]}\}),
Agg(\{Z_{T\to A}^{[D]}, Z_{V\to A}^{[D]}\})
]
\right).
$$

## 5. Training objectives, complexity, and architectural trade-offs

The model is evaluated on both classification and regression-like settings. For 7-class sentiment classification or categorical emotion recognition, the paper gives cross-entropy as the standard objective:
$$
L_{CE} = - \sum_{c=1}^C y_c \log p_c.
$$
For continuous sentiment prediction, regression losses such as MAE/MSE and correlation metrics are reported; the text gives MAE in the form
$$
L_{MAE} = \frac{1}{N} \sum_{i=1}^N | y_i - \hat{y}_i |.
$$
Optimization details are said to appear in Appendix A, and the main text states that the architecture trains stably across aligned and unaligned settings [2206.07981].

The paper also emphasizes complexity control. Naively dense multi-scale interactions increase parameters and make training harder as depth grows. MCMulT addresses this through the block mechanism, which combines dense global interactions via MACT with sparse local interactions via CT. Cross-attention between sequences of lengths $L_m$ and $L_n$ has time complexity
$$
O(L_m L_n)
$$
per layer. The block-scale reuse and windowed local context are presented as mechanisms for managing this cost.

An ablation on CMU-MOSEI reports that performance improves as the number of blocks $B$ increases to $4$ and the number of CT layers per block $L$ increases to $3$, both described as best settings in that study. The paper also notes that increasing MulT depth beyond $5$ layers degrades performance, whereas MCMulT at equivalent depth outperforms MulT-12. This suggests that the gains arise from multi-scale cooperative design rather than from naive depth expansion.

## 6. Empirical results, positioning, and limitations

The empirical evaluation spans **CMU-MOSI**, **CMU-MOSEI**, and **IEMOCAP**. CMU-MOSI contains **2,199 clips** and uses metrics including **Acc7, Acc2, F1, MAE, and Corr**. CMU-MOSEI contains **23,454 clips** with the same metric suite. IEMOCAP is evaluated with **accuracy and F1 for each class** [2206.07981].

On **CMU-MOSI, aligned**, MCMulT achieves **Acc7=40.7, Acc2=83.9, F1=83.2, MAE=0.866, Corr=0.701**, compared with MulT at **Acc7=40.0, Acc2=83.0, F1=82.8, MAE=0.871, Corr=0.698**. On **CMU-MOSI, unaligned**, MCMulT improves over MulT with **Acc7=40.3 vs. 39.1**, **Acc2=82.2 vs. 81.1**, and **F1=82.3 vs. 81.0**. On **CMU-MOSEI, aligned**, MCMulT improves **Acc7 to 52.4**, **Acc2 to 83.1**, and **F1 to 82.8**, while having comparable **MAE** and **Corr** to MulT. On **CMU-MOSEI, unaligned**, it outperforms MulT with **Acc7=51.8 vs. 50.7**, **Acc2=83.0 vs. 81.6**, and **F1=82.8 vs. 81.6**. On **IEMOCAP**, the paper states that MCMulT consistently improves accuracy and F1 across emotions relative to MulT and prior baselines.

With respect to terminology, the article’s central clarification is that the paper does **not** use the label “CM-T.” Instead, its cross-modal transformer functionality is realized through **CT** and **MACT**. Relative to MulT, MCMulT extends the single-scale directional cross-modal Transformer in three ways: **multi-scale integration**, **cooperative hierarchy**, and the **block mechanism**. The paper characterizes the resulting strengths as improved robustness on unaligned sequences and better capture of long-range contingencies, with the principal trade-off being added architectural complexity managed through block-scale sparsification.

The stated limitations are also specific. Overly dense multi-scale connections can inflate parameters and hinder training, as shown by weaker performance of **MCMulT-Dense**. Benefits depend on architectural choices rather than sheer depth, and the optimal values of block count and layer count are dataset-dependent. The authors further propose extending the model to other multimodal tasks such as **VQA**, **image–text matching**, and **cross-modal pretraining**. This suggests that the CM-T mechanisms described here—multi-scale cross-attention, cooperative hierarchy construction, and soft alignment across asynchronous streams—are intended as a general multimodal design pattern rather than a sentiment-analysis-specific trick.

Source: https://www.emergentmind.com/topics/cross-modal-transformer-cm-t