Cross-modal Transformer (CM-T) in Multimodal Analysis

• Cross-modal Transformer (CM-T) is a mechanism that uses directional cross-attention to integrate asynchronous visual, acoustic, and textual features for sentiment analysis.
• It projects each modality into a common embedding space using 1D convolution and sinusoidal positional encoding, handling unaligned sequences effectively.
• The architecture builds cooperative feature hierarchies through multi-scale interactions and block-scale fusion, improving performance on datasets like CMU-MOSI and CMU-MOSEI.

Cross-modal Transformer (CM-T), in the context of multimodal sentiment analysis in videos, denotes a Transformer-based mechanism for modeling directional interactions among visual, acoustic, and textual streams through cross-attention. In "Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos," the exact acronym “CM-T” does not appear; instead, the authors use CT for a single-scale crossmodal transformer layer and MACT for a multi-scale attentive crossmodal transformer, both embedded in the broader MCMulT architecture (Ma et al., 2022). Within that formulation, CM-T is not a standalone model name but a family of cross-modal attention operations that learn soft alignments between asynchronous modalities, progressively construct feature hierarchies, and support both aligned and unaligned multimodal sequences.

1. Task definition and modal representations

The target problem is multimodal sentiment analysis in videos, where prediction requires integrating visual behaviors such as facial expressions and head gestures, acoustic behaviors such as paralinguistic cues, and textual/verbal content. The three modalities are denoted by $m \in \{V, A, T\}$, with raw sequences

$X_m \in \mathbb{R}^{L_m \times d_m}.$

The paper also notes an implementation convention in which text may be denoted by $L$, with raw data $X_L, X_V, X_A$ and lengths $T_{L,V,A}$ (Ma et al., 2022).

A central motivation for cross-modal transformers in this setting is that the three streams are often unaligned. Audio, video, and text operate at different sampling rates, so trivial synchronization is not generally available. The architecture is therefore designed to learn cross-modal interactions directly from unaligned sequences without manual alignment, while also performing strongly in aligned settings.

Before cross-modal interaction, each modality is projected into a common embedding dimension $d$ through temporal $1$D convolution, after which sinusoidal positional embeddings are added:

$$\hat{X}_{m} = \mathrm{Conv1D}(X_m, k_m) \in \mathbb{R}^{L_m\times d},$$

$Z^{[0]}_{m} = \hat{X}_m + PE(L_m).$

This establishes the low-level modality-specific sequence representation on which the cross-modal transformer operates.

2. Cross-modal attention and the multi-scale formulation

The core CM-T operation is directional cross-attention between a target modality and a source modality. For a target modality $m$ and source modality $X_m \in \mathbb{R}^{L_m \times d_m}.$0, the paper defines the cross-modal attention layer as

$X_m \in \mathbb{R}^{L_m \times d_m}.$1

where queries are derived from the target and keys/values from the source (Ma et al., 2022).

The formulation follows standard multi-head attention:

$X_m \in \mathbb{R}^{L_m \times d_m}.$2

with

$X_m \in \mathbb{R}^{L_m \times d_m}.$3

and

$X_m \in \mathbb{R}^{L_m \times d_m}.$4

Residual connections and layer normalization are applied around attention and feed-forward sublayers.

What distinguishes MCMulT from single-scale cross-modal transformers such as MulT is the introduction of multi-scale representations. Instead of attending only to one level of source features, MCMulT exploits progressively enhanced hidden states across layers as distinct scales. The multi-scale features for modality $X_m \in \mathbb{R}^{L_m \times d_m}.$5 are written as

$X_m \in \mathbb{R}^{L_m \times d_m}.$6

Higher-level hidden states encode progressively more abstract and context-rich semantics, while earlier states retain lower-level temporal patterns. This suggests that the CM-T mechanism is not merely a modality-to-modality aligner; it is also a hierarchy constructor that reuses semantic content from multiple representational depths.

3. Cooperative hierarchy building

The multi-scale cross-modal transformer in MCMulT is explicitly cooperative. When a modality $X_m \in \mathbb{R}^{L_m \times d_m}.$7 builds its $X_m \in \mathbb{R}^{L_m \times d_m}.$8-th level representation from source modality $X_m \in \mathbb{R}^{L_m \times d_m}.$9, it aggregates multiple preceding scales from the source side through

$L$0

with initialization

$L$1

A multi-scale attention then combines these interaction terms into an enhanced feature $L$2 (Ma et al., 2022).

The subsequent update uses a position-wise feed-forward network with residual structure:

$L$3

$L$4

The architecture is bi-directional and iterative. After one modality’s enhanced representation improves the other modality at a deeper level, that improved state is fed back to refine subsequent scales of the first modality. The paper characterizes this as progressive feature-hierarchy construction “in a cooperative manner,” and further states that it helps prevent error signals from fading across long paths. A plausible implication is that CM-T in this formulation is less a one-shot fusion layer than a recurrently interleaved hierarchy-building process across modality pairs.

4. Unaligned sequence modeling and block-scale fusion

A major technical role of CM-T in MCMulT is the handling of unaligned multimodal sequences. Because cross-attention is computed between

$L$5

with $L$6, the resulting attention map has shape $L$7. This provides soft, dynamic alignment scores between asynchronous streams without requiring manual word-level synchronization (Ma et al., 2022).

Positional encodings preserve temporal order within each modality despite differing sampling rates. The architecture further distinguishes between local and global interactions. Local interactions correspond to same-scale, single-source relations, while global interactions aggregate across multiple scales and source levels. The paper states that this balance improves robustness on unaligned data.

At the architectural level, these interactions are organized into multi-scale cross-modal Transformer blocks (MCTBs). Each block contains one multi-scale attentive cross-modal transformer (MACT) and several single-scale CT layers. MACT handles global multi-scale interactions by using outputs from multiple preceding blocks of the source modality, while CT handles local interactions using same-scale source features.

For final prediction, source-specific pathways are concatenated for each target modality:

$L$8

Each concatenated sequence is then passed through an additional Transformer for temporal aggregation and into a fully connected prediction layer. In compact form,

$L$9

5. Training objectives, complexity, and architectural trade-offs

The model is evaluated on both classification and regression-like settings. For 7-class sentiment classification or categorical emotion recognition, the paper gives cross-entropy as the standard objective:

$X_L, X_V, X_A$0

For continuous sentiment prediction, regression losses such as MAE/MSE and correlation metrics are reported; the text gives MAE in the form

$X_L, X_V, X_A$1

Optimization details are said to appear in Appendix A, and the main text states that the architecture trains stably across aligned and unaligned settings (Ma et al., 2022).

The paper also emphasizes complexity control. Naively dense multi-scale interactions increase parameters and make training harder as depth grows. MCMulT addresses this through the block mechanism, which combines dense global interactions via MACT with sparse local interactions via CT. Cross-attention between sequences of lengths $X_L, X_V, X_A$2 and $X_L, X_V, X_A$3 has time complexity

$X_L, X_V, X_A$4

per layer. The block-scale reuse and windowed local context are presented as mechanisms for managing this cost.

An ablation on CMU-MOSEI reports that performance improves as the number of blocks $X_L, X_V, X_A$5 increases to $X_L, X_V, X_A$6 and the number of CT layers per block $X_L, X_V, X_A$7 increases to $X_L, X_V, X_A$8, both described as best settings in that study. The paper also notes that increasing MulT depth beyond $X_L, X_V, X_A$9 layers degrades performance, whereas MCMulT at equivalent depth outperforms MulT-12. This suggests that the gains arise from multi-scale cooperative design rather than from naive depth expansion.

6. Empirical results, positioning, and limitations

The empirical evaluation spans CMU-MOSI, CMU-MOSEI, and IEMOCAP. CMU-MOSI contains 2,199 clips and uses metrics including Acc7, Acc2, F1, MAE, and Corr. CMU-MOSEI contains 23,454 clips with the same metric suite. IEMOCAP is evaluated with accuracy and F1 for each class (Ma et al., 2022).

On CMU-MOSI, aligned, MCMulT achieves Acc7=40.7, Acc2=83.9, F1=83.2, MAE=0.866, Corr=0.701, compared with MulT at Acc7=40.0, Acc2=83.0, F1=82.8, MAE=0.871, Corr=0.698. On CMU-MOSI, unaligned, MCMulT improves over MulT with Acc7=40.3 vs. 39.1, Acc2=82.2 vs. 81.1, and F1=82.3 vs. 81.0. On CMU-MOSEI, aligned, MCMulT improves Acc7 to 52.4, Acc2 to 83.1, and F1 to 82.8, while having comparable MAE and Corr to MulT. On CMU-MOSEI, unaligned, it outperforms MulT with Acc7=51.8 vs. 50.7, Acc2=83.0 vs. 81.6, and F1=82.8 vs. 81.6. On IEMOCAP, the paper states that MCMulT consistently improves accuracy and F1 across emotions relative to MulT and prior baselines.

With respect to terminology, the article’s central clarification is that the paper does not use the label “CM-T.” Instead, its cross-modal transformer functionality is realized through CT and MACT. Relative to MulT, MCMulT extends the single-scale directional cross-modal Transformer in three ways: multi-scale integration, cooperative hierarchy, and the block mechanism. The paper characterizes the resulting strengths as improved robustness on unaligned sequences and better capture of long-range contingencies, with the principal trade-off being added architectural complexity managed through block-scale sparsification.

The stated limitations are also specific. Overly dense multi-scale connections can inflate parameters and hinder training, as shown by weaker performance of MCMulT-Dense. Benefits depend on architectural choices rather than sheer depth, and the optimal values of block count and layer count are dataset-dependent. The authors further propose extending the model to other multimodal tasks such as VQA, image–text matching, and cross-modal pretraining. This suggests that the CM-T mechanisms described here—multi-scale cross-attention, cooperative hierarchy construction, and soft alignment across asynchronous streams—are intended as a general multimodal design pattern rather than a sentiment-analysis-specific trick.