---
title: Factorized Multimodal Transformer (FMT)
url: https://www.emergentmind.com/topics/factorized-multimodal-transformer-fmt
type: topic
---

# Factorized Multimodal Transformer (FMT)

The Factorized Multimodal Transformer (FMT) is a neural architecture designed for multimodal sequential learning, explicitly modeling comprehensive intra- and inter-modal dynamics across temporal sequences. FMT is distinguished by a factorization strategy that decomposes the attention space into all nonempty subsets (“factors”) of the input modalities, allowing for fine-grained and semantically targeted modeling of modality interactions at multiple interaction scales and facilitating robust learning even in low-resource settings [1911.09826]. Subsequent efforts have generalized or specialized the factorized attention paradigm, such as in action recognition and autonomous driving contexts [2309.05032, 2508.11537].

## 1. Architectural Principles and Factorization of Modal Interactions

FMT is structured around the explicit enumeration of all nonempty subsets, or “factors,” of the modality set (typically $\{L, V, A\}$: language, vision, acoustic). For each factor, a dedicated self-attention channel is applied over the subspace corresponding to the modalities in that factor, enabling asynchronous modeling of unimodal, bimodal, and trimodal dependencies across the entire temporal sequence.

Input embedding is performed per modality using modality-specific linear projections followed by positional encoding. Resulting embeddings $\hat m_{(t,i)} = E_M(m_{(t,i)})$ are aligned to a common clock and concatenated to form the initial sequence input $\hat x_i^0 \in \mathbb{R}^{T \times e_x}$.

The transformer core consists of $K$ stacked Multimodal Transformer Layers (MTL). Each MTL contains $U$ parallel Factorized Multimodal Self-attention (FMS) units, each applying attention for all factors in parallel (seven in the three-modality case: $L$, $V$, $A$, $LV$, $LA$, $VA$, $LVA$). The output for each factor is residualized and normalized, then aggregated via a lightweight 1D convolutional summarization network ($S_1$). $U$ parallel FMS units capture distinct semantic patterns, and a second summarizer ($S_2$) collapses across units to yield the MTL output [1911.09826].

## 2. Attention Mechanisms and Mathematical Formalization

For each factor $f$, the FMS layer projects the input sequence onto the subspace corresponding to $f$, yielding $X_f \in \mathbb{R}^{T \times e_f}$. Factor-specific attention parameters $(W^f_{Q}, W^f_{K}, W^f_{V})$ yield
\[
Q^f = X_f W_Q^f,\quad K^f = X_f W_K^f, \quad V^f = X_f W_V^f.
\]
Scaled dot-product attention is computed as
\[
A^f = \mathrm{softmax}\left(\frac{Q^f (K^f)^T}{\sqrt{d}}\right)V^f
\]
with a residual and layer normalization:
\[
Z^f = \mathrm{LayerNorm}(X_f + A^f W_O^f),\quad W_O^f \in \mathbb{R}^{d \times e_x}.
\]
Outputs from all factors are stacked and reduced by $S_1$ to $T \times e_x$. $U$ FMS units are summarized by $S_2$ as
\[
\hat x_i^{k+1} = S_2( \text{concat}_{u=1}^U Y^u )
\]
where each $Y^u$ is the result of feed-forward, residual, and normalization on the $u$-th FMS output. The final output after $K$ layers is processed via a unidirectional GRU, whose last hidden state is linearly mapped to the prediction space.

Each attention head, for every factor, has full temporal receptive field, permitting modeling of long-range and asynchronous dependencies. This factorization distinguishes FMT: the heads are not split along dimension, but along semantic decomposition of modalities [1911.09826].

## 3. Optimization, Training Protocols, and Parameter Efficiency

The FMT design incorporates modality-specific features (GloVe/P2FA [language], Emotient FACET [vision], COVAREP [acoustic]) and standard sequence alignment. Training uses Adam optimization (learning rates $1\mathrm{e}{-3}$–$1\mathrm{e}{-4}$, $\beta_1=0.9$, $\beta_2=0.999$), batch size 20, and up to 200 epochs with early stopping. Dropout ($0$–$0.1$) is used at key points. Lightweight summarization CNNs ($S_1$, $S_2$) regularize the factor/unit aggregations.

Parameter sharing is achieved by representing all factor interactions within a single model stack, in contrast to models such as MulT which instantiate multiple separate transformer streams [1911.09826]. Empirical analysis establishes that FMT achieves its robust generalization and high accuracy without an explosion in parameter count, leveraging semantically targeted attention rather than simply increasing the number of heads.

## 4. Empirical Results and Comparative Evaluation

FMT establishes state-of-the-art results on three well-studied datasets and 21 label types:

- **CMU-MOSI (Sentiment Analysis):** FMT achieves BA=81.5%/83.5% (neg/non-neg; neg/pos), F1=81.4/83.5, MAE=0.837, Corr=0.744. Against MulT’s best BA=83.0, F1=82.8, MAE=0.87, Corr=0.698, FMT shows improvements in F1 and correlation [1911.09826].
- **IEMOCAP (Emotion Recognition):** For “Sad,” FMT: BA=88.0 vs. MulT: 86.7; “Angry,” FMT: 89.7 vs. MulT: 87.4; “Neutral,” FMT: 74.0 vs. 72.4.
- **POM (Speaker Traits):** FMT outperforms MulT across all 16 traits. E.g., "Confident": MA7=40.9% vs. 34.5%; "Passionate": 42.4% vs. 34.5%; "Humorous": 48.3% vs. 43.3%.

Ablation studies demonstrate that removing any factor category (unimodal, bimodal, trimodal) leads to performance drops of 1–2% BA, and substituting summarization with naive reductions loses approximately 1% BA. FMT with a single FMS unit (U=1, 7 heads) outperforms a standard Transformer with up to 35 heads (full temporal field) by >5% BA, confirming semantic factorization as central [1911.09826].

## 5. Generalization, Scalability, and Related Models

**Scalability**: The number of factors grows exponentially with the number of modalities ($2^M-1$), presenting computational challenges for applications beyond three modalities. Practical mitigations include domain-informed pruning of weak factors and greedy or stepwise factor selection.

**Generalization**: FMT is robust under low-resource conditions, attributed to its capacity control through factorized and summarized attention, permitting effective learning from datasets ranging from ~2,000 to 5,000 samples [1911.09826].

**Parameter Efficiency**: FMT’s factorized structure attains parameter savings versus architectures that implement parallel unimodal and cross-modal transformer stacks. Analogous factorized paradigms have been developed subsequently. The Unified Contrastive Fusion Transformer (UCFFormer) utilizes Factorized Time–Modality Attention for modality-and-time factorization, yielding substantial FLOP and parameter reductions over full joint attention, with no accuracy loss [2309.05032]. In the autonomous driving domain, MultiPark achieves explicit factorization over gear, longitudinal, and lateral parking behaviors for efficient multimodal path prediction, outperforming non-factorized baselines in both accuracy and speed [2508.11537].

| Model         | Main Factorization Strategy             | Parameter Efficiency | Representative Domains           |
|---------------|----------------------------------------|---------------------|----------------------------------|
| FMT           | Explicit (all subsets of modalities)    | High                | Multimodal sequential learning   |
| UCFFormer     | Time–Modality axis                     | High                | Human action recognition         |
| MultiPark     | Parking behavior (gear/lon/lat modes)  | High                | Autonomous vehicle planning      |

## 6. Limitations and Future Directions

Several open challenges remain. The exponential growth of factors for large $M$ motivates automated factor pruning, dynamic gating per example, or cross-layer/hierarchical strategies. Integration with pretrained multimodal representations (e.g., vision–language transformers), extension to online or more fine-grained alignment (frame-level), and exploitation of outcome-oriented objectives are cited as active research areas. The principled application of factorization—targeting semantic intersections of modalities—remains central to recent advances in multimodal modeling, as evidenced by extensions in action recognition and control domains [1911.09826, 2309.05032, 2508.11537].

Source: https://www.emergentmind.com/topics/factorized-multimodal-transformer-fmt