---
title: Multi-Stream Action Transformer (MSAT)
url: https://www.emergentmind.com/topics/multi-stream-action-transformer-msat
type: topic
---

# Multi-Stream Action Transformer (MSAT)

A Multi-Stream Action Transformer (MSAT) is a transformer-based architecture designed for fine-grained action understanding in videos, characterized by explicit separation and interaction of multiple feature streams—typically spatial and motion modalities—using modality-aware attention. MSAT models leverage two (or more) streams of raw video data, advanced multi-modal attention mechanisms, and modular transformer stacks to capture both intra- and inter-modality dependencies, enabling robust per-frame action localization and classification, particularly in untrimmed or long-form video input [2305.19624][2208.01753].

## 1. Input Modalities and Feature Representation

MSAT architectures operate on multiple, semantically distinct input channels. Commonly, this includes a spatial stream derived from RGB frames and a motion stream based on optical flow or spatio-temporal encodings. Each input stream is independently embedded via a modality-specific backbone:

- **Spatial stream**: $I^S = \{I^s_t\}_{t=1}^T$, processed by a $\rho_{\rm spatial}$ backbone (e.g., I3D or ResNet-18), generating $X^S \in \mathbb{R}^{T \times Z}$.
- **Motion stream**: $I^{M'} = \{I^{m'}_t\}_{t=1}^T$ (possibly distorted optical flow) is subjected to correction (see Section 4), then processed by $\rho_{\rm motion}$ yielding $X^M \in \mathbb{R}^{T \times Z}$ with $Z=1024$ [2305.19624].

Alternative instantiations (e.g., STAN [2208.01753]) may derive temporal stream tokens from low-resolution RGB clips using 3D CNNs, while spatial tokens are extracted from center frames using 2D CNNs. Positional embeddings and learned CLS tokens are appended to each sequence.

## 2. Transformer Backbone and Layer Design

The fused MSAT representation is processed by multi-layer transformer stacks, designed to aggregate information across time and modalities. Each of the $L$ layers consists of:

- **LayerNorm $\rightarrow$ Multi-Modal Attention (MMA) $\rightarrow$ residual addition**
- **LayerNorm $\rightarrow$ two-layer MLP (hidden size $Z$) $\rightarrow$ residual addition**

In the canonical MSAT (as in [2305.19624]), the $l$-th layer operates as:
\[
\begin{align*}
\widehat{O}^l &= {\rm MMA}({\rm Norm}(O^{l-1})) + O^{l-1} \\
O^l           &= {\rm MLP}({\rm Norm}(\widehat{O}^l)) + \widehat{O}^l
\end{align*}
\]
with the first layer input $O^0$ given by concatenation of $X^S$ and $X^M$. The backbone typically uses $L=6$ layers and $H=3$ multi-modal heads per layer.

Alternative two-stream transformers [2208.01753] instantiate standard multi-head self-attention for each stream individually, with per-stream $L$-layer stacks. Fusion occurs via extraction and combination (typically weighted) of the modality-wise CLS tokens after the transformer encoders.

## 3. Multi-Modal Attention and Stream Interaction

The MSAT's distinguishing feature is its parameterized multi-modal attention. For each layer and head, queries, keys, and values are projected independently for spatial and motion streams:
\[
\begin{align*}
Q^S = X^S W_q^S, &\quad K^S = X^S W_k^S, \quad V^S = X^S W_v^S, \\
Q^M = X^M W_q^M, &\quad K^M = X^M W_k^M, \quad V^M = X^M W_v^M
\end{align*}
\]
Attention sub-maps are computed as:
\[
\begin{aligned}
{\rm Attn}^{S-S} &= {\rm Softmax}\left(\frac{Q^S (K^S)^\top}{\sqrt{Z_m}}\right)V^S, \\ 
{\rm Attn}^{S-M} &= {\rm Softmax}\left(\frac{Q^S (K^M)^\top}{\sqrt{Z_m}}\right)V^M, \\
{\rm Attn}^{M-S} &= {\rm Softmax}\left(\frac{Q^M (K^S)^\top}{\sqrt{Z_m}}\right)V^S, \\
{\rm Attn}^{M-M} &= {\rm Softmax}\left(\frac{Q^M (K^M)^\top}{\sqrt{Z_m}}\right)V^M \\
\end{aligned}
\]
where $Z_m=512$. The four attention maps are concatenated across a modality dimension and fused by a $1\times1$ convolution:
\[
{\rm MMA}(X^S,X^M) = {\rm Conv}_{1\times1}\left[
{\rm Attn}^{S-S} \Vert
{\rm Attn}^{S-M} \Vert
{\rm Attn}^{M-S} \Vert
{\rm Attn}^{M-M}
\right]
\]
This scheme supports both intra-stream (S–S, M–M) and cross-stream (S–M, M–S) interactions, a critical factor in performance improvements on complex video action datasets [2305.19624]. In contrast, some designs [2208.01753] exclusively use independent stream-wise self-attention but can optionally incorporate cross-stream attention at fusion or intermediate layers.

## 4. Motion Distortion Correction

Camera-induced motion distortion in optical flow degrades motion features in unconstrained video. MSAT addresses this by a three-step correction procedure [2305.19624]:

1. **Foreground–background segmentation**: Person detector separates background motion vectors $I_B^{V'} = \{s'_n\}$.
2. **GMM Fitting**: A Gaussian Mixture Model ($M=16$ components) is fitted to the background flow, updated iteratively via the EM algorithm:

    E-step:
    \[
    P(z_{nm}=1|s'_n) = \frac{\pi_m \mathcal N(s'_n|\mu_m,\Sigma_m)}{\sum_{i}\pi_i \mathcal N(s'_n|\mu_i,\Sigma_i)}
    \]
    M-step:
    \[
    \hat\mu_m = \frac{1}{\sum_n P(z_{nm})}\sum_n P(z_{nm})s'_n
    \]
3. **Motion restoration**: Each vector $s'_n$ is corrected by subtracting the mean of its assigned Gaussian component, $s_n = s'_n - \mu_{m(n)}$, yielding the corrected flow $I^M$ and vector field $I^V$.

This correction achieves substantial robustness compared to using raw flow or more elaborate pose/depth/flow fusion, as evidenced by the associated ablation studies.

## 5. Output Head, Fusion, and Objective

After $L$ transformer layers, a unified joint representation $O^L \in \mathbb{R}^{T \times Z}$ is produced. For dense, frame-wise action detection:

- A 1D convolution (kernel width=3) projects $O^L$ to per-frame logits, yielding $\hat Y_t = {\rm Softmax}({\rm Conv}(O^L))_t$
- Post-processing merges predictions above threshold into action segments [2305.19624]

Alternative two-stream designs (e.g., [2208.01753]) perform late fusion of stream-level representations via layernorm/MLPs/weighted sums, followed by a multi-layer classification head.

The loss combines per-frame cross-entropy and a temporal IoU (tIoU) regularizer:
\[
\mathcal{L} = -\sum_{t=1}^T\sum_{c=1}^C y_{t,c} \log \hat y_{t,c} + \alpha\,{\rm Loss}_{\rm tIoU}\, , \quad \alpha=1
\]
For long-form classification benchmarks, the loss reduces to binary cross-entropy over segment or video-level predictions [2208.01753].

## 6. Training Regimes and Implementation

- **Backbone:** I3D or R(2+1)D (motion), ResNet-18 (spatial), pre-trained on Kinetics/ImageNet as appropriate.
- **Optimization:** Adam or AdamW, learning rates $1\times10^{-5}$ to $1\times10^{-4}$; weight decay $1\times10^{-6}$ to $1\times10^{-4}$.
- **Sequence length:** Up to $T=2304$ frames for detection tasks; scene-level tokens for long-form classification.
- **Regularization:** Dropout, data augmentation, and LayerNorm; I3D backbone regularization carried over.
- **Parameter count:** STAN-Small $\sim45$M, STAN-Large $\sim92.5$M [2208.01753].
- **Ablations** show all four MMA types (S–S, M–M, S–M, M–S) are essential; cross-modal attention and motion correction each yield $\sim2$ point mAP improvement on THUMOS14 [2305.19624].

## 7. Empirical Results and Evaluation

MSAT approaches have demonstrated superior results in both dense action detection and long-form video understanding benchmarks:

| Dataset         | Metric    | MSAT (avg mAP/acc) | Prior SOTA   |
|-----------------|-----------|--------------------|--------------|
| THUMOS14        | frame-mAP | 68.5               | 66.8         |
| ActivityNet v1.3| frame-mAP | 39.0               | 36.6         |
| Instructional   | frame-mAP | 68.1               | ~45.2        |
| MMX-Trailer-20  | mAP       | 0.751*             | 0.601–0.640  |
| LVU (Scene Rec) | accuracy  | 58.33              | 56.9         |

*STAN-Large, see [2208.01753].

Ablation studies show:
- Single-type attention (S–S or M–M) under-performs; cross-modal attention alone is intermediate; full MMA yields optimal results.
- Removal of motion correction or cross-modal attention produces measurable degradation in mAP [2305.19624].

## 8. Distinguishing Characteristics and Theoretical Significance

MSAT models pioneer explicit, parameterized attention pathways for both intra- and cross-modality reasoning, with streamlined fusion of attention sub-maps via convolution. The use of motion-distortion correction enables robust application in untrimmed, real-world video. By comparison, related architectures such as the two-stream Spatio-Temporal Attention Network (STAN) implement parallel, stream-specific transformer stacks with optional cross-stream attention and late fusion [2208.01753].

A plausible implication is that modality decomposition, explicit cross-modal attention, and practical motion correction constitute essential design choices for state-of-the-art action detection and long-form understanding. These principles yield robust, scalable temporal localization under challenging visual conditions.

---

References:  
[2305.19624] "A Multi-Modal Transformer Network for Action Detection"  
[2208.01753] "Two-Stream Transformer Architecture for Long Video Understanding"

Source: https://www.emergentmind.com/topics/multi-stream-action-transformer-msat