Multi-Stream Action Transformer (MSAT)

• Multi-Stream Action Transformer (MSAT) is a transformer-based architecture that integrates spatial and motion streams using explicit, modality-aware attention for fine-grained video action detection.
• It employs modular transformer stacks with multi-modal attention that capture both intra- and cross-stream interactions, achieving notable improvements in metrics like mAP on benchmarks such as THUMOS14.
• The architecture incorporates motion distortion correction and fusion via convolution, ensuring robust per-frame action localization even in untrimmed, long-form video scenarios.

A Multi-Stream Action Transformer (MSAT) is a transformer-based architecture designed for fine-grained action understanding in videos, characterized by explicit separation and interaction of multiple feature streams—typically spatial and motion modalities—using modality-aware attention. MSAT models leverage two (or more) streams of raw video data, advanced multi-modal attention mechanisms, and modular transformer stacks to capture both intra- and inter-modality dependencies, enabling robust per-frame action localization and classification, particularly in untrimmed or long-form video input (Korban et al., 2023, Fish et al., 2022).

1. Input Modalities and Feature Representation

MSAT architectures operate on multiple, semantically distinct input channels. Commonly, this includes a spatial stream derived from RGB frames and a motion stream based on optical flow or spatio-temporal encodings. Each input stream is independently embedded via a modality-specific backbone:

• Spatial stream: $I^S = \{I^s_t\}_{t=1}^T$, processed by a $\rho_{\rm spatial}$ backbone (e.g., I3D or ResNet-18), generating $X^S \in \mathbb{R}^{T \times Z}$.
• Motion stream: $I^{M'} = \{I^{m'}_t\}_{t=1}^T$ (possibly distorted optical flow) is subjected to correction (see Section 4), then processed by $\rho_{\rm motion}$ yielding $X^M \in \mathbb{R}^{T \times Z}$ with $Z=1024$ (Korban et al., 2023).

Alternative instantiations (e.g., STAN (Fish et al., 2022)) may derive temporal stream tokens from low-resolution RGB clips using 3D CNNs, while spatial tokens are extracted from center frames using 2D CNNs. Positional embeddings and learned CLS tokens are appended to each sequence.

2. Transformer Backbone and Layer Design

The fused MSAT representation is processed by multi-layer transformer stacks, designed to aggregate information across time and modalities. Each of the $L$ layers consists of:

• LayerNorm $\rightarrow$ Multi-Modal Attention (MMA) $\rightarrow$ residual addition
• LayerNorm $\rho_{\rm spatial}$0 two-layer MLP (hidden size $\rho_{\rm spatial}$1) $\rho_{\rm spatial}$2 residual addition

In the canonical MSAT (as in (Korban et al., 2023)), the $\rho_{\rm spatial}$3-th layer operates as: $\rho_{\rm spatial}$4 with the first layer input $\rho_{\rm spatial}$5 given by concatenation of $\rho_{\rm spatial}$6 and $\rho_{\rm spatial}$7. The backbone typically uses $\rho_{\rm spatial}$8 layers and $\rho_{\rm spatial}$9 multi-modal heads per layer.

Alternative two-stream transformers (Fish et al., 2022) instantiate standard multi-head self-attention for each stream individually, with per-stream $X^S \in \mathbb{R}^{T \times Z}$0-layer stacks. Fusion occurs via extraction and combination (typically weighted) of the modality-wise CLS tokens after the transformer encoders.

3. Multi-Modal Attention and Stream Interaction

The MSAT's distinguishing feature is its parameterized multi-modal attention. For each layer and head, queries, keys, and values are projected independently for spatial and motion streams: $X^S \in \mathbb{R}^{T \times Z}$1 Attention sub-maps are computed as: $X^S \in \mathbb{R}^{T \times Z}$2 where $X^S \in \mathbb{R}^{T \times Z}$3. The four attention maps are concatenated across a modality dimension and fused by a $X^S \in \mathbb{R}^{T \times Z}$4 convolution: $X^S \in \mathbb{R}^{T \times Z}$5 This scheme supports both intra-stream (S–S, M–M) and cross-stream (S–M, M–S) interactions, a critical factor in performance improvements on complex video action datasets (Korban et al., 2023). In contrast, some designs (Fish et al., 2022) exclusively use independent stream-wise self-attention but can optionally incorporate cross-stream attention at fusion or intermediate layers.

4. Motion Distortion Correction

Camera-induced motion distortion in optical flow degrades motion features in unconstrained video. MSAT addresses this by a three-step correction procedure (Korban et al., 2023):

1. Foreground–background segmentation: Person detector separates background motion vectors $X^S \in \mathbb{R}^{T \times Z}$6.
2. GMM Fitting: A Gaussian Mixture Model ($X^S \in \mathbb{R}^{T \times Z}$7 components) is fitted to the background flow, updated iteratively via the EM algorithm:

   E-step:

   $X^S \in \mathbb{R}^{T \times Z}$8

   M-step:

   $X^S \in \mathbb{R}^{T \times Z}$9

3. Motion restoration: Each vector $I^{M'} = \{I^{m'}_t\}_{t=1}^T$0 is corrected by subtracting the mean of its assigned Gaussian component, $I^{M'} = \{I^{m'}_t\}_{t=1}^T$1, yielding the corrected flow $I^{M'} = \{I^{m'}_t\}_{t=1}^T$2 and vector field $I^{M'} = \{I^{m'}_t\}_{t=1}^T$3.

This correction achieves substantial robustness compared to using raw flow or more elaborate pose/depth/flow fusion, as evidenced by the associated ablation studies.

5. Output Head, Fusion, and Objective

After $I^{M'} = \{I^{m'}_t\}_{t=1}^T$4 transformer layers, a unified joint representation $I^{M'} = \{I^{m'}_t\}_{t=1}^T$5 is produced. For dense, frame-wise action detection:

• A 1D convolution (kernel width=3) projects $I^{M'} = \{I^{m'}_t\}_{t=1}^T$6 to per-frame logits, yielding $I^{M'} = \{I^{m'}_t\}_{t=1}^T$7
• Post-processing merges predictions above threshold into action segments (Korban et al., 2023)

Alternative two-stream designs (e.g., (Fish et al., 2022)) perform late fusion of stream-level representations via layernorm/MLPs/weighted sums, followed by a multi-layer classification head.

The loss combines per-frame cross-entropy and a temporal IoU (tIoU) regularizer: $I^{M'} = \{I^{m'}_t\}_{t=1}^T$8 For long-form classification benchmarks, the loss reduces to binary cross-entropy over segment or video-level predictions (Fish et al., 2022).

6. Training Regimes and Implementation

• Backbone: I3D or R(2+1)D (motion), ResNet-18 (spatial), pre-trained on Kinetics/ImageNet as appropriate.
• Optimization: Adam or AdamW, learning rates $I^{M'} = \{I^{m'}_t\}_{t=1}^T$9 to $\rho_{\rm motion}$0; weight decay $\rho_{\rm motion}$1 to $\rho_{\rm motion}$2.
• Sequence length: Up to $\rho_{\rm motion}$3 frames for detection tasks; scene-level tokens for long-form classification.
• Regularization: Dropout, data augmentation, and LayerNorm; I3D backbone regularization carried over.
• Parameter count: STAN-Small $\rho_{\rm motion}$4M, STAN-Large $\rho_{\rm motion}$5M (Fish et al., 2022).
• Ablations show all four MMA types (S–S, M–M, S–M, M–S) are essential; cross-modal attention and motion correction each yield $\rho_{\rm motion}$6 point mAP improvement on THUMOS14 (Korban et al., 2023).

7. Empirical Results and Evaluation

MSAT approaches have demonstrated superior results in both dense action detection and long-form video understanding benchmarks:

Dataset	Metric	MSAT (avg mAP/acc)	Prior SOTA
THUMOS14	frame-mAP	68.5	66.8
ActivityNet v1.3	frame-mAP	39.0	36.6
Instructional	frame-mAP	68.1	~45.2
MMX-Trailer-20	mAP	0.751*	0.601–0.640
LVU (Scene Rec)	accuracy	58.33	56.9

*STAN-Large, see (Fish et al., 2022).

Ablation studies show:

• Single-type attention (S–S or M–M) under-performs; cross-modal attention alone is intermediate; full MMA yields optimal results.
• Removal of motion correction or cross-modal attention produces measurable degradation in mAP (Korban et al., 2023).

8. Distinguishing Characteristics and Theoretical Significance

MSAT models pioneer explicit, parameterized attention pathways for both intra- and cross-modality reasoning, with streamlined fusion of attention sub-maps via convolution. The use of motion-distortion correction enables robust application in untrimmed, real-world video. By comparison, related architectures such as the two-stream Spatio-Temporal Attention Network (STAN) implement parallel, stream-specific transformer stacks with optional cross-stream attention and late fusion (Fish et al., 2022).

A plausible implication is that modality decomposition, explicit cross-modal attention, and practical motion correction constitute essential design choices for state-of-the-art action detection and long-form understanding. These principles yield robust, scalable temporal localization under challenging visual conditions.

---

References: (Korban et al., 2023) "A Multi-Modal Transformer Network for Action Detection" (Fish et al., 2022) "Two-Stream Transformer Architecture for Long Video Understanding"