Tubelet Transformers for Video Analysis

• Tubelet Transformers are deep learning architectures that form spatio-temporal tubelets representing tracked instances across video frames, enabling coherent video processing.
• They utilize adaptive token-agglomeration and localized attention mechanisms to aggregate and link spatial patches effectively for accurate action and object localization.
• Empirical results show up to a 16.14% mAP gain and 4× faster inference, streamlining end-to-end training and boosting video understanding performance.

Tubelet Transformers are a class of deep learning architectures designed for coherent spatio-temporal processing in videos, built on the Transformer framework. They are characterized by the explicit formation, manipulation, and prediction of tubelets—spatio-temporal representations corresponding to particular instances (actors, objects, or human-object pairs) tracked through a video sequence. Tubelet Transformers have been applied to tasks such as video-based human-object interaction detection (Tu et al., 2022), spatio-temporal action localization (Gritsenko et al., 2023), and action tube detection (Zhao et al., 2021), and consistently demonstrate improvements in both accuracy and efficiency over prior multi-stage and memory-bank-based methods.

1. Foundational Concepts and Definitions

Tubelet Transformers process a video $x \in \mathbb{R}^{T \times H \times W \times 3}$ by encoding patch-level or regional features, which are then abstracted and linked into tubelets—sequences of embeddings representing single semantic entities (such as humans or objects) across frames. The core idea is to incorporate spatio-temporal structure directly into the backbone and attention mechanisms, producing compact yet expressive tubelet tokens: for instance, $z_{tube}^i \in \mathbb{R}^D$ encodes the $i$th instance tracked across $T$ frames (Tu et al., 2022). Tubelet queries—used for detection or localization—are often implemented as learnable parameters: $Q_i = \{ q_{i,1}, ..., q_{i,T_{\rm out}} \}$, where $q_{i,t} \in \mathbb{R}^{C'}$ (Zhao et al., 2021).

2. Tubelet Construction: Agglomeration and Linking Mechanisms

Tubelet construction occurs via a series of spatial and temporal abstraction steps. In architectures such as TUTOR (Tu et al., 2022), patch tokens extracted from a convolutional backbone are abstracted via token-agglomeration layers over irregular spatial windows, using a learned convolution to generate offsets $\Delta p_n$ for adaptive grouping. The agglomeration can be mathematically described as:

$$z_{(t,s+1)}^i = W_{agg}\left[\bigoplus_{n=1}^4 z_{(t,s)}(p_n + \Delta p_n)\right],$$

where $W_{agg}$ projects concatenated token features, progressively reducing spatial resolution while doubling channel dimension over several stages.

Temporal linking follows, employing a similarity-based assignment (e.g., Gumbel-Softmax smoothed dot-product and non-max-suppression) to associate tokens from different frames belonging to the same entity, producing tubelet tokens:

$$z_{tube}^i = z_q^i + W_o \frac{ \sum_{t\neq q} \hat A_t^{(i, \phi(i,t))} W_v z_t^{\phi(i, t)} }{ \sum_{t\neq q} \hat A_t^{(i, \phi(i, t))} }$$

where $\phi(i, t)$ gives the matched token in frame $t$ for query $i$ (Tu et al., 2022).

Other models initialize tubelet queries as sequences of spatio-temporal anchors (e.g., $q_{t,s} = q^t_t + q^s_s$ in STAR (Gritsenko et al., 2023)), and update these through layers of factorized self-attention and cross-attention, encouraging temporal coherence and spatial consistency without explicit assignments.

3. Attention Schemes and Transformer Architectures

Tubelet Transformers exploit factorized and localized attention to model inter-instance and temporal dependencies efficiently. For example, TUTOR interleaves S-blocks—Transformer blocks with restricted self-attention over small irregular windows—between its agglomeration stages:

• Irregular Window Partition (IWP) computes adaptive windows via learned 3x3 convolutions.
• Window-based Multi-head Self-Attention (IW-MSA) operates only within the window, drastically reducing computation ($O(S_w^2 H_s W_s T C_s)$ vs $O(H_s^2 W_s^2 T^2 C_s)$ for global attention).

Tubelet-attention modules in TubeR (Zhao et al., 2021) alternate spatial self-attention (across queries at a time step) and temporal self-attention (across time steps for each query). This yields updated tubelet query features, enabling both intra-frame and cross-time modeling.

Decoder layers in end-to-end models like STAR (Gritsenko et al., 2023) factorize attention: first attending within spatial or temporal dimensions, then performing cross-attention restricted to corresponding encoder tokens, thus encoding strong spatio-temporal inductive biases.

4. Prediction Heads and Losses

Tubelet Transformer decoders produce predictions for each tubelet across frames:

• Bounding box regression: outputs $(x, y, w, h)$ per frame for each tubelet (e.g., $$y_{coor} \in \mathbb{R}^{N \times T_{\rm out} \times 4}$$) (Zhao et al., 2021).
• Action/object classification: outputs action or object class scores (linear+sigmoid or softmax) (Tu et al., 2022).
• Action switch regression: predicts action presence per time step via a sigmoid head (Zhao et al., 2021).

Supervision is implemented using weighted sums of binary cross-entropy, multi-class cross-entropy, L1 box regression loss, and IoU or GIoU loss. Assignment between predictions and ground truth employs the Hungarian algorithm, with tubelet-level matching enforcing temporal consistency in localization (Gritsenko et al., 2023).

5. Architectural Variants and Practical Implementations

Table: Tubelet Transformer Variants

Model	Backbone	Tubelet Construction	Attention Scheme
TUTOR (Tu et al., 2022)	ResNet+FPN	Agglomeration + Gumbel Linking	S-block (IW-MSA)
TubeR (Zhao et al., 2021)	CSN-152, 3D ConvNet	Learnable tubelet queries	Spatial+Temporal tubelet-attn
STAR (Gritsenko et al., 2023)	ViViT Factorized	Anchor queries + sum	Factorized self/cross-attn

Each variant builds tubelet representations via selective spatial and temporal abstraction: TUTOR emphasizes explicit instance linking, TubeR uses flexible tubelet-queries for direct prediction, and STAR implements fully end-to-end tubelet parsing with factorized attention and flexible supervision schemas.

6. Performance, Evaluation, and Impact

Tubelet Transformers outperform prior state-of-the-art models on benchmarks:

• TUTOR (Tu et al., 2022) achieves a relative mAP gain of 16.14% on VidHOI (from ~22.0% to ~26.84% mAP) and a 4× inference speedup (2.0 FPS vs. 0.5 FPS on 8 RTX-2080Ti, FLOPs reduced from 0.81 TFLOPs to 0.25 TFLOPs, parameters from 243M to 82M).
• STAR (Gritsenko et al., 2023) attains 41.7 mAP on AVA keyframe (previous best: ~33 mAP), 44.6 mAP on AVA-Kinetics, 90.3% frame AP and 71.8% video [email protected] on UCF101-24 (+11.6 points over TubeR), and 92.6% video [email protected] on JHMDB.
• TubeR (Zhao et al., 2021) demonstrates strong results across AVA, UCF101-24, and JHMDB51-21.

These architectures enable elimination of external proposal generators, memory banks, and post-processing such as NMS, resulting in streamlined training and inference pipelines. Flexible supervision accommodates sparse or dense tubelet annotations, with ablations showing minimal accuracy drop under sparse labeling (Gritsenko et al., 2023).

7. Advantages, Limitations, and Future Directions

Tubelet Transformers enable end-to-end modeling of spatio-temporal interactions in video, facilitating compact and expressive instance tracking and action localization. Key advantages include temporal coherence, unified handling of dense and sparse annotation regimes, and substantial gains in accuracy and efficiency on multiple benchmarks (Tu et al., 2022, Gritsenko et al., 2023, Zhao et al., 2021).

Limitations include computational and memory demands of Transformer backbones at high resolution or long clip length, reliance on strong pretraining, and necessity for auxiliary linking mechanisms in very long untrimmed videos. Tubelet-level matching trades off frame-level accuracy for temporal consistency, with optimal balance being dataset-dependent.

A plausible implication is continued architectural refinement to minimize computational cost, adapt tubelet matching for open-domain streaming, and integrate multimodal instance conditioning. The Tubelet Transformer approach establishes a principled framework for any video understanding task requiring spatio-temporal localization, offering a unified paradigm for region tracking, action localization, and human-object interaction detection.