Stacked Temporal Attention Modules

• Stacked temporal attention modules are architectural designs that repeatedly incorporate temporal attention blocks to progressively refine inter-frame dynamics.
• They are deployed in diverse applications such as video diffusion, Video-LLMs, and neuroimaging, using variations like global, motion-adaptive, and consolidation-based attention.
• Empirical studies demonstrate that optimal stacking, component ordering, and initialization strategies can significantly enhance temporal coherence and downstream performance.

Searching arXiv for recent and related work on stacked temporal attention modules. Stacked temporal attention modules are temporal modeling structures in which temporal operators are inserted repeatedly rather than used as a single post-hoc aggregation step. In recent diffusion-based video generation, the term denotes injecting a temporal attention block into every transformer block of a UNet and cascading their roles across depth: global temporal attention in down-sampling and middle blocks, and motion-adaptive temporal attention in up-sampling blocks (Hong et al., 18 Mar 2026). Closely related formulations place temporal attention directly inside vision encoders for Video-LLMs, refine clip saliency with multiple global-attention layers, or combine temporal self-attention with temporal consolidation in neuroimaging, so the term identifies a family of architectural patterns rather than one fixed module (Rasekh et al., 29 Oct 2025, Yang et al., 2021, Jha et al., 2021).

1. Conceptual scope and terminological range

Across the literature, “stacked” refers to repetition of temporal modules along different axes of computation. In video diffusion, temporal blocks are stacked across UNet depth; in Video-LLMs, temporal sublayers are inserted throughout the vision encoder; in egocentric action recognition, multiple global-attention layers iteratively refine clip importance; and in rs-fMRI classification, a stack of temporal self-attention layers is followed by a stack of temporal consolidation blocks (Hong et al., 18 Mar 2026, Rasekh et al., 29 Oct 2025, Yang et al., 2021, Jha et al., 2021).

Work	Where the temporal module is stacked	Defining pattern
(Hong et al., 18 Mar 2026)	All 16 UNet transformer blocks	Global down/mid, motion-adaptive up
(Rasekh et al., 29 Oct 2025)	Many or all vision encoder blocks	Spatial self-attention, then per-patch temporal self-attention
(Yang et al., 2021)	Multiple global-attention layers	Global query refined across layers over fixed clip features
(Jha et al., 2021)	5 encoder layers plus 3 TCM blocks	Temporal self-attention followed by temporal consolidation
(Yuan et al., 2024)	Once per time step	Causal TAM using only the previous hidden state
(Garnot et al., 2020)	Not stacked	Single temporal module with parallel compact heads

This range matters because “stacked temporal attention modules” does not imply a single canonical implementation. Some systems stack multi-head temporal self-attention blocks, some stack global-attention refinement layers, and some use only one temporal attention block per time step or one temporal module for the whole network. The phrase therefore denotes an architectural strategy—repeated temporal interaction—more than a uniform operator class (Yuan et al., 2024, Garnot et al., 2020).

2. Architectural organization and placement strategies

A current and explicit formulation appears in “Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion” (Hong et al., 18 Mar 2026). The method augments a frozen Stable Diffusion latent diffusion pipeline with lightweight temporal attention modules. Input videos of 8 frames at $256\times256$ are encoded by a frozen VAE encoder into latents $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$, with $c=4$ and $h=w=32$. Temporal attention is the only trainable addition and is applied after each spatial transformer within the UNet, inside a SpatialTemporalWrapper that reshapes features and computes attention along time. Temporal attention blocks are injected into all 16 transformer blocks of the UNet: 6 blocks in the down-sampling path use global temporal attention, 1 middle block uses global temporal attention, and 9 blocks in the up-sampling path use motion-adaptive temporal attention. A single temporal attention block is added to each UNet transformer block, so temporal modules are stacked across depth as $6+1+9=16$. Each temporal block uses 4 heads, with head dimension 64 and FFN inner dimension 512; across all blocks, this adds 25.8M trainable parameters, which is 2.9% of the 860M base UNet.

In STAVEQ2, stacked temporal attention is organized directly inside the vision encoder of a Video-LLM (Rasekh et al., 29 Oct 2025). Each transformer block applies spatial self-attention independently on each frame, temporal self-attention independently on each patch index across the $T$ frames, and then an MLP, with residual and layer normalization at each stage. “Stacked” temporal attention means that temporal modules are inserted repeatedly after spatial attention inside many or all vision transformer blocks. The reported ablations emphasize ordering and depth: temporal attention after spatial attention is preferred, and broader insertion across the encoder is more effective than sparse placement.

STAM uses a different stack topology (Yang et al., 2021). Instead of updating all local states layer by layer, it computes a global feature $g^0$ that summarizes the entire sequence, then applies a stack of global-attention layers in which the global feature $g^{l-1}$ is the query while keys and values remain tied to the fixed local clip features $X$. This is explicitly contrasted with naive stacking, where both local features and attention states are repeatedly transformed. The stack is therefore a progressive refinement of globally informed clip weighting rather than a conventional Transformer over clip tokens.

MHATC combines a stack of temporal attention layers with a stack of temporal consolidation blocks (Jha et al., 2021). The Multi-Head Attention Encoder contains $L=5$ encoder layers with $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$0 heads, residual connections, and a position-wise feed-forward network, while the Temporal Consolidation Module is itself stacked in 3 blocks, progressively reducing channel dimensionality from $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$1 while keeping temporal length constant. This separates temporal dependency modeling from region-wise consolidation.

3. Core mathematical forms

A common formulation is temporal self-attention over per-position temporal sequences. In the diffusion architecture of (Hong et al., 18 Mar 2026), per-frame features $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$2 are reshaped to

$$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$3

with $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$4, so attention is computed independently at each spatial position across the $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$5 frames. The global temporal block adds learnable temporal positional encodings $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$6 and computes

$$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$7

For the adaptive block, motion is quantified by a per-video scalar $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$8, and a distance-biased term is injected into attention logits: $$z_0 \in \mathbb{R}^{(B\cdot T)\times c\times h\times w}$$9 The output is then modulated by motion-aware residual gating,

$c=4$0

This realizes locality as a soft distance-based bias rather than a hard window or mask.

STAVEQ2 uses a decoupled spatiotemporal block (Rasekh et al., 29 Oct 2025). After spatial self-attention produces $c=4$1, temporal attention is applied per patch index $c=4$2 over

$c=4$3

Queries, keys, and values are projected from $c=4$4, 1D rotary position embeddings are applied along time to $c=4$5 and $c=4$6, and attention is

$c=4$7

The block then applies an MLP with residual: $c=4$8

STAM replaces token-token self-attention with a single global query over clips (Yang et al., 2021). If $c=4$9, then at layer $h=w=32$0,

$h=w=32$1

$h=w=32$2

Only the global query is refined across layers; keys and values stay anchored to the backbone clip features.

A fourth pattern is causal temporal fusion. In the cross-view localization TAM of (Yuan et al., 2024), the current fused features $h=w=32$3 query only the previous hidden state $h=w=32$4: $h=w=32$5

$h=w=32$6

This design is explicitly uni-directional and causal and does not attend across all past frames or future frames.

4. Optimization, initialization, and computational regimes

In the diffusion setting, stacked temporal attention is trained under the standard diffusion denoising loss

$h=w=32$7

with correlated Gaussian noise generated by the AR(1) process

$h=w=32$8

Only temporal attention parameters are updated; the UNet, VAE, and text encoder are frozen. The paper reports that the standard denoising objective alone provides sufficient implicit temporal regularization and that $h=w=32$9 performs better for motion diversity than adding an auxiliary temporal consistency loss. Training uses 100K WebVid videos, 8 frames per video with stride 2, resized to $6+1+9=16$0, AdamW with $6+1+9=16$1, cosine schedule with 500 warmup, gradient norm clipping 1.0, 30 epochs, batch size 55 per GPU on A100, FP16, and gradient checkpointing (Hong et al., 18 Mar 2026).

Initialization is a recurrent design concern in stacked temporal modules. In (Hong et al., 18 Mar 2026), output projections and the final FFN layer are zero-initialized so that temporal blocks start as exact identity mappings, avoiding disturbance of the frozen UNet. STAVEQ2 adopts an analogous strategy: stage 1 zero-initializes the output projection $6+1+9=16$2 of temporal attention so the early network behaves like the pre-trained spatial-only encoder, trains only the temporal blocks and their layer norms, and then in stage 2 inserts LoRA adapters into the linear layers of the vision encoder and into the LLM for joint alignment (Rasekh et al., 29 Oct 2025).

Compute scaling differs sharply across variants. In (Hong et al., 18 Mar 2026), temporal attention operates across the short time dimension $6+1+9=16$3 at each spatial location, so per-block temporal computation is $6+1+9=16$4. In STAVEQ2, the decoupled spatiotemporal block has attention complexity $6+1+9=16$5, which is explicitly contrasted with joint space-time attention $6+1+9=16$6 (Rasekh et al., 29 Oct 2025). This computational distinction is central to why stacked temporal attention is often inserted as a decoupled sublayer rather than replacing the full backbone attention pattern.

A boundary of the term appears in systems that are stackable and temporal but are not Transformer-style attention. TAU states that it “does not construct query/key/value projections nor use the Transformer-style softmax attention,” instead decomposing the temporal module into intra-frame statical attention and inter-frame dynamical attention (Tan et al., 2022). STE likewise is “implemented with temporal convolutions applied in sliding windows across frames, not with multi-head attention,” even though it is explicitly stackable and used to grow temporal receptive fields and token compression ratios (Li et al., 28 Jan 2025). These cases show that the research area often groups several temporally stacked operators under one conceptual umbrella.

5. Empirical behavior and ablation patterns

Empirical results show that stacking is beneficial when module placement, ordering, and update rules are matched to the task. In lightweight video diffusion, the stacked temporal modules are the primary drivers of inter-frame coherence: on 400 WebVid validation captions, “Ours (SD 1.5)” reports CLIP 0.302, FC 0.980, MM 0.051, FID 89.96, and FVD 85.99; “Ours (SD 2.1)” reports CLIP 0.303, FC 0.983, MM 0.040, FID 92.09, and FVD 92.49; while “SD 1.5 with correlated noise but no temporal modules” drops to CLIP 0.267, FC 0.777, MM 0.326, FID 180.48, and FVD 211.02 (Hong et al., 18 Mar 2026). The same paper reports that adding explicit temporal loss with $6+1+9=16$7 raises FC to 0.990 but lowers MM to 0.025 and worsens FID/FVD to 95.82/97.21, whereas $6+1+9=16$8 gives FC 0.983, MM 0.040, and FID/FVD 92.09/92.49. Noise correlation also acts as an inference-time control: for a model trained with $6+1+9=16$9, inference at $T$0 yields MM 0.158 and FC 0.929, while $T$1 yields MM 0.008 and FC 0.999.

In Video-LLMs, stacked temporal insertion improves temporal reasoning rather than only low-level coherence. STAVEQ2 reports that the best ordering is spatial first, temporal second, with temporal head scaling 0.25, reaching 76.04% versus a 73.14% baseline on SSv2-T10; adding STA in all 32 blocks yields 76.04%, while 16 STA blocks uniformly give 74.73% and 16 early blocks give 74.97%. The paper also reports that InternVideo2-Chat 8B improves from 84.17% to 95.18% on SSv2-T10 with STA, and that improvements on VITATECS, MVBench, and Video-MME reach up to +5.5% depending on benchmark and task (Rasekh et al., 29 Oct 2025).

By contrast, some studies show that naive stacking is ineffective. STAM explicitly reports that on EGTEA split1 with I3D and $T$2 clips, vanilla stacking without a global feature gives 65.6 / 65.5 / 65.5 for 1 / 2 / 3 layers, whereas self-attention initialization gives 66.2 / 66.5 / 66.9 and a light-weight CNN initialization reaches 67.1% with 3 layers (Yang et al., 2021). The point is not merely that more layers help, but that the query design must encode global context of the whole video.

MHATC reports the same pattern of non-monotonic depth utility in clinical time series (Jha et al., 2021). With 2 heads, the number of encoder layers improves performance from 73.56% at $T$3, to 75.48% at $T$4, 76.62% at $T$5, 77.00% at $T$6, and 77.40% at $T$7, followed by a slight degradation to 76.80% at $T$8. The best reported mean results under 10-fold cross-validation on ABIDE I are Accuracy 77.4%, Sensitivity 78.6%, and Specificity 74.0%.

6. Limitations, boundary cases, and recurrent misconceptions

A persistent misconception is that any temporal module repeated in depth is a stacked temporal attention module in the strict multi-head self-attention sense. The surveyed literature does not support that equivalence. The cross-view localization TAM is used once per time step and “does not report stacking multiple TAM layers at a single time step,” while L-TAE is explicitly “a single temporal attention module with $T$9 parallel compact heads followed by an MLP” and “does not stack multiple temporal attention blocks” (Yuan et al., 2024, Garnot et al., 2020). Conversely, TAU and STE are described as stackable temporal modules but not as Q/K/V attention blocks (Tan et al., 2022, Li et al., 28 Jan 2025).

Another misconception is that deeper stacks are automatically superior. The evidence is mixed. STAVEQ2 finds that adding STA in all 32 blocks is best, but STAM shows that naive stacking does not help, and MHATC peaks at $g^0$0 before degrading (Rasekh et al., 29 Oct 2025, Yang et al., 2021, Jha et al., 2021). This suggests that placement, ordering, query construction, and residual design are more consequential than depth alone.

Common limitations are also consistent across domains. In (Hong et al., 18 Mar 2026), experiments are limited to $g^0$1 resolution and 8 frames; a single scalar $g^0$2 and short $g^0$3 limit modeling of complex, long-range dynamics; and temporal attention scales as $g^0$4 per position. STAVEQ2 states that longer videos are supported, but complete scaling studies and broader domain shifts remain future work (Rasekh et al., 29 Oct 2025). A plausible implication is that stacked temporal attention modules are most effective when temporal scale is aligned with the representational granularity of the backbone: local per-patch temporal attention in vision encoders, globally anchored clip reweighting in egocentric recognition, and motion-conditioned temporal locality in latent video diffusion.

In this sense, stacked temporal attention modules are best understood not as one module type but as a design principle: temporal communication is introduced repeatedly, at multiple depths or stages, so that temporal dependencies are refined where the backbone already forms semantically meaningful intermediate representations.