Query-based Cross-Frame Propagation

• Query-based cross-frame propagation is a method that reuses token-level computations from prior video frames, reducing redundancy in heavy transformer operations.
• It employs a lightweight Decision Layer for hard token gating and a Restoration Layer for calibrating reused outputs, ensuring minimal accuracy loss.
• Empirical evaluations demonstrate significant throughput improvements, achieving up to 2.64× speedup in GPU-based video-language tasks.

Query-based cross-frame propagation is a class of mechanisms enabling scalable and efficient representation sharing across consecutive frames in video-LLMs and video transformers. By exploiting temporal redundancy in video, these approaches selectively reuse internal token-level computations—often called "queries"—learned for prior frames, reducing redundant execution of heavy operations such as attention and feed-forward networks. State-of-the-art designs, such as the “Déjà Vu” framework and its core ReuseViT architecture, introduce dynamic gating and restoration modules at the transformer layer level to balance computational reuse with accuracy, achieving substantial empirical speedup without significant accuracy loss on GPU inference workloads (Hwang et al., 17 Jun 2025).

1. ReuseViT: Layerwise Query Gating and Restoration

At the heart of query-based cross-frame propagation in “Déjà Vu” is ReuseViT—a Vision Transformer (ViT) encoder, e.g., CLIP-ViT-L/14, augmented per layer with two lightweight modules:

• The Decision Layer, a two-layer MLP, executes hard per-token gating on whether each token’s query, key, value (QKV) and feed-forward network (FFN) output are recomputed or fetched (reused) from a cached reference frame.
• The Restoration Layer, also a small MLP, performs per-token calibration to correct reused outputs, compensating for minor frame variations.

Within each transformer layer $\ell$ for frame $t$, the input tokens $T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$ (with $N$ patches plus a CLS token) are processed as follows. After standard LayerNorm and QKV projections, the Decision Layer computes a hard reuse mask $\mathcal{M}_{t,\ell,i}\in\{0,1\}$ for every token $i$, based on a feature vector incorporating:

• Maximum cosine similarity to the same spatial position in past/future frames,
• Class-token attention,
• One-hot frame type,
• Codec metadata (e.g., motion residuals).

For $\mathcal{M}_{t,\ell,i}=0$ (“recompute”), QKV and FFN are executed as usual. For $\mathcal{M}_{t,\ell,i}=1$ (“reuse”), the output from a cached reference frame $t'$ is adopted and then passed through the Restoration Layer, which receives the token delta $$\Delta_{t,\ell,i}=T^{cur}_{t,\ell,i}-T^{ref}_{t',\ell,i}$$, returns a correction $t$0, and outputs $t$1.

Self-attention is always recomputed, maintaining modeling power even when subsets of tokens are reused.

The backbone is trained end-to-end with a composite objective balancing embedding similarity and reuse-rate regularization, where a grouped-frame setting explicitly models error accumulation across $t$26-frame segments.

2. Mathematical Formulation of Query Propagation

Let $t$3 be a token embedding in layer $t$4 of frame $t$5. The Decision Layer computes a gating decision $t$6, setting a hard reuse mask

$t$7

Soft gating with Gumbel-Softmax is used during training: $t$8 For reused tokens, calibration occurs: $t$9

$T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$0

$T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$1

Final token values are chosen as

$T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$2

The main training losses are

$T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$3

$T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$4

$T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$5

where $T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$6 is the target reuse rate, and $T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$7 the balance factor.

3. Computational Scheduling and GPU Realization

To turn theoretical FLOP savings into realized throughput on GPU, multiple compaction and scheduling strategies are essential.

• Layer-wise scheduling: All frames in a segment are processed for layer $T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$8 before moving to $T_{t,\ell-1} \in \mathbb{R}^{(N+1)\times D}$9, enabling immediate cache eviction of unneeded activations and improving memory utilization.
• Sparse computation compaction: All tokens across all frames marked for “recompute” are gathered into a dense matrix batch, enabling large, efficient QKV+FFN GPU kernel calls, avoiding overhead from tiny, scattered operations.
• Cache memory compaction: Cached activations for frames no longer referenced are freed, reducing peak GPU memory.

These measures increase the effective batch size (e.g., from 67 to 183 frames on an RTX 3090) and convert a theoretical 50–70% FLOP reduction into measured end-to-end speedups.

4. Inference Pipeline and Pseudocode

The following pseudocode outlines K-frame propagation with L layers:

$N$0

Key features are per-token gating, restoration, compaction, and immediate cache release.

5. Empirical Results and Ablations

Main Throughput and Accuracy Results

Task	Model	Dataset	Baseline Throughput	Déjà Vu (≤2% error)
Video Retrieval	CLIP4Clip	MSR-VTT	1.0×	1.81×
Video QA	FrozenBiLM	How2QA	1.0×	2.64×
Video Grounding	TempCLIP	NExT-GQA	1.0×	2.54×

• Déjà Vu achieves up to 2.64× throughput improvement at ≤2% accuracy drop (Hwang et al., 17 Jun 2025).
• At 61% token reuse rate, decision+restoration overhead is just +4% FLOPs; adaptive policies sustain a 72% reuse rate at the same accuracy, offsetting restoration cost.

Component Ablations

Configuration	Speedup
Hard gating only	1.25×
+ Sparse compaction	1.45×
+ Memory compaction	1.62×

• Using three reference segments yields the best trade-off; four reduces accuracy (over-reuse error).
• Static reuse schemes (e.g., Eventful) degrade over time due to drift in cosine similarity; Déjà Vu adapts reuse as content changes.

6. Architectural and Practical Insights

• The Decision Layer produces substantial gains over fixed reuse strategies.
• Grouped-frame training increases robustness to error accumulation across longer temporal segments.
• Immediate per-layer cache eviction is crucial for supporting large-batch GPU inference.
• The Restoration Layer, though a small throughput hit, is necessary for recapturing accuracy in practical deployment.

A plausible implication is that token-level dynamic reuse, mediated by fast MLP decision and calibration modules, is an optimal balance between per-frame redundancies and online variance in real-world video analytics settings.

7. Domain Scope and Impact for Video-LLMs

Query-based cross-frame propagation, as instantiated in ReuseViT and the broader Déjà Vu framework, provides a scalable method for leveraging temporal redundancy in video-language tasks. This unlocks significant practical speedup for real-world systems—enabling, for example, a CLIP-based video understanding engine to process multi-million-frame data at a fraction of the previous computational and memory cost. The ability to dynamically modulate reuse rates according to scene content, together with fine-grained cache and computation scheduling, makes this approach robust for large-scale, heterogeneous video data management (Hwang et al., 17 Jun 2025).