Papers
Topics
Authors
Recent
Search
2000 character limit reached

Latent Visual Cache for Video Reasoning

Published 1 Jul 2026 in cs.CV and cs.CL | (2607.02607v1)

Abstract: Video reasoning requires Large Multimodal Models (LMMs) to remain grounded in dense evidence, yet existing systems largely adopt "read-once, generate-many" paradigm, in which visual grounding weakens during generation. This phenomenon has been widely observed and is known as Visual Anchoring Decay. To fill this gap, we introduce Latent Video Cache (Latent-VC), a recurrent latent visual cache inserted into the decoder to preserve compact visual memories throughout reasoning. The cache is trained with supervised contrastive cache alignment and vision-grounded GRPO with a latent grounding reward, while maintaining strict train-inference alignment through native decoder hidden states. Built on Qwen3.5-9B, Latent-VC consistently outperforms strong CoT and SFT+GRPO baselines across six video benchmarks, with especially clear gains on grounding-intensive and long-video tasks. In addition, it also achieves higher accuracy with substantially shorter responses, suggesting that latent visual caching improves video reasoning by preserving visual evidence rather than relying on longer textual chains.

Citations (1)

Summary

  • The paper introduces a novel Latent Visual Cache (Latent-VC) architecture that integrates compact recurrent latent memory into video reasoning, boosting the performance of video reasoning tasks and mitigating visual anchoring decay by which longer reasoning lengths leads to drift from visual context.
  • Latent-VC architecture outperformed CoT and SFT+GRPO baselines on all evaluated benchmarks. The most significant gains were seen in grounding-intensive tasks and with higher video budgets, with average gains between 14.0 to 16.6 in multiple settings.
  • Latent-VC significantly reduces response lengths on image-grounded video tasks by up to 64% while improving accuracy, notably on longer videos and more challenging question splits, indicating enhanced efficiency in summarizing visual information.

Motivation: Visual Anchoring Decay in read-once video reasoning

The paper addresses a structural weakness of current video Large Multimodal Models (LMMs): the "read-once, generate-many" paradigm, in which the full visual token sequence is prepended to the prompt and then attended to only implicitly during autoregressive decoding. As chain-of-thought (CoT) reasoning lengthens, attention over the initial spatiotemporal tokens progressively dilutes — a phenomenon the authors term Visual Anchoring Decay — causing the model to drift toward linguistic priors and produce hallucinated or visually unsupported intermediate steps. The problem is most acute for grounding-intensive tasks (e.g., spatial intelligence, long-video comprehension) where evidence is temporally distributed across many frames.

The Latent-VC architecture

Latent Video Cache (Latent-VC) inserts a compact recurrent latent memory into the decoder between the prompt and the answer. Concretely, after encoding a sampled clip into frozen visual-prefix tokens Z\mathbf{Z}, the model emits SS special <|lvc|> slots. Each slot's input embedding is either a learned seed token (first slot) or the previous slot's decoder hidden state hs−1\mathbf{h}_{s-1}, forming a recurrent chain inside the transformer decoder:

hs=Dθ(e~s,  C<s)\mathbf{h}_s = \mathcal{D}_\theta(\tilde{\mathbf{e}}_s,\; \mathbf{C}_{<s})

The latent states are deterministic computation states, not sampled vocabulary tokens, so the probabilistic factorization applies only to answer tokens. Crucially, the raw video prefix remains in context; the cache acts as a persistent high-fidelity buffer that reduces reliance on direct attention over thousands of raw video tokens. Because everything operates on native decoder hidden states at both training and inference time, Latent-VC avoids the distribution shift that affects prior latent-reasoning approaches such as COCONUT-style methods.

Two-stage optimization

Stage I — supervised contrastive cache alignment. During SFT on key-frame annotations, the SS latent slots are partitioned into contiguous blocks proportional to per-key-frame merged-token counts. Each block's mean hidden state ci,m\mathbf{c}_{i,m} is passed through a trainable projection head PϕP_\phi and aligned via an InfoNCE loss against mean-pooled frozen key-frame features vi,m\mathbf{v}_{i,m}, combined with a masked cross-entropy loss (λlvc=0.1\lambda_{\mathrm{lvc}}=0.1, τ=0.07\tau=0.07). This binds cached latents to specific visual moments rather than allowing degenerate generic summaries.

Stage II — vision-grounded GRPO. Starting from the Stage I checkpoint, Group Relative Policy Optimization (GRPO) uses a composite reward combining accuracy (SS0), format (SS1), temporal grounding (SS2), and a novel latent grounding reward (SS3, threshold SS4). The latent reward measures, for each annotated key frame, the maximum cosine similarity between projected hidden states along the sampled completion trajectory and the frozen key-frame target, averaged over frames and shaped by a piecewise threshold function. It is computed from completion-conditioned trajectories (not the fixed pre-answer rollout), so it varies across the SS5 completions and survives group normalization. The reward is detached — gradients do not flow through the similarity or projection head — but it biases the policy toward visually grounded token trajectories, with KL regularization (SS6) against the Stage I reference limiting reward hacking.

At inference no key-frame annotations are needed; the model performs a fixed 8-step latent rollout before answering.

Main results

Built on Qwen3.5-9B and trained on the Open-o3-Video collection (7,047 SFT samples from STGR; 32,231 GRPO samples across six sources), Latent-VC is evaluated on six benchmarks under a unified Video-R1 protocol at 16/32/64 frame budgets, averaged over three runs.

Setting VSI-Bench VideoMMMU MMVU MVBench TempCompass VideoMME
CoT (64f) 27.9 44.0 61.6 52.0 66.5 48.5
SFT+GRPO (64f) 52.6 59.7 67.2 60.8 72.2 64.6
Latent-VC-9B (64f) 61.9 65.3 68.6 62.5 72.6 66.1

Four findings stand out. First, Latent-VC beats the CoT baseline on every benchmark–budget pair, with average gains of 13.7, 14.5, and 16.1 points at 16, 32, and 64 frames. Second, it also beats a same-backbone SFT+GRPO baseline everywhere (+3.0 to +3.4 average), which the authors argue attributes the improvement to the latent cache itself rather than stronger RL optimization. Third, gains are largest on grounding-intensive benchmarks: up to +34.0 points on VSI-Bench, +24.0 on VideoMMMU, and +17.6 on VideoMME over CoT. Fourth, the advantage grows with the visual budget: at 64 frames the CoT baseline actually degrades on VSI-Bench (27.9) relative to 32 frames, while Latent-VC rises monotonically to 61.9 — direct evidence that the read-once paradigm fails to exploit additional visual context whereas latent caching does not. The method transfers to smaller backbones: Latent-VC-4B improves the Qwen3.5-4B average from 47.6 to 61.5 at 64 frames (+13.9).

Mechanistic analyses and efficiency

Attention analysis supports the causal claim: measuring generation-time attention mass over video tokens, the baseline decays rapidly while Latent-VC maintains a substantially higher attention plateau in middle and late stages, with head-level heatmaps showing stronger, more distributed video-token attention in later layers. A depth ablation shows performance peaks at 8 latent steps (61.9 on VSI-Bench) and declines beyond (58.1 at 64 steps), indicating a moderate recurrent depth suffices and excessive recursion introduces redundancy.

A notable efficiency result: Latent-VC produces responses 49–64% shorter than CoT while being uniformly more accurate (e.g., VSI-Bench drops from 1209 to 464 tokens), and its response length remains essentially stable as the frame budget increases from 16 to 64. Gains are largest on longer videos (+68.6% relative on the long subset of Video-MME) and harder question splits, indicating the model absorbs additional visual evidence as compact internal computation rather than verbose textual chains. A qualitative case study shows correct spatial-relation inference in 90 tokens where both the base model and SFT+GRPO fail after hundreds of tokens.

Limitations and open questions

The authors concede three limitations directly relevant to interpreting these results. Evaluation is confined to the Qwen3.5 family, leaving generalization to other architectures and longer-context video LMMs unverified. The Stage II latent grounding reward depends on temporal-spatial supervision (annotated key frames), restricting applicability where such annotations are scarce or noisy — a meaningful constraint given that this reward is central to the method. Finally, the interpretability of the latent cache states remains an open problem; despite the case study, there are no diagnostic tools for inspecting what the cached representations encode, and robustness under noisy or streaming video is untested. An additional implicit assumption is that block-to-key-frame assignment during SFT (annotation-driven, not learned routing) does not bias which visual moments get anchored.

Conclusion

Latent-VC demonstrates that a recurrent latent visual cache, optimized by contrastive alignment followed by vision-grounded GRPO, can counteract Visual Anchoring Decay in video LMMs while strictly maintaining train-inference consistency through native hidden states. The combination of consistent gains across six benchmarks — particularly on grounding-intensive and long-video settings — together with markedly shorter outputs, supports the paper's central hypothesis that preserving visual evidence in latent space, rather than extending textual reasoning chains, is an effective basis for grounded long-form video reasoning.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.