GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs

Abstract: Long-video understanding in VLMs is bottlenecked by a single monolithic forward pass over thousands of frames at quadratic attention cost. A common mitigation is to first select a small subset of informative frames before the forward pass; common for training-free selectors via auxiliary encoder-space similarities. Such signals are capped by contrastive pretraining, which usually fails on reasoning-heavy queries (negation, cross-frame counting, holistic summarization). We propose GridProbe, an efficient training-free posterior-probing inference paradigm that scores evidence in answer space using a frozen VLM's own reasoning and then selects question-relevant frames adaptively, resulting in sub-quadratic attention cost with little to no accuracy loss. We arrange frames on a $K{\times}K$ grid and run lightweight row R and column C probes, where each probe reads its peak posterior as a query-conditioned confidence. The outer product of R and C yields an interpretable importance map whose skewness and kurtosis drive Shape-Adaptive Selection, a closed-form rule that reliably replaces the fixed frame budget $M$ with a per-question $M_{\mathrm{eff}}$. We show empirically that $M_{\mathrm{eff}}$ tracks intrinsic question difficulty without ever seeing the answer, a sign of test-time adaptive compute. On Video-MME-v2, GridProbe matches the monolithic baseline within $1.6$ pp Avg Acc at $3.36\times$ TFLOPs reduction, while on LongVideoBench it Pareto-dominates the baseline ($+0.9$ pp at $0.35\times$ compute). Because the selector and QA models can be decoupled, pairing a small 2B selector with a stronger 4B or 8B QA is strictly Pareto-dominant over the 2B monolithic baseline (up to $+4.0$ pp at $0.52\times$ compute, on average), with no retraining. Finally, the interpretability of the importance maps opens future avenues for behavioral diagnostics, grounding, and frame-selection distillation.

• The paper introduces GridProbe, a novel method that leverages posterior probing to dynamically select frames, enabling efficient long-video inference.
• It employs a two-stage pipeline with row/column probing to construct importance maps that reflect query complexity and adapt compute allocation.
• Empirical results demonstrate Pareto improvements in accuracy versus compute, achieving significant TFLOPs reduction on long-video benchmarks.

GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs

Introduction

This paper introduces GridProbe, a novel training-free, posterior-probing inference paradigm addressing the computational bottlenecks in long-video Vision-LLMs (VLMs). Conventional video VLMs suffer from quadratic attention costs when handling thousands of frames, motivating frame selection methods that often operate in encoder space. GridProbe innovates in both compute and selection quality by probing directly within the answer space of a frozen VLM, constructing question-conditioned importance maps that adaptively determine per-query frame subset sizes. This approach enables efficient, interpretable, and reasoning-grounded inference that is strictly Pareto-dominant with respect to existing baselines in accuracy-compute trade-offs.

Methodology

GridProbe replaces the traditional monolithic forward pass with a two-stage pipeline. Frames are sampled uniformly into a $K \times K$ grid (yielding $K^2$ candidate frames), then axis-aligned probing along rows and columns is performed. Each probe consists of evaluating the QA VLM's posterior confidence for a subset of frames, quantifying evidence via the peak probability over possible answers.

Figure 1: GridProbe pipeline: Stage 1 utilizes $2K$ row/column probes on $K^2$ candidate frames to yield an importance map, followed by Stage 2 which executes a focused pass on adaptively-selected top-$M_{\mathrm{eff}}$ frames.

The importance map $$M[r,c] = c^{\mathrm{row}_r} \cdot c^{\mathrm{col}_c}$$ is derived from multiplying the probe confidences of local (row) and global (column) subsets, efficiently combining temporal and distributed reasoning signals. Unlike prior encoder-space selectors, this answer-space probing leverages the VLM's complete reasoning capacity, particularly benefiting queries involving negation, temporal counting, or holistic summarization.

Figure 2: Encoder-space (top) vs answer-space (bottom) selection signals, illustrating the superior correlation of answer-space confidence with reasoning-heavy query relevance.

Adaptive computation is achieved by analyzing the skewness and kurtosis of the importance map distribution. The per-question effective frame budget $M_{\mathrm{eff}}$ is set with a closed-form rule, allowing the number of selected frames to dynamically respond to the intrinsic complexity and evidence distribution of the QA task.

Empirical Evaluation

GridProbe is evaluated on the Video-MME-v2 (Fu et al., 6 Apr 2026) and LongVideoBench [wu2024longvideobench] benchmarks using Qwen3-VL-Instruct backbone models at 2B, 4B, and 8B parametric scales. The primary metrics include Average Accuracy, Non-Linear grouped scoring, and per-question TFLOPs for compute profiling.

Figure 3: (a) VMME-V2 Pareto frontier for QA model sizes: GridProbe variants (green) Pareto-dominate the 2B baseline; (b) Compute reduction across $K$ at fixed 2B QA.

GridProbe exhibits superior compute-efficiency:

• On Video-MME-v2, single-model operation trades $1.63$pp Avg Acc for a $3.36\times$ reduction in TFLOPs, matching the monolithic baseline within $K^2$0pp with substantial compute savings.
• On LongVideoBench, GridProbe Pareto-dominates the baseline ($K^2$1pp at $K^2$2 compute).
• Cross-model pipelines (e.g., 2B selector combined with a 4B or 8B QA) yield up to $K^2$3pp gain at $K^2$4 compute, achieved without retraining.
• The adaptive $K^2$5 mirrors intrinsic question difficulty, allocating compute proportionally to complexity as detected from the importance map.

  Figure 4: GridProbe’s adaptive $K^2$6 (blue) mirrors baseline accuracy (red) as a function of signed importance map skewness, evidencing compute allocation aligned with question difficulty.

GridProbe also demonstrates interpretability: importance maps make the VLM's evidence-gathering explicit and enable behavioral diagnostics.

Figure 5: Three Video-MME-v2 queries exercise distinct distribution-shape regimes ($K^2$7): GridProbe adapts $K^2$8 from highly localized (5 frames) to holistic (140 frames) scenarios.

Ablation Studies

• Selector Quality: At matched $K^2$9, GridProbe outperforms encoder-space selectors (MDP3) in both accuracy and compute, especially on reasoning-heavy tasks.
• Adaptive Sizing: Dynamic $2K$0 driven by shape statistics improves grouped scores by $2K$1 Non-Lin and $2K$2 Long bin pp at comparable compute.
• Selector Size and Frame Ordering: Smaller selectors paired with stronger QA backbones provide one-directional amortization benefits. Maintaining temporal ordering in the focused pass is preferred.
• Collation Variant: Tiling selected frames into a single image greatly reduces compute (to $2K$3) with a mild accuracy penalty.

Practical and Theoretical Implications

GridProbe offers a fixed-schedule, training-free mechanism for adaptive test-time compute, avoiding the orchestration overhead historically associated with agentic or multi-step inference strategies. Empirical results show that per-query compute allocation strongly correlates with question difficulty, a key advance for resource-constrained VLM deployment.

The answer-space probing paradigm is robust to reasoning complexities, outperforming encoder-space or static-budget multimodal frame scorers on distributed and compositional queries. The interpretability of importance maps provides a foundation for future work in behavioral analysis, evidence grounding, and frame-selection distillation.

Future Directions

Several extensions are natural: generalizing probe confidence to open-ended QA, adapting the shape-statistic weighting to video length or density, and incorporating additional per-cell information such as attention magnitudes or external retrieval signals. GridProbe’s framework is readily extensible, potentially applicable to hierarchical or multi-scale video representation, and amenable to integration with agentic planning or learnable selectors for further gains.

Conclusion

GridProbe establishes a principled, efficient, and interpretable posterior-probing inference paradigm for long-video VLMs. It operates in answer space, adaptively allocates compute in direct response to question complexity, and achieves strict Pareto improvement on accuracy-compute trade-offs without retraining. The framework's interpretability and adaptive behavior mark it as a robust solution for practical deployment and further exploration in reasoning-heavy video-language tasks.