When to Think and When to Look: Uncertainty-Guided Lookback (2511.15613v1)

Abstract: Test-time thinking (that is, generating explicit intermediate reasoning chains) is known to boost performance in LLMs and has recently shown strong gains for large vision LLMs (LVLMs). However, despite these promising results, there is still no systematic analysis of how thinking actually affects visual reasoning. We provide the first such analysis with a large scale, controlled comparison of thinking for LVLMs, evaluating ten variants from the InternVL3.5 and Qwen3-VL families on MMMU-val under generous token budgets and multi pass decoding. We show that more thinking is not always better; long chains often yield long wrong trajectories that ignore the image and underperform the same models run in standard instruct mode. A deeper analysis reveals that certain short lookback phrases, which explicitly refer back to the image, are strongly enriched in successful trajectories and correlate with better visual grounding. Building on this insight, we propose uncertainty guided lookback, a training free decoding strategy that combines an uncertainty signal with adaptive lookback prompts and breadth search. Our method improves overall MMMU performance, delivers the largest gains in categories where standard thinking is weak, and outperforms several strong decoding baselines, setting a new state of the art under fixed model families and token budgets. We further show that this decoding strategy generalizes, yielding consistent improvements on five additional benchmarks, including two broad multimodal suites and math focused visual reasoning datasets.

• The paper introduces an uncertainty-guided lookback decoding method that dynamically injects visual grounding prompts to improve LVLM reasoning efficiency and accuracy.
• The methodology systematically compares chain-of-thought reasoning depth with breadth sampling, revealing diminishing returns and benefits concentrated in specific model capacities.
• Empirical results show up to 6-point accuracy gains and significant token reduction across multiple benchmarks, offering a robust compute–accuracy trade-off.

Uncertainty-Guided Lookback: Adaptive Visual Reasoning in LVLMs

Introduction

The paper "When to Think and When to Look: Uncertainty-Guided Lookback" (2511.15613) systematically investigates the efficacy and failure modes of “thinking”—explicit test-time chain-of-thought (CoT) reasoning—in state-of-the-art large vision–LLMs (LVLMs). Contrary to prevailing assumptions about the monotonic utility of deeper reasoning, this work demonstrates that increased reasoning depth does not necessarily correlate with greater accuracy or better visual grounding. The authors introduce a principled analysis across InternVL3.5 and Qwen3-VL families on MMMU and several visual reasoning benchmarks, developing a token-level uncertainty-guided lookback controller that selectively inserts image-refocusing prompts based on dynamic uncertainty signals. This adaptive decoding paradigm enables more efficient and robust multimodal reasoning, significantly advancing the state of test-time reasoning strategies for LVLMs.

Quantitative Analysis of Breadth vs. Depth in LVLM Reasoning

The authors offer a systematic comparison of sampling-based breadth (multiple solution paths) and chain-of-thought depth (longer reasoning traces) in LVLM decoding modes. By evaluating ten variants from InternVL3.5 and Qwen3-VL under standardized, generous token budgets and multi-sample (pass@$k$) settings, distinctive scaling dynamics are observed.

For all model capacities, increasing the sampling rate $k$ produces steep early accuracy improvements, especially for lower-capacity models (Figure 1). Yet, beyond $k=8$, marginal benefits plateau, highlighting diminishing returns to additional sampling. Explicit “thinking” mode (CoT) consistently elevates the performance curves, but its relative impact narrows as model scale increases.

Figure 1: Pass@k accuracy on MMMU for 10 LVLM variants, illustrating early steep gains with larger k and pronounced but eventually saturating improvements from thinking mode.

Mean accuracy and variance analysis reinforce that reasoning traces enhance solution reliability and stability, particularly in lower-capacity models. However, these gains are not uniformly distributed across task types. For certain recognition-intensive domains, instruct (“no thinking”) decoding remains competitive or superior.

Figure 2: Category-level performance heatmaps; warmer indicates above-average z-scored accuracy, revealing category-specific regimes where reasoning is beneficial or detrimental.

Figure 3: Pass@k boxplots further illustrate that thinking variants are both more accurate and less variable across evaluation seeds than instruct baselines.

A careful failure analysis reveals cases where instruct mode solves tasks with minimal tokens, while the same model’s thinking variant consistently fails, often producing longer but off-target chains.

Figure 4: Example where the 32B thinking model fails on all 10 samples, while instruct with a single token is correct.

Token Economy and the Computation–Reasoning Tradeoff

Token usage analysis exposes the computational overhead of test-time reasoning. Thinking variants systematically consume more tokens per response, especially on easier questions and in smaller models, where the increased sequence length does not translate into accuracy improvement. Scaling to larger model sizes leads to shorter, more efficient and generally correct chains, suggesting emergent concision and selectivity in high-capacity LVLMs.

Figure 5: Token footprints of instruct vs. thinking, split by model size and question difficulty, demonstrating that reasoning inflates computational cost—often without corresponding accuracy gains on easy items, especially in small models.

Diagnosing Visual Grounding: Token-Level Visual Sensitivity Probes

To diagnose when reasoning fails to ground in visual context, the authors propose token-level probes based on perplexity contrasts under three image conditions: real, noise, and absent. The $\Delta_\text{content}$ (real-noise) and $\Delta_\text{presence}$ (noise-absent) metrics quantify how image content or mere presence shapes the probability of next-token predictions. Correct trajectories exhibit sustained negative $\Delta_\text{content}$ dips, signifying repeated, image-dependent reasoning, while wrong or drifting chains lack these grounding signals.

Figure 6: Token-level DeltaPPL dynamics trace differences in per-token perplexity between real, noise, and absent image contexts along reasoning step index and position, clearly separating grounded from ungrounded chains.

Mining these statistics yields phrase templates (“lookback” and lexical pause cues) that frequently co-occur with visual re-examination and high uncertainty. These cues can be used for lightweight online detection of loss of visual grounding.

Figure 7: Examples of extracted “lookback” sentences that explicitly prompt the model to re-examine task-relevant visual information at moments of high uncertainty.

Uncertainty-Guided Lookback Decoding

Building on these diagnostic insights, the paper introduces a training-free, model-agnostic decoding controller. Upon detecting lexical pause phrases (mined from $\Delta_\text{presence}$-peaked steps) during streamed autoregressive generation, the controller injects visually grounding “lookback” prompts (derived from $\Delta_\text{content}$-negative steps). Optionally, parallel short branches are explored post-lookback, each scored for visual helpfulness via offline content contrast. The best branch is continued, ensuring deliberate refocusing on image context in uncertain or drifting trajectories.

This method triggers adaptive, tightly-localized use of reasoning compute—substantially raising overall accuracy while reducing excess token usage compared to naive always-on thinking. Unlike prior static CoT protocols, the lookback approach dynamically routes computation to the instances and locations where visual reasoning confers maximum benefit.

Empirical Results and Numerical Highlights

Across MMMU and five additional vision–language and math-focused reasoning datasets, uncertainty-guided lookback yields consistent improvements over standard thinking and strong adaptive text-only baselines (DEER, DeepConf, REFRAIN). On MMMU, Pass@1 accuracy is raised by 2–3 points for all Qwen3-VL sizes, while average token usage drops by 35–45%. Domain-specific gains in complex reasoning categories (Diagnostics, Energy, Power) approach +6 points, shifting the Pareto frontier for the accuracy–compute tradeoff. On math and multimodal benchmarks, gains of +4 to +6 points are observed at lower scales, with retained or improved efficiency at all scales.

Major empirical findings:

• Thinking mode only helps in select regimes; long-wrong chains from small models are common and costly.
• Adaptive lookback outperforms strongest text-only adaptive CoT approaches across vision-language tasks.
• Improvements generalize robustly across domains, model capacities, and difficulty levels.

Theoretical and Practical Implications

This work provides the first systematic empirical evidence that test-time explicit reasoning in LVLMs demands principled, instance- and token-level control, rather than blind application. The combination of token-level uncertainty diagnostics with adaptive grounding prompts constitutes a general recipe for controlling multimodal deliberation, applicable to a range of foundation model architectures and task families.

Pragmatically, lookback decoding achieves superior compute efficiency without retraining, making it suitable for real-world latency- or resource-bound applications. The method naturally extends to other settings with access to visual frontends and autoregressive token streams.

Future developments are anticipated in:

• Integrating fine-grained stateful controllers for prompt injection and dynamic chain pruning in LVLM toolchains.
• Cross-modal early-exit strategies and fine-grained visual grounding detection, potentially with online learning of lexical trigger vocabularies.
• More general probe-based reasoning controllers across audio, video, and other non-textual modalities.

Conclusion

More test-time “thinking” is not generally better for contemporary LVLMs. Instead, targeted, uncertainty-aware insertion of visual grounding steps can improve both accuracy and computational efficiency. The uncertainty-guided lookback approach operationalizes this insight in a model-agnostic way, setting new standards for adaptive, efficient multimodal reasoning. This paradigm offers a blueprint for future LVLM deployments seeking robust, cost-effective, and highly interpretable visual reasoning strategies.

(2511.15613): https://arxiv.org/abs/(2511.15613)