Visually Grounded Thinking

• Visually grounded thinking is a multimodal reasoning paradigm that integrates step-by-step textual thoughts with explicit visual evidence to justify answers.
• It addresses the binding problem by imposing sequential visual structures, such as horizontal line interventions, to reduce cross-object interference.
• Various methods employ explicit grounding tags, replay actions, and specialized rewards to ensure that reasoning steps are stably linked to their corresponding image regions.

Visually grounded thinking denotes a family of multimodal reasoning regimes in which intermediate reasoning is tied to concrete visual evidence rather than remaining purely in language space. In recent vision–language research, the term covers several operational forms: interleaving natural-language thoughts with explicit point or box groundings, issuing replay or crop actions that retrieve image regions during inference, imposing low-level spatial structure on the image to induce sequential parsing, and learning word, sentence, or concept representations whose semantics are anchored in images rather than text alone (Zhang et al., 15 Jun 2026, Wang et al., 13 Jun 2025, Izadi et al., 27 Jun 2025). Across these formulations, the central claim is that visual reasoning should not only produce an answer, but should also preserve or expose the perceptual evidence that makes the answer correct.

1. Formal definitions and representational scope

A common formalization treats visually grounded thinking as a reasoning trace over an image $I$ and question $Q$ in which textual thoughts are interleaved with explicit grounding tags. In one formulation, a visually grounded reasoning trace $T$ consists of natural-language token sequences inside > ... </think> together with tags of the form <obj> \phi \mid \text{coords} </obj>, where $\phi$ is a descriptive phrase and the coordinates are given either in box mode as $[x_1,y_1,x_2,y_2]$ or in point mode as $[x,y]$, all in the normalized $[0,1000]$ image reference frame. The model implements a policy $\pi_\theta:(I,Q)\to T$ such that each grounding tag identifies the exact image region supporting the immediately preceding or following thought step (Zhang et al., 15 Jun 2026).

A closely related formulation defines visually grounded reasoning as generating an answer $A$ to question $Q$ about an image $Q$0 by interleaving textual reasoning tokens with discrete replay actions that fetch visual features. In that setting, the reasoning chain contains natural-language tokens, a special ground signal <sot>[x_1,y_1,x_2,y_2]<eot>, and an embedded visual feature sequence drawn from a precomputed feature map $Q$1; the final answer is read off at the end of the chain (Wang et al., 13 Jun 2025). This interface makes grounding an executable operation inside the reasoning process rather than a post hoc explanation.

A more axiomatic account appears in the theoretical framework called “Visually Grounded Reasoning” for visual question answering. There, a VQA model is written as $Q$2, and two predicates are defined on each inference: correct reasoning $Q$3 and correct visual grounding $Q$4. The central proposition is $Q$5 together with $Q$6, making correct visual grounding and correct reasoning jointly necessary for a correct answer (Reich et al., 2024). This formulation is not an interface specification; it is a criterion for what a correct multimodal inference must satisfy.

Taken together, these definitions distinguish visually grounded thinking from purely textual chain-of-thought. It is not only “reasoning about images,” but reasoning whose semantically important steps remain connected to the relevant regions, objects, or spatial partitions of the image.

2. Binding, serial attention, and visual input structuring

A major motivation for visually grounded thinking is the binding problem. In vision–LLMs, the binding problem is the failure to reliably associate perceptual features with their correct visual referents when multiple objects share similar features. The reported consequence is systematic error on counting, visual search, scene description, and spatial-relation inference, especially in cluttered or confusable scenes. The proposed diagnosis is that current VLMs process visual features largely in parallel and lack mechanisms for spatially grounded, sequential attention (Izadi et al., 27 Jun 2025).

One inference-only intervention addresses this limitation by modifying the image itself. For an image of height $Q$7, $Q$8 equidistant horizontal lines are overlaid at positions

$Q$9

In the reported experiments, $T$0, producing four horizontal bands, optionally labeled “Row 1” through “Row 4.” The accompanying prompt adds the clause “Scan the image sequentially based on horizontal lines exist in the image,” or, in counting, instructs the model to count targets row by row and then sum them (Izadi et al., 27 Jun 2025).

The significance of this design lies in where the intervention occurs. The paper explicitly contrasts it with purely textual strategies, including Chain-of-Thought prompting, and reports that textual strategies are insufficient and can even degrade performance, whereas the visual modification is essential for the observed gains. The explanation given is an “Early Intervention”: by structuring the image before any encoding, the model’s visual tokens remain localized, cross-object interference is reduced at the first layer, horizontal lines carve the scene into manageable subregions, and the simultaneous feature load is lowered. This suggests that visually grounded thinking can be induced not only by explicit region references during reasoning, but also by changing the visual input so that sequential parsing becomes easier to realize (Izadi et al., 27 Jun 2025).

3. Grounding operators and architectural mechanisms

Recent systems instantiate visually grounded thinking through distinct grounding operators. In VGR, the model builds on LLaVA-NeXT with a CLIP-ViT-L/14@336 vision encoder, a 2-layer MLP adapter, and a Vicuna-v1.5 backbone. During generation, the LLM may emit <sot> followed by four continuous coordinate values and then <eot>; a detection head maps the hidden state at <eot> to predicted box parameters, and the corresponding region is extracted from the full feature map, pooled, flattened, and appended immediately after the grounding signal into the context (Wang et al., 13 Jun 2025). Grounding is therefore implemented as selective feature replay.

Point-RFT uses a different operator. Each rationale is written inside <think> ... </think>, and each reasoning step begins with a grounding tag of the form <points x0="..." y0="..." x1="..." y1="...">description</points>, which records raw-pixel coordinate pairs for the referenced visual element. The model is trained so that the rationale and the coordinate reference co-occur in a single autoregressive output string, culminating in <answer> ... </answer> (Ni et al., 26 May 2025). This realizes the paper’s description that the model “thinks while pointing.”

ViGoRL treats grounding as a tool-mediated action. At each step, the decoder emits a tagged block <think> ... containing the next textual thought, followed by a <tool_call> ... </tool_call> specifying the next coordinate $T$1, or a final <answer> ... </answer>. The environment executes the crop request and returns a centered crop $T$2, enabling multi-turn reasoning with zoomed-in visual feedback (Sarch et al., 29 May 2025). Here the grounding operator is not only representational but also interactive.

GRiP introduces a simpler controller abstraction. Its policy decides, step by step, whether to emit text or invoke a bounding-box tool via a <LOCATE> action. The paper describes this as a two-level hierarchy comprising a perceptual subroutine, which extracts a box and updates visual memory, and a cognitive subroutine, which generates the next text reasoning based on language and the updated visual memory (Wei et al., 27 Nov 2025). The resulting alternation between localization and language generation is presented as hierarchical and flexible visual grounded reasoning.

In video-centered social reasoning, VEGAS extends the same principle temporally. Its LGS sampler computes relevance scores for frames given the question, selects top-$T$3 key frame embeddings, and a Temporal Attention Module restores order before the sequence is fed to the LLM (Li et al., 3 Apr 2025). Although the object of grounding is now a subset of frames rather than a 2D box or point, the architectural logic remains the same: reasoning conditions on explicitly selected perceptual evidence.

4. Supervision, rewards, and internalization

The training of visually grounded thinking has rapidly shifted from static supervised traces to compound objectives that reward both answer quality and grounding quality. VGR first constructs VGR-SFT, a 158.1K-sample dataset assembled through cold-start generation, reject sampling, and annotation-model scaling. Its detection head is supervised by

$T$4

while text tokens are trained with cross-entropy and replayed image tokens do not contribute to loss (Wang et al., 13 Jun 2025).

“Thinking with Visual Grounding” synthesizes 19,909 traces containing 107,613 total grounding tags over 72,381 distinct objects. Training proceeds by supervised fine-tuning on traces containing <think>, <obj>, coordinates, and boxed final answers, followed by grounding-aware reinforcement learning. The reward combines answer correctness, dense grounding reward, formatting rewards, and a truncation penalty, with weights $T$5, $T$6, and $T$7 (Zhang et al., 15 Jun 2026). The dense reward differs by interface: box mode uses IoU against ground-truth boxes, whereas point mode uses per-object F1 against masks.

Point-RFT adopts a two-stage recipe specialized for visual document understanding. First, a 71K-example format-finetuning stage teaches the model to produce grounded rationales; second, reinforcement finetuning on ChartQA uses a reward $T$8, where $T$9 checks whether the output matches the required <think> ... <answer> format and $\phi$0 is an indicator for answer correctness (Ni et al., 26 May 2025). The paper’s claim is that grounded CoT is more effective than text-only CoT for this class of tasks.

GRiP’s reinforcement learning stage aggregates four scalar rewards—$\phi$1, $\phi$2, $\phi$3, and $\phi$4—into

$\phi$5

and optimizes the policy with GRPO. The Salience-Weighted IoU reward prioritizes “mission-critical” objects, while the Multi-Heuristic Reward scores the generated trajectory against three expert reasoning styles: bottom-up evidence-driven, top-down context-aware, and deductive verification (Wei et al., 27 Nov 2025). This formalizes a distinction between grounding fidelity and reasoning flexibility.

A different line of work argues that explicit grounding need not remain visible at inference. iVGR reports that off-the-shelf models trained with grounded CoT often perform worse when forced to emit boxes than when prompted with standard textual CoT. Its solution is a dual-stream RL framework: a grounded stream maintains localization pressure, while a textual stream is aligned to the grounded one by a consistency reward in $\phi$6 scored against a high-quality grounded reference stored in a rollout archive (Zhang et al., 29 May 2026). The paper’s hypothesis is that visual localization can be internalized into textual CoT and that mandatory explicit grounding can introduce unnecessary interference with answer prediction.

5. Empirical profile across tasks and domains

The reported empirical gains are concentrated in tasks where the answer depends on correct feature-to-object assignment, small-region localization, or access to fine-grained perceptual detail. On a controlled 2D synthetic benchmark with GPT-4o, the horizontal-line intervention improves visual-search harmonic mean from 0.48 to 0.73, counting accuracy from 12.00% to 38.83%, scene-description edit distance from 1.94 to 1.62, and spatial-relationship accuracy from 43.00% to 52.50%; the paper further reports $\phi$7 by paired bootstrap and consistency across GPT-4o, Claude 3.5-Sonnet, Qwen2.5-VL, and LLaMa4 (Izadi et al., 27 Jun 2025).

VGR reports that, relative to the LLaVA-NeXT-7B baseline, it uses only 30% of the image token count while improving MMStar from 37.6 to 41.7, AI2D from 66.6 to 73.7, and ChartQA from 54.8 to 67.7. Ablations attribute part of the gain to the grounding signals, part to replay features, and part to the detection loss (Wang et al., 13 Jun 2025). The results position selective replay as an efficiency-oriented variant of visually grounded thinking.

In “Thinking with Visual Grounding,” adding grounded thinking to Gemma3-4B-IT improves counting on TallyBench from 33.33% to 39.31% in point mode with grounding and on CountQA from 9.87% to 11.65%. On spatial reasoning, box mode with grounding reaches 68.08% on VSR, 59.93% on EmbSpatial, 38.68% on SpatialMQA, and 37.68% on Multihop, while Gemma3-27B-IT records 69.25%, 62.09%, 38.99%, and 30.94% respectively (Zhang et al., 15 Jun 2026). The authors conclude that point grounding is well suited to counting and box grounding to spatial tasks.

Point-RFT reports a substantial document-reasoning effect. On ChartQA, accuracy increases from 70.88% for the base model to 83.92% for text-only Base-RFT, 87.21% for Point-SFT, and 90.04% for Point-RFT. Its out-of-domain average rises from 35.94% to 53.16%, and the detailed breakdown includes Counting from 21.00 to 78.50 and TabMWP from 61.00 to 70.90 (Ni et al., 26 May 2025). The paper presents this as evidence that grounded CoT generalizes better than text-only CoT.

ViGoRL extends the picture to spatial reasoning, GUI grounding, and web interaction. On V*Bench, Qwen-2.5-7B rises from 78.0% to 86.4%; on ScreenSpot-V2, the 7B model reaches 91.0%; on VisualWebArena, the 7B model reaches 11.2%; and on SAT-2, ViGoRL-7B reaches 67.5% (Sarch et al., 29 May 2025). The same paper reports behavioral changes: regions explored increase from 1.44 in zero-shot to 3.5 in ViGoRL, grounded subgoals per trace from 0.07 to 1.1, visual verification steps from 0.14 to 0.39, and backtracking events from 0.00 to 0.47.

GRiP reports 51.3 on TreeBench versus 37.0 for Qwen2.5-VL-7B, with mIoU increasing from 30.0 to 45.0, and 91.9 on V* Bench, alongside HR-Bench scores of 78.6 in 4K and 75.0 in 8K (Wei et al., 27 Nov 2025). VEGAS, in turn, situates visual grounding in social intelligence: on supervised MCQ evaluation it reports 80.90% versus 75.94% for MMTC-ESC and 74.13% for DeSIQ, and in zero-shot open-ended answering 66.0% versus 52.5% for Video-LLaVA (Li et al., 3 Apr 2025). AffectVisDial broadens the domain further by providing 50,000 10-turn visually grounded dialogs, 500,000 QA pairs, dialog-based emotion prediction, and affective explanation generation, with a reported 62.2% accuracy for a RoBERTa emotion classifier and best explanation models around BLEU $\phi$8 and Emo-F1 $\phi$9 when using textual image summaries plus full dialog (Haydarov et al., 2023).

These results support a narrow empirical conclusion that is repeated across papers: visually grounded thinking is most useful when success depends on local evidence selection, instance individuation, or the ability to audit intermediate perceptual commitments.

6. Antecedents, misconceptions, and unresolved questions

The contemporary literature has clear antecedents in grounded semantics and multimodal inference. Vu et al.’s grounded textual entailment task adds an image $[x_1,y_1,x_2,y_2]$0 to the premise–hypothesis pair $[x_1,y_1,x_2,y_2]$1, treating the image as the relevant “world” or “situation.” Their visually augmented BiMPM improves from 86.41 to 86.99 on the full V-SNLI test set and from 72.55 to 73.75 on V-SNLI_hard, but the foil-image test produces only small drops, which the authors interpret as evidence that image grounding is shallow rather than optimal (Vu et al., 2018). This earlier result already distinguished multimodal input from genuinely grounded reasoning.

Parallel work on representation learning showed that visual grounding can alter semantics even when the end task is not explicit visual reasoning. Kiela et al. train sentence encoders to predict image features or alternative captions and report improved transfer over text-only models, alongside stronger grounded word embeddings (Kiela et al., 2017). Merkx et al. show that visually grounded embedding similarities better predict human reaction times in semantic priming and correlate well with human similarity ratings, with unique variance beyond large text-only models (Merkx et al., 2022). Shahmohammadi et al. argue that a learned alignment can indirectly ground unseen and abstract words and report improvements not only for concrete but also abstract words, especially under cognitively plausible data regimes (Shahmohammadi et al., 2022). In a more compositional direction, “Visually Grounded Concept Composition” constructs a Concept and Relation Graph and reports improved text-to-image matching under significant compound divergence from training data (Zhang et al., 2021).

Several recurrent misconceptions are explicitly challenged in the literature. One is that out-of-distribution VQA accuracy is a reliable proxy for visual grounding. The theoretical VGR framework argues that popular OOD splits still permit shortcut exploitation and proposes GQA-AUG to better enforce the requirement that answer correctness implies both reasoning and visual grounding (Reich et al., 2024). A second misconception is that explicit grounding at inference is always beneficial. iVGR reports the opposite for some grounded-CoT models and attributes the degradation to noisy localization and coordination overhead (Zhang et al., 29 May 2026). A third is that adding visual input automatically yields deep grounding. The grounded textual entailment results, especially the small foil-image effect, caution that visual information can act as a side-channel rather than as a genuine world model (Vu et al., 2018).

This suggests that visually grounded thinking is best understood not as a single technique but as a design space. Some systems externalize evidence with points, boxes, or replay actions; some impose structure on the image before encoding; some learn grounding-sensitive rewards; and some seek to internalize localization so that textual reasoning remains visually informed without mandatory coordinate emission. A plausible implication is that future progress will depend less on whether a model can emit a box and more on whether its intermediate commitments remain stably coupled to the image regions that make them true.