PointVG-R: Geometric Reasoning for Visual Grounding

• The paper presents a novel geometric reasoning pipeline that decomposes visual grounding into four explicit stages, each supervised by specific geometric metrics.
• PointVG-R builds on a pretrained vision-language model by integrating a structured visual chain-of-thought and image-ray re-encoding to interpret pointing gestures accurately.
• The approach uses Adaptive GRPO combined with cold-start supervised fine-tuning on the EgoPoint-CoT dataset, achieving significant improvements in precise pointing localization.

Searching arXiv for the cited paper and closely related context papers to ground the article. {"^^^^^^^^2query2 OR \2^^^^^^^^"PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought\"","max_results":5} {"^^^^^^^^2query2 visual grounding egocentric chain of thought multimodal LLM geometric reasoning","max_results":^^^^2(Li et al., 23 Jun 2026) OR \2query2^^^^} PointVG-R is a reasoning-guided Multi-modal LLM (MLLM) for pointing-based visual grounding that aims to mitigate the cognitive vulnerability of models in interpreting gestural spatial relations by internalizing geometric-aware reasoning and enabling the model to think with images through the strategic integration of Reinforcement Learning (RL) and cold-start data (&&&^^^^2query2^^^^&&&). The method addresses a setting in which models must precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Its central premise is that traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images. PointVG-R therefore couples a structured Visual Chain-of-Thought (V-CoT), image-ray re-encoding, supervised fine-tuning on EgoPoint-CoT, and RL with Adaptive Importance Weighting based on Group Variance (&&&^^^^2query2^^^^&&&).

^^^^2(Li et al., 23 Jun 2026) OR \2^^^^. Problem formulation and conceptual orientation

Pointing-based visual grounding requires a model to infer a referent object from the conjunction of a scene and a pointing gesture. In the formulation used by PointVG-R, the system does not reduce the problem to a single final bounding-box prediction. Instead, it decomposes the reasoning process into explicit intermediate variables: PRESERVED_PLACEHOLDER_^^^^2query2^^^^ for the hand-box, PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^ for fingertip keypoints, $r_3$ for a “draw_ray” tool call, and $r_4$ for the object-box (&&&^^^^2query2^^^^&&&).

This decomposition is intended to simulate the iterative cognitive process humans employ when interpreting pointing gestures. The paper characterizes this as an explicit four-step loop in which spatial relationships—point, ray, and box—are both explicitly emitted as tokens and directly supervised by geometric metrics, specifically MPJPE, cosine, and IoU. A common simplification in multimodal grounding is to treat the visual input as a static latent context and the reasoning process as predominantly linguistic; PointVG-R is organized around the opposite assumption that geometric structure should remain active throughout inference and training (&&&^^^^2query2^^^^&&&).

Inference factorizes as

$$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$

with $R=\{r_1,r_2,r_3\}$ and final box $b=r_4$. This factorization makes the geometric subproblems first-class outputs rather than latent internal states. A plausible implication is that the model’s failure modes can be localized more precisely than in one-shot box regression systems, because errors may arise at hand detection, keypoint extraction, ray construction, or target alignment rather than being collapsed into a single incorrect box.

2. Architecture and representational design

PointVG-R extends a pretrained vision-language foundation, specifically Qwen2.5-VL-Instruct, by adding a structured V-CoT interface and an image-ray re-encoding module (&&&^^^^2query2^^^^&&&). Its architecture has four main components.

The first component is the Image Encoder $E_y$, a ViT-style visual backbone that is frozen during training and maps the input egocentric image $I \in \mathbb{R}^{H\times W\times 3}$ to a global feature tensor

$F_v = E_y(I)\in\mathbb{R}^{d\times N}.$

Here PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2query2^^^^ is the feature dimension and PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2(Li et al., 23 Jun 2026) OR \2^^^^ the number of spatial tokens.

The second component is the Gesture Encoder, implemented through the language backbone PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \22^^^^ augmented with special tokens such as <|box_start|> ... <|box_end|>, <|point_start|> ... <|point_end|>, and <tool_call> ... </tool_call>. Through these tokens, the model decodes the four reasoning variables in sequence: hand-box, fingertip keypoints, ray tool call, and object-box.

The third component is Geometric Reasoning and Image-Ray Re-encoding. After extracting PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^3, a 2D ray is drawn from fingertip into the scene. The model then crops or attends to a thin region PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^4 around that ray and re-encodes it via a lightweight encoder PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^5. The fused feature is

PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^6

This fused representation enriches spatially aligned context for the final box prediction.

The fourth component is the Language Decoder, also PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^7, which consumes PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^8 and the partial token sequence to autoregressively produce the next reasoning step or final answer. The architecture therefore alternates between token-level reasoning and geometry-conditioned visual re-encoding. This suggests that PointVG-R is not merely producing explanatory text around a detector output; rather, the intermediate textual structure is coupled to additional visual computation.

3. Geometric-aware reasoning pipeline

The geometric-aware reasoning pipeline is organized as four explicit stages (&&&^^^^2query2^^^^&&&). Each stage has a distinct prediction target and, during training, its own supervision or reward signal.

In Step ^^^^2(Li et al., 23 Jun 2026) OR \2^^^^, Hand Detection, the model emits PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2^^^^9 between <|box_start|> and <|box_end|> tokens. This stage establishes the local spatial anchor from which the pointing gesture will be interpreted.

In Step 2, Keypoint Extraction, the model predicts two 2D points, root and tip, written as $r_3$^^^^2query2^^^^. Supervision is provided through scale-normalized MPJPE:

$r_3$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^

In Step 3, Pointing Geometry, the model constructs a 2D ray from $r_3$2 into the scene, crops or attends to $r_3$3, re-encodes it via $r_3$4, and fuses the result with $r_3$5. The corresponding ray-angle consistency reward is

$r_3$6

where $r_3$7.

In Step 4, Target Alignment, the model predicts $r_3$8 conditioned on $r_3$9, and the final reward uses

$r_4$^^^^2query2^^^^

The paper’s example CoT trajectory shows the intended syntax: the model identifies the hand box, specifies two points for the finger, issues a draw_ray tool call with start and end coordinates, and then predicts the target box. Because the ray is explicitly represented and then used to re-encode image evidence, the V-CoT is operational rather than purely descriptive. A common misconception is to view chain-of-thought in multimodal systems as an auxiliary explanation layer; in PointVG-R, the chain-of-thought is part of the geometric inference pathway itself.

4. EgoPoint-CoT and supervision signals

To train Stage ^^^^2(Li et al., 23 Jun 2026) OR \2^^^^, the authors built EgoPoint-CoT, a high-quality visual Chain-of-Thought dataset featuring detailed reasoning trajectories (&&&^^^^2query2^^^^&&&). The dataset contains approximately ^^^^2(Li et al., 23 Jun 2026) OR \2^^^^5,^^^^2query2query2query2^^^^ egocentric images with a 7:2:^^^^2(Li et al., 23 Jun 2026) OR \2^^^^ split, each image containing a pointing hand and a target. The annotation structure matches the four-stage reasoning pipeline: $r_4$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^ for the hand box via MMPose plus expert vetting, $r_4$2 for fingertip keypoints (root, tip), $r_4$3 for the ray tool call (start, end), and $r_4$4 for the target box and caption for semantic alignment.

The dataset also includes ^^^^2(Li et al., 23 Jun 2026) OR \2^^^^,^^^^2query2^^^^52 negative samples with no visible pointing, paired with a “No hand” CoT. Average inter-annotator agreement is reported as Fleiss’ $r_4$5. These details are significant because the training signal is not confined to final localization. Instead, it covers the full reasoning trajectory and includes negative cases in which the correct outcome is to identify the absence of a visible pointing gesture.

The construction of EgoPoint-CoT provides the cold-start data required for supervised fine-tuning before RL. This suggests that the subsequent policy optimization is not learning geometric structure from sparse end rewards alone; it is refining an already structured policy whose outputs have been aligned to a predefined reasoning syntax and geometric annotation scheme.

5. Optimization: cold-start SFT and Adaptive GRPO

PointVG-R is trained in two distinct phases (&&&^^^^2query2^^^^&&&). Stage ^^^^2(Li et al., 23 Jun 2026) OR \2^^^^ is Supervised Fine-Tuning (Cold-Start SFT), using

$r_4$6

In this phase, LoRA is applied on $r_4$7 with rank $r_4$8 and $r_4$9. The trainable parameters are the LoRA adapters, token embeddings, and lm_head. The reported hyperparameters are $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$^^^^2query2^^^^, batch $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^, and epochs $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$2.

Stage 2 uses Adaptive GRPO. The paper states that this phase performs full-parameter updates on $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$3 with the backbone frozen. The RL objective is

$$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$4

The reward is multi-dimensional. $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$5, with $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$6, $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$7, and $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$8. There is also a valid syntax or structure reward $$P(b, R \mid I, Q; \theta) = P(r_1 \mid F_v)\; P(r_2 \mid F_v, r_1)\; P(r_3 \mid F_v, r_1, r_2)\; P(r_4 \mid F_v, r_1, r_2, r_3),$$9 and a repetition penalty $R=\{r_1,r_2,r_3\}$^^^^2query2^^^^.

The distinctive element is Adaptive Importance Weighting based on Group Variance. For each prompt, the method samples $R=\{r_1,r_2,r_3\}$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^ rollouts, computes group variance $R=\{r_1,r_2,r_3\}$2, maintains a global EMA $R=\{r_1,r_2,r_3\}$3, and assigns a clipped importance weight

$R=\{r_1,r_2,r_3\}$4

The rescaled advantage is $R=\{r_1,r_2,r_3\}$5, and the final GRPO loss is given in one-step PPO style. The training hyperparameters include AdamW with $R=\{r_1,r_2,r_3\}$6, $R=\{r_1,r_2,r_3\}$7, $R=\{r_1,r_2,r_3\}$8, batch $R=\{r_1,r_2,r_3\}$9, warmup $b=r_4$^^^^2query2^^^^ steps, cosine decay to $b=r_4$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^ minimum, $b=r_4$2 epochs, PPO clipping $b=r_4$3–$b=r_4$4, dual-clip $b=r_4$5, $b=r_4$6 rollouts per prompt, temperature $b=r_4$7, top-$b=r_4$8, length $b=r_4$9, global variance EMA $E_y$^^^^2query2^^^^, $E_y$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^, $E_y$2, $E_y$3, and KL-regularization weight $E_y$4 with target KL $E_y$5.

6. Empirical performance, ablations, and limitations

All experiments use the EgoPoint-CoT test set and report Precision@IoU thresholds $E_y$6, $E_y$7, and $E_y$8, plus mean IoU (mIoU) (&&&^^^^2query2^^^^&&&). The main comparison reports the following results: Qwen2.5-VL (7B) zero-shot achieves $E_y$9, $I \in \mathbb{R}^{H\times W\times 3}$^^^^2query2^^^^, $I \in \mathbb{R}^{H\times W\times 3}$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^, and $I \in \mathbb{R}^{H\times W\times 3}$2; Qwen2.5-VL SFT (standard) achieves $I \in \mathbb{R}^{H\times W\times 3}$3, $I \in \mathbb{R}^{H\times W\times 3}$4, $I \in \mathbb{R}^{H\times W\times 3}$5, and $I \in \mathbb{R}^{H\times W\times 3}$6; Qwen2.5-VL + V-CoT SFT achieves $I \in \mathbb{R}^{H\times W\times 3}$7, $I \in \mathbb{R}^{H\times W\times 3}$8, $I \in \mathbb{R}^{H\times W\times 3}$9, and $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$^^^^2query2^^^^; and PointVG-R (V-CoT + Adaptive RL) achieves $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$^^^^2(Li et al., 23 Jun 2026) OR \2^^^^, $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$2, $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$3, and $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$4. The paper states that PointVG-R outperforms the strongest cold-start baseline by $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$5 mIoU points ($F_v = E_y(I)\in\mathbb{R}^{d\times N}.$6 vs. $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$7) and improves over standard SFT by $F_v = E_y(I)\in\mathbb{R}^{d\times N}.$8 points.

The ablation studies further localize the sources of improvement. Intermediate V-CoT supervision, adding hand ($F_v = E_y(I)\in\mathbb{R}^{d\times N}.$9) and ray plus keypoint (PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2query2query2^^^^), yields cumulative gains up to PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2query2(Li et al., 23 Jun 2026) OR \2^^^^ mIoU. For the RL group statistic, group variance outperforms standard GRPO by PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2query22^^^^ mIoU and also outperforms entropy, gap, and std. For the mapping function, the square-root ratio PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2query2^^^^3 yields best stability and PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2query2^^^^4 mIoU versus linear. This suggests that both the explicit geometric supervision and the variance-aware reweighting are functionally important, rather than either component being sufficient on its own.

The limitations identified in the paper are operational rather than conceptual. The autoregressive V-CoT emits multiple intermediate tokens (PRESERVED_PLACEHOLDER_^^^^2(Li et al., 23 Jun 2026) OR \2query2^^^^5), increasing inference latency relative to one-shot detectors. The four-step token sequence may also be cumbersome for very low-power devices such as AR glasses. The proposed extensions include token pruning or hybrid regression heads to collapse multiple steps into fewer tokens, distillation into a lightweight model that compiles away intermediate syntax, and generalization of the V-CoT approach to other spatial tasks, including 3D pointing, object retrieval, and navigational instruction following. These directions indicate that PointVG-R is positioned both as a concrete pointing-localization system and as a design pattern for integrating explicit geometric reasoning into MLLMs.