- The paper presents a novel visual chain-of-thought framework that decomposes pointing localization into four stages: hand detection, fingertip extraction, pointing ray construction, and bounding box alignment.
- It employs a two-stage training paradigm with supervised fine-tuning and reinforcement learning using adaptive importance weighting, achieving a 15.86-point mIoU improvement over traditional MLLMs.
- Empirical results on the EgoPoint-CoT dataset demonstrate robust spatial reasoning in complex scenarios, paving the way for enhanced applications in human–robot interaction and assistive vision.
PointVG-R: Internalizing Geometric Reasoning in Multimodal LLMs for Pointing Localization
Motivation and Background
Gesture-based visual grounding, specifically the localization of objects indicated by pointing gestures in egocentric images, mandates a fine-grained understanding of spatial relationships and geometric cues. Traditional MLLMs encode images into global feature representations and perform reasoning primarily within the linguistic modality, neglecting explicit spatial constraints of pointing behaviors. This results in cognitive fragility—a tendency to hallucinate, misinterpret, or favor salient objects over the spatially relevant ones in complex first-person scenarios. PointVG-R addresses this gap by incorporating explicit geometric reasoning, emulating human sequential cognitive procedures for gesture interpretation and object anchoring.
Methodological Framework
Visual Chain-of-Thought Reasoning
PointVG-R formalizes pointing-based grounding as a probabilistic joint over spatial bounding box b and intermediate reasoning chain R. The model executes a structured four-step visual chain-of-thought: (1) hand detection, (2) fingertip keypoint extraction, (3) pointing ray geometry construction, and (4) target bounding box alignment. Each step is regulated by specialized tokens and visual attention anchoring, ensuring deterministic and physically interpretable spatial parsing.
Two-Stage Training Paradigm
The model employs a two-stage training protocol:
- Supervised Fine-Tuning ("cold-start"): Using the EgoPoint-CoT dataset, the model is fine-tuned with Low-Rank Adaptation, internalizing geometric priors and syntactic reasoning patterns. Explicit intermediate supervision tightly constrains the generation space, injecting structured knowledge for sequential reasoning.
- Reinforcement Learning with Adaptive Importance Weighting: Policy optimization is conducted via Group-Relative Policy Optimization (GRPO), with rewards encompassing localization accuracy, intermediate reasoning correctness, and output format validity. A group-variance-based importance weighting module dynamically recalibrates sample contributions by assessing intra-group reward variance, attenuating gradient noise from suboptimal rollouts and stabilizing convergence.
Reward Design and Adaptive Strategy
The composite reward Rbase comprises IoU metrics for both hand and object boxes, angular consistency for the pointing ray, and scale-normalized keypoint accuracy. Format rewards verify valid reasoning output structure and penalize repeated or hallucinated reasoning traces. The adaptive weighting utilises the square root of the group variance relative to a global running baseline, emphasizing groups with pronounced heterogeneity—thus yielding more informative gradient updates.
Dataset: EgoPoint-CoT
EgoPoint-CoT is a high-resolution, egocentric image dataset specifically annotated for chain-of-thought spatial reasoning. It contains ~15K images annotated with hand and object bounding boxes, fingertip coordinates, pointing rays, and logical reasoning descriptions. Semantic diversity is ensured through 345 object categories and strongly variable scene complexity (mean object count ~8.8/image). Negative samples and blocking trajectories are injected to suppress model hallucinations. Expert annotation reliability is validated by Fleiss’ K = 0.91.
Empirical Results
Main Results
PointVG-R achieves state-of-the-art performance on pointing localization, outperforming SOTA MLLMs by 15.86 points in mean IoU (mIoU). Direct comparisons show traditional baselines achieve mIoU in the 0.42–0.45 range (e.g., Qwen2.5-VL, LLaVA-OneVision), while PointVG-R reaches 0.7570 with explicit V-CoT and adaptive GRPO. Models with intermediate supervision see significant gains over standard fine-tuning protocols, especially on backbones with higher reasoning capacity.
Ablation Studies
- Intermediate V-CoT Supervision: Progressive ablation demonstrates that explicit hand localization and ray/keypoint supervision stabilize grounding and improve performance across stricter metrics (e.g., [email protected], mIoU). Decomposition of the task into geometric subproblems is more effective than optimizing only the final box.
- Importance Weighting Statistics: Group variance statistically outperforms entropy, top-bottom gap, and standard deviation in reward-based scaling, yielding consistent mIoU improvements (>3.5 points over alternatives).
- Scaling Functions: Square-root mapping achieves superior stability and sensitivity compared to linear/power/logarithmic alternatives, avoiding over-amplification or compression of inter-group differences.
Qualitative Analysis
PointVG-R demonstrates robust spatial alignment in challenging scenarios where baseline models misinterpret gestures, default to saliency-driven predictions, or fail in cluttered environments. Explicit modeling of hand-object spatial relationships enables precise target disambiguation, as observed in visually ambiguous real-world cases.
Implications and Future Directions
PointVG-R exemplifies a paradigm shift toward "thinking with images" in visual grounding tasks, demonstrating that structured, interpretable spatial reasoning chains materially improve gesture-based localization. The autoregressive nature of V-CoT reasoning introduces non-negligible inference latency, posing challenges for real-time deployment on edge devices (e.g., AR wearables). Future research will be directed toward optimizing reasoning chain complexity, token overhead reduction, and acceleration of inference under sequential logic constraints.
Practically, PointVG-R's explicit reasoning framework may enhance multimodal dialogue systems, human–robot interaction, and assistive vision technologies by providing stable, interpretable spatial understanding. Theoretically, reinforcement learning with group-level reward statistics could generalize to other multimodal tasks requiring fine-grained iterative reasoning and robust sample prioritization.
Conclusion
PointVG-R operationalizes geometric reasoning as a chain-of-thought in multimodal LLMs, bridging the gap between visual perception and logical spatial inference for pointing-based grounding. Through structured intermediate supervision and adaptive RL optimization, it achieves superior precision and robustness, validated by both quantitative and qualitative analyses. The approach represents a robust foundation for future advances in human-centric spatial intelligence and interpretable multimodal reasoning (2606.24539).
