VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection

Abstract: Grounding deictic gestures in natural images is fundamental to AR and human-robot collaboration, providing a basis for seamless spatial interaction. While Transformer-based visual models have achieved significant progress in general object detection, their global attention mechanisms often neglect micro-geometric relationships, degrading orientation accuracy. In pointing tasks, this deficiency manifests as an inability to accurately capture the pointing ray implied by finger poses, which results in pointing drift and localization ambiguity when dealing with distant or densely packed objects. To address this, we propose VistaRef, a framework designed to explicitly enhance spatial orientation awareness. First, we develop the Local Hand Entity Modeling (LHEM) module, which incorporates hand-pose embeddings to strengthen the model's capability to capture subtle finger deviations. Second, drawing inspiration from multi-view geometry, we construct the Geometric Ray Modeling (GRM) module to transform implicit orientation information into explicit spatial geometric features, guiding feature aggregation and deep fusion via attention mechanisms. Furthermore, we introduce a novel Orientation-Consistent Alignment Loss (OCAL) to synergistically supervise hand presence and pointing consistency, ensuring that all architectural improvements collectively serve the core objective of spatial localization. Experimental results demonstrate that VistaRef significantly outperforms the baseline, achieving a 14-point absolute gain in grounding accuracy. Qualitative analysis further confirms that VistaRef effectively models the geometric correlation from hand to target, bridging the spatial perception gap inherent in traditional Transformers for complex scenarios. Code: https://github.com/lingli1724/VistaRef.

• The paper introduces VistaRef, a modular framework that overcomes deictic gesture grounding limitations by modeling explicit spatial orientations via hand and ray modules.
• It leverages hierarchical modules—TGV, LHEM, GRM, and OCAL—to achieve robust hand-pose perception and precise target localization.
• Empirical results demonstrate significant Precision and mIoU improvements over baselines, validating explicit geometric supervision in challenging environments.

VistaRef: Enhancing Spatial Orientation Awareness for Pointing-to-Object Visual Grounding

Motivation and Problem Formulation

The task of grounding deictic gestures, especially hand pointing, in natural images is critical for AR, human-robot interaction, and embodied multimodal interfaces. Traditional Transformer-based visual grounding models achieve high semantic accuracy on general object detection but manifest systematic deficiencies in spatial orientation sensitivity—specifically, the inability to robustly resolve the geometric pointing ray implied by finger poses. This leads to severe localization drift and ambiguity in scenes with densely packed or visually similar objects, compromising interaction accuracy.

VistaRef directly addresses these shortcomings by introducing modules that transform implicit geometric priors into explicit feature constraints, enabling robust spatial disambiguation and deterministic ray alignment. The framework is designed to achieve two fundamental objectives: precise hand pose perception and robust geometric modeling from hand to target, thereby enhancing spatial reasoning beyond the capacity of global attention mechanisms.

Figure 1: Examples of pointing-to-object detection with subtle directional variations, highlighting the necessity for fine-grained spatial orientation awareness.

Technical Architecture

VistaRef's hierarchical modeling pipeline explicitly mirrors the physical progression of pointing gestures—hand reference points, pointing vector construction, and target projection. The core modules include:

• Text-Guided Visual Aggregation (TGV): Utilizes a frozen BEiT-3 backbone to encode joint vision-language representations, providing global semantic context conditioned by cross-modal affinity matrices.
• Local Hand Entity Modeling (LHEM): Extracts hand-pose embeddings and dynamically proposes local regions $\Omega_{\text{hand}}$, enabling robust and adaptive gesture feature sampling.
• Geometric Ray Modeling (GRM): Performs explicit keypoint localization for hand root and fingertip, derives geometric primitives (orientation and scale), and projects them via an MLP into high-dimensional ray embeddings.
• Ray-Aware Target Grounding: Conducts cross-attention between synthesized gestural queries and global visual features, facilitating spatially consistent and direction-aware target localization.
• Orientation-Consistent Alignment Loss (OCAL): Introduces multi-task geometric supervision, including hand entity, keypoint, and ray orientation losses, ensuring physically consistent spatial mappings and conditional gating based on gesture presence.

  Figure 2: Overview of the VistaRef framework, illustrating the cascade from text-guided aggregation, GRM/LHEM modules, to ray-aware grounding.

  

  Figure 3: Architectural details of Geometric Ray Modeling and Local Hand Entity Modeling.

Empirical Results

VistaRef is evaluated on the EgoPoint-Ground benchmark, designed for gesture-driven referent localization in egocentric vision. Experiments address zero-shot, hybrid (linguistic + gesture), and real-world training protocols. Key quantitative results include:

• Hybrid Data Performance: VistaRef (BEiT-3-L) achieves [email protected] of 82.21% and mIoU of 71.13%—an improvement of 9.99% and 6.36%, respectively, over the strongest prior baselines.
• Real-World Test: VistaRef attains mIoU gains of 4.41% (Base) and 10.88% (Large) compared to hybrid and maintains >4.7% lead over proposal-driven baselines; [email protected] exceeds 82% for Large variant.
• Zero-shot Transfer: Conventional grounding models attain less than 27% [email protected] when transferred, confirming the inability of vanilla attention mechanisms to interpret deictic signals.
• Model Complexity: VistaRef achieves competitive parameter counts (226M) and throughput (22.92 fps), marking a significant improvement in computational efficiency relative to comparable models.

  Figure 4: Visualization of grounding results illustrating VistaRef’s deterministic ray alignment versus baseline localization drift in cluttered scenarios.

  

  Figure 5: Detailed fingertip localization for robust keypoint-driven geometric modeling.

Ablation and Module Analysis

Extensive ablation studies quantify the contribution of each module and loss term:

• The addition of GRM or LHEM yields ≈9% mIoU improvements each over baseline, synergistically increasing performance when fused via cross-attention.
• OCAL loss components are validated: hand classification is fundamental (Surge from 46.11% to 61.65% mIoU); geometric ray and keypoint regression further elevate performance, with full OCAL achieving optimal 69.19% mIoU.

Qualitative Analysis

VistaRef's attention maps reveal:

• Strong spatial alignment along the geometric ray from hand to target, with progressive attention transition (hand → ray → target).
• Robust suppression of distractors—even in highly salient and cluttered environments—facilitated by explicit spatial priors.
• Stable directionality and localization performance under challenging (long-range, multi-object) conditions.

Figure 6: Visualization of spatially aligned, direction-aware attention maps, demonstrating robust target focus along pointing rays.

Figure 7: Performance in salient-object scenarios, exhibiting resilience to distractor objects.

Figure 8: Robustness to visual interference in cluttered multi-object settings.

Theoretical and Practical Implications

VistaRef establishes the indispensability of geometric priors and hand-centric models in deictic grounding, challenging the orthodoxy of global attention-based grounding methods. By integrating ray modeling and adaptive local sampling, it bridges the gap between physical intent and semantic grounding, yielding direct improvements for spatially articulated multimodal interfaces. Its modular architecture and low parameter sensitivity facilitate deployment in real-time, resource-constrained settings, and its explicit geometric supervision scales to complex environments.

The demonstrated improvements in spatial reasoning suggest a shift toward physically self-consistent, direction-aware models for embodied AI. Future developments may entail integration of semantic reasoning for class-attribute disambiguation and adaptation to extreme scale or occlusion scenarios.

Conclusion

VistaRef significantly advances pointing-to-object detection by explicitly modeling spatial orientation and geometric correspondence. Its hierarchical architecture with LHEM, GRM, and orientation-consistent alignment loss achieves robust spatial localization in cluttered and ambiguous environments, outperforming state-of-the-art baselines both quantitatively and qualitatively. While limitations remain regarding distant targets and overlapping referents, VistaRef provides a scalable foundation for gesture-enabled visual grounding in embodied AI systems, setting a benchmark for spatial orientation awareness in multimodal perception (2606.24498).