VistaRef: Gesture-Based Object Detection

• VistaRef is a framework for deictic gesture grounding that predicts the bounding box of an object indicated by a pointing finger in RGB images.
• It leverages Local Hand Entity Modeling, Geometric Ray Modeling, and an Orientation-Consistent Alignment Loss to explicitly capture hand geometry and spatial orientation.
• Evaluation on the EgoPoint-Ground dataset shows a 14-point accuracy gain over baselines, highlighting its promise for AR, human-robot collaboration, and gesture-based interfaces.

VistaRef is a framework for pointing-to-object detection, or deictic gesture grounding, in natural images. Given a single RGB image and the fixed query “Please frame the bbox where a finger points to the item,” it predicts the bounding box of the target object indicated by a pointing finger. The method is designed to strengthen what it calls visual spatial orientation awareness: rather than treating pointing as ordinary visual grounding, it makes the hand root, fingertip, and implied pointing ray explicit within the model’s representation, feature aggregation, and supervision. Its core components are Local Hand Entity Modeling (LHEM), Geometric Ray Modeling (GRM), and the Orientation-Consistent Alignment Loss (OCAL), and the reported evaluation shows a 14-point absolute gain in grounding accuracy over the baseline (Li et al., 23 Jun 2026).

1. Task definition and motivation

VistaRef addresses a specialized visual grounding problem in which the linguistic input is fixed and the ambiguity lies almost entirely in spatial orientation. The system receives an RGB image and the query

$$\text{“Please frame the bbox where a finger points to the item.”}$$

and must output a normalized bounding box

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$

for the pointed object. The annotation format includes hand bounding boxes, two keypoints per pointing hand—root and fingertip—object bounding boxes for all visible objects, and a binary is_positive label indicating whether a valid pointing target exists.

The paper argues that standard Transformer-based visual grounding models are strong at semantic matching and coarse spatial relations, but weaker at micro-geometric orientation. In pointing tasks, the relevant constraint is not merely object identity or coarse location; it is the directed relation from hand root to fingertip and then onward to the target. The reported failure modes are pointing drift and localization ambiguity, especially when targets are distant, densely packed, or surrounded by semantically salient distractors. This indicates that deictic grounding is not well approximated by generic referring expression grounding alone (Li et al., 23 Jun 2026).

Within this formulation, visual spatial orientation awareness means that the model should represent a directed geometric relation from hand root to fingertip and from the hand to the target object. VistaRef operationalizes that requirement by constructing an explicit ray representation and using it as both a feature and a training signal.

2. Architectural formulation

VistaRef uses a frozen BEiT-3 vision-language Transformer backbone, in Base or Large form. From the backbone it obtains a visual token sequence

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$

a text token sequence $\mathbf{F}_l$, and a text-guided global visual representation $\mathbf{h}_{vl} \in \mathbb{R}^d$.

The first stage is text-guided visual aggregation, which weights visual patches using attention derived from the text-guided representation:

$$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$

This produces a global representation conditioned on the fixed grounding query.

VistaRef then applies four modules in sequence. GRM regresses the hand root and fingertip, constructs an explicit ray descriptor, and maps it into a ray embedding. LHEM uses the predicted keypoints to carve out a dynamic hand-centered region and pool a local hand feature. Ray-aware target grounding fuses the ray embedding and local hand feature into a query vector, cross-attends that query against global visual tokens, and then regresses the final target box.

The final decision feature is

$$\mathbf{F}_{final} = [\mathbf{h}_{vl}; \gamma \cdot \mathbf{e}_{ray}; \mathbf{t}],$$

with $\gamma = 6.0$, and the bounding box is predicted by

$$\mathbf{b} = \sigma(MLP_{bbox}(\mathbf{F}_{final})).$$

The paper positions VistaRef as an extension of OneRef’s one-tower architecture, but with explicit geometric ray modeling, explicit local hand feature extraction, and an orientation-structured loss.

3. Local hand entity modeling and geometric ray modeling

GRM begins with keypoint localization from the global text-guided feature:

$$\{\mathbf{p}_r, \mathbf{p}_t\} = \sigma(MLP_{kp}(\mathbf{h}_{vl})), \qquad \mathbf{p}_r, \mathbf{p}_t \in [0,1]^2,$$

where $$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$0 is the hand root and $$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$1 is the fingertip. From these, VistaRef forms the displacement

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$2

the length

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$3

and the unit direction

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$4

These are concatenated into a 7-dimensional geometric descriptor

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$5

which is then projected into a ray embedding

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$6

LHEM uses the predicted ray to define a hand-focused local region. Its center is

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$7

The region scales with ray length:

$$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$8

with $$\mathbf{b} = [x_{center}, y_{center}, w, h] \in [0,1]^4$$9, $\mathbf{F}_v \in \mathbb{R}^{N \times d},$0, and

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$1

This yields an axis-aligned rectangular hand region $\mathbf{F}_v \in \mathbb{R}^{N \times d},$2.

A binary patch mask is then formed as

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$3

with a fallback to the patch nearest $\mathbf{F}_v \in \mathbb{R}^{N \times d},$4 if the mask is empty. The local hand feature is computed by masked average pooling:

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$5

The local hand feature and ray embedding are fused into a ray-aware hand query:

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$6

This query is used in cross-attention over global visual patches:

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$7

The architectural intent is explicit: the model should not merely attend to globally relevant visual regions, but should aggregate evidence along a hand-conditioned, orientation-aware geometric prior.

4. Losses and optimization

VistaRef uses a two-part objective:

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$8

The base grounding loss is

$\mathbf{F}_v \in \mathbb{R}^{N \times d},$9

where $\mathbf{F}_l$0 is the Generalized IoU loss.

OCAL adds hand-centric and orientation-centric supervision. For positive samples, it is

$\mathbf{F}_l$1

Here, $\mathbf{F}_l$2 is a hand presence classification loss, $\mathbf{F}_l$3 supervises the predicted root and fingertip coordinates, and $\mathbf{F}_l$4 enforces orientation consistency between the predicted ray and the ground-truth finger vector. The paper describes the ray term as a cosine-style alignment between predicted and ground-truth orientation. For negative samples, only $\mathbf{F}_l$5 is active, while the keypoint and ray terms are disabled.

The reported loss weights are

$\mathbf{F}_l$6

and within $\mathbf{F}_l$7,

$\mathbf{F}_l$8

Training proceeds in two stages. A warm-up stage runs for 10 epochs with the backbone frozen, batch size 64, and learning rate $\mathbf{F}_l$9 under a cosine schedule. A subsequent fine-tuning stage runs for 20 epochs with batch size 8 and learning rate $\mathbf{h}_{vl} \in \mathbb{R}^d$0, with box-mask constraints enabled to improve localization. The implementation also uses EMA-based loss normalization with momentum 0.99 and $\mathbf{h}_{vl} \in \mathbb{R}^d$1.

5. Dataset and evaluation protocol

VistaRef is evaluated on EgoPoint-Ground, an egocentric pointing-based visual grounding dataset containing 15,455 images, split into 10,837 training images, 1,546 validation images, and 3,072 test images (Li et al., 23 Jun 2026). The dataset spans 345 object categories and includes indoor and outdoor scenes. Each image is annotated with hand bounding boxes, object bounding boxes for all visible objects, finger root and fingertip keypoints, an is_positive label, and the target bounding box when a valid pointed object exists.

The evaluation uses standard overlap-based grounding metrics. For a predicted box $\mathbf{h}_{vl} \in \mathbb{R}^d$2 and ground-truth box $\mathbf{h}_{vl} \in \mathbb{R}^d$3,

$\mathbf{h}_{vl} \in \mathbb{R}^d$4

Precision at threshold $\mathbf{h}_{vl} \in \mathbb{R}^d$5 is

$\mathbf{h}_{vl} \in \mathbb{R}^d$6

and mean IoU is

$\mathbf{h}_{vl} \in \mathbb{R}^d$7

The paper emphasizes that these metrics expose different aspects of performance: [email protected] reflects coarse localization robustness, [email protected] is the main accuracy measure, [email protected] tests fine localization, and mIoU summarizes overall localization quality.

6. Empirical performance, interpretation, and limitations

The reported results show substantial gains over strong referring and grounding baselines (Li et al., 23 Jun 2026). In the “Train on Hybrid” setting with BEiT-3-L, the best non-VistaRef baseline listed is PropVG, with $\mathbf{h}_{vl} \in \mathbb{R}^d$8 and $\mathbf{h}_{vl} \in \mathbb{R}^d$9, whereas VistaRef reaches $$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$0 and $$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$1. Relative to OneRef with the same backbone, the gain is from 0.7113 to 0.8221 on [email protected] and from 0.6131 to 0.7113 on mIoU. In the “Train-Real Data Test” condition, PropVG achieves $$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$2, while VistaRef reaches 0.8201.

The zero-shot results are also instructive. Among standard referring expression models, the best reported zero-shot [email protected] is 0.2665 for SimVG with BEiT-3-L. This indicates that strong generic grounding models do not transfer well to pointing-based grounding without explicit supervision for hand geometry and directionality.

Ablation results separate the effects of the proposed modules. Starting from the OneRef baseline at mIoU 0.5939, GRM alone yields 0.5754, LHEM alone yields 0.6839, GRM plus LHEM yields 0.6824, and the full combination with cross-attention reaches 0.6919. This suggests that GRM is most effective when it is not used in isolation but is coupled to local hand modeling and ray-aware attention. The OCAL ablation shows a similar pattern: the baseline without OCAL gives mIoU 0.4611; adding only $$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$3 gives 0.6165, only $$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$4 gives 0.5483, only $$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$5 gives 0.5673, $$\mathbf{h}_{vl} = \sum_{i=1}^{N} \left( \sigma(\mathbf{A}_{attn,i}) \cdot \mathbf{F}_{v,i} \right).$$6 gives 0.6663, and the full OCAL gives 0.6919.

Efficiency overhead is modest relative to the accuracy gains. With BEiT-3-Base, OneRef has 224.82M parameters and runs at 33.28 fps, while VistaRef has 226.01M parameters and runs at 22.92 fps. PropVG is slower at 7.12 fps. The added computation therefore appears concentrated in the hand- and ray-specific modules rather than in a wholesale backbone expansion.

The paper identifies two principal limitations. First, precision remains fragile when the hand region is extremely small, since small keypoint errors induce large angular deviations in the inferred ray. Second, semantic reasoning remains limited when multiple overlapping objects share similar geometry; in such cases, heavy reliance on the ray can still leave ambiguity. A plausible implication is that further gains may require tighter integration of explicit geometric constraints with stronger category-level or part-level semantic disambiguation.

VistaRef is positioned for augmented reality, human-robot collaboration, gesture-based interfaces, and broader embodied reference understanding. Its central contribution is methodological: it shows that Transformer-based grounding can be improved substantially by making hand-centered local evidence and directed geometric structure explicit, rather than leaving deictic orientation to emerge implicitly from global attention alone.