Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration

Abstract: Selecting out-of-reach objects is a fundamental task in mixed reality (MR). Existing methods rely on a single cue or deterministically fuse multiple cues, leading to performance degradation when the dominant cue becomes unreliable. In this work, we introduce a probabilistic cue integration framework that enables flexible combination of multiple user-generated cues for intent inference. Inspired by natural grasping behavior, we instantiate the framework with pointing direction and grasp gestures as a new interaction technique, Point&Grasp. To this end, we collect the Out-of-Reach Grasping (ORG) dataset to train a robust likelihood model of the gestural cue, which captures grasping patterns not present in existing in-reach datasets. User studies demonstrate that our selection method with cue integration not only improves accuracy and speed over single-cue baselines, but also remains practically effective compared to state-of-the-art methods across various sources of ambiguity. The dataset and code are available at https://github.com/drlxj/point-and-grasp.

• The paper introduces a Bayesian multi-cue integration framework that combines directional and grasp cues for selecting out-of-reach objects in mixed reality.
• It leverages a neural model and spatial Gaussian approach to estimate object likelihood, adapting to variable gesture poses and ambiguous settings.
• Empirical evaluations show that the method outperforms single-cue techniques by yielding faster, more accurate, and robust selections in cluttered environments.

Probabilistic Multi-Cue Integration for Out-of-Reach Object Selection in Mixed Reality

Introduction and Problem Motivation

Selecting out-of-reach objects is a critical challenge in mixed reality (MR) systems, where intent inference is hampered by spatial clutter, target occlusion, and semantic similarity between objects. Historically, interaction techniques have relied on either directional cues (e.g., raycasting from hand/controller) or semantic gestures (e.g., grasp postures) as proxies for user intention. While ray-based selection is intuitive, it suffers substantial accuracy degradation in crowded or occluded scenes. Conversely, gesture-based selection can resolve certain ambiguities but introduces its own issues when multiple objects afford similar grasps. Most prior work either uses a single cue or combines cues using rigid, deterministic rules that fail when input reliability shifts—exposing a brittleness in multi-cue frameworks. The paper "Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration" (2604.22491) addresses this by proposing a Bayesian integration framework that probabilistically fuses spatial (directional) and semantic (grasping) cues for robust, context-adaptive object selection.

Figure 1: Overview of the multi-cue inference framework combining endpoint distributions for directional cues with a learned gesture–object likelihood model through Bayesian cue integration.

Probabilistic Cue Integration Architecture

The proposed framework formalizes out-of-reach selection as a posterior inference problem over candidate objects $\mathcal{O}$, conditioned on observed behavioral cues $\mathcal{C}$. Bayesian combination is used, assuming cue independence given the object: $$p(o \mid \mathcal{C}) \propto p(o) \prod_{m} p(c_m \mid o)$$ Directional evidence is modeled as a Gaussian over ray endpoints; grasping evidence is supplied by a neural compatibility model trained on labeled hand-object pairings. Each cue yields an object likelihood that the system fuses to estimate user intent.

• Directional Cue: A spatial Gaussian centered at each object with variance $\sigma_d$, computed based on the offset between the ray endpoint and object center.
• Gestural Cue: A neural model estimates the compatibility between the observed hand pose and candidate objects, extracting distinct features for the hand (joint positions, processed via an MLP) and the object (point cloud, mapped via a BPS encoding and MLP), then fusing these features for classification.

This architectural choice supports robust adaptation to ambiguous settings, allowing information from the more reliable cue to dominate the posterior in contexts where the other is uninformative.

Out-of-Reach Grasping Dataset (ORG) and Cue Modeling

Existing datasets capture only in-reach, contact-rich grasps, which cannot model mid-air gestures performed for distant objects in MR interaction. To ground probabilistic modeling, the authors introduce the Out-of-Reach Grasping (ORG) dataset: 10,260 gesture-object pairs, covering both in-reach and out-of-reach conditions, annotated for compatibility.

Figure 2: Trial procedure for VR data collection, capturing both in-reach and out-of-reach grasping gestures, as well as compatibility labels with alternative objects.

Systematic analysis reveals significant structural differences between in-reach and out-of-reach gestures (see Figure 3): the latter are more extended and variable across users, producing nontrivial distributional shifts that must be handled in modeling.

Figure 3: Per-joint mean L2 distance between in-reach and out-of-reach gestures shows largest deviations at distal joints, with notable across-user variability.

Signed Distance Field analysis further demonstrates out-of-reach gestures are generally less flexed (Figure 4).

Figure 4: Distribution of signed distances reveals more extended postures for out-of-reach gestures.

Critically, confusion matrices built from human compatibility judgments highlight a systematic many-to-many correspondence between gestures and objects—cylindrical grasp objects (cup, mug, wineglass) form highly confusable clusters (Figures 5 and 6).

Figure 5: Object confusion matrix: Some gestures (e.g., for mugs) are ambiguously compatible with other cylindrical objects, revealing many-to-many mapping structure.

Figure 6: Clustering shows confusability is structured; similar affordances (e.g., tool-like or cup-like) form tight clusters.

This motivates the necessity for non-deterministic (probabilistic) modeling: gesture-object compatibility cannot be captured by one-to-one mappings, and a probabilistic score better reflects the true ambiguity present.

Empirical Evaluation

Study 1: Single-Cue vs. Cue Integration

A controlled study compared Point, Grasp, and Point&Grasp in scenes with factorial manipulation of spatial and semantic ambiguity: Point suffers in dense layouts; Grasp fails when objects are semantically similar. The Bayesian integration method consistently achieves the fastest selection times and highest trial completion rates across all ambiguity regimes (see Figure 7).

Figure 8: Four ambiguity regimes, varying spatial and semantic similarity of objects, used in evaluation scenes.

Figure 9: Experimental setup for VR study, collecting both directional and grasping cues via hand tracking.

Figure 7: Point&Grasp maintains robust speed and accuracy under all ambiguity conditions, outperforming both single-cue methods.

Post hoc analysis shows that successful selections were frequently driven by a single dominant cue, with the probabilistic framework flexibly prioritizing the cue with higher discriminative value (Figure 10). Even when both cues were ambiguous, the fused posterior could occasionally "rescue" correct selection.

Figure 10: Distribution of trial outcomes by cue agreement: system adapts to cue informativeness, succeeding most often when one cue is reliable.

Subjective evaluations indicated grasp gestures were easy to learn, consistent with everyday action schemas, and the combined interaction was quickly mastered.

Study 2: State-of-the-Art Comparison

Comparison against BubbleRay (an adaptive raycasting selection technique) and Expand (two-stage refinement) demonstrates the scalability of cue integration. Point&Grasp matches or exceeds selection speed and completion rate of these methods but does so without requiring iterative refinement (Expand) or suffering in dense/ambiguous layouts (BubbleRay). Notably, as semantic ambiguity is reduced (i.e., when grasp cues become highly reliable), Point&Grasp shows pronounced advantage in both speed and trial completion (Figure 11).

Figure 12: Visualization of techniques compared: BubbleRay, Expand, and Point&Grasp, highlighting the difference in interaction feedback and internal evidence sources.

Figure 11: Point&Grasp achieves faster or more robust performance than state-of-the-art baselines, especially as ambiguity shifts between spatial and semantic dimensions.

Implications, Limitations, and Future Directions

Systems Integration: Results demonstrate that probabilistic cue fusion can match or exceed the performance and robustness of high-performing directional methods without the overhead of multi-step refinement. This approach is compatible with real-time deployment as cue likelihood computation is efficient.

Cue Expansion: The Bayesian structure is inherently extensible: other modalities (e.g., gaze, speech, contextual priors) can be incorporated by specifying appropriate likelihood models.

Data Limitations: Although the gesture-object model exhibits generalization (cross-user and partial cross-object), accuracy declines for entirely unseen objects—dataset enrichment is essential for deployment in less-constrained scenarios.

Actionable Extensions: Posterior distributions offer a principled basis for refining ambiguous selections or triggering targeted confirmations, which is critical for maintaining usability as object set size and scene complexity increase.

User Strategy: Analysis reveals users intuitively exploit the cue most informative in context—relying on grasp for clustered targets, direction for well-spaced but semantically similar objects, with the fusion model naturally adapting to these shifts.

Conclusion

This work establishes probabilistic multi-cue integration as a principled approach to robust, flexible out-of-reach object selection in mixed reality. By treating directional and gestural cues as complementary evidence and fusing them via Bayesian inference, the Point&Grasp method consistently delivers superior performance across regimes of spatial and semantic ambiguity, matching or surpassing state-of-the-art interaction techniques. The introduction of the ORG dataset not only enables this modeling but also highlights systematic structural differences between in-reach and out-of-reach gestures, justifying dedicated modeling efforts for MR contexts. Theoretically, this approach paves the way for more general, scalable multimodal interaction pipelines, where intent is inferred as posterior over candidates integrating all available behavioral signals. Practically, such systems promise increased robustness and fluidity in MR object selection and interaction, supporting both novice and power users as interface complexity grows.

Reference:

"Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration" (2604.22491)