Reference-View Ray Voting

• The paper introduces reference-view ray voting as a semantic 3D lifting method to recover manipulation-relevant keypoints when direct triangulation consensus is weak.
• It complements multi-view triangulation by sampling candidate 3D points along a reference camera ray and selecting the candidate with the best cross-view semantic consistency via VLM.
• Empirical results show improved 3D grounding accuracy and manipulation reliability in cluttered, occluded, and ambiguous scenes.

Reference-view ray voting is a geometric-semantic lifting procedure for recovering manipulation-relevant 3D keypoints from calibrated multi-view RGB images in a zero-shot dexterous manipulation pipeline. In "Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning" (Kim et al., 17 Jun 2026), it appears as a fallback and complement to direct multi-view triangulation when view-wise VLM groundings are inconsistent, weak, or partially occluded. Rather than training an end-to-end policy, the system uses a VLM to generate reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D through multi-view fusion; reference-view ray voting is the component that searches along a semantic camera ray for a 3D point that is semantically and geometrically consistent across neighboring views (Kim et al., 17 Jun 2026).

1. Problem setting and motivation

The method addresses 3D grounding from only calibrated RGB views, without task-specific training. The required 3D targets include grasp points, placement targets, tool contact points, tool terminal poses, and waypoints. The paper states that, for dexterous manipulation, small localization mistakes can cause unstable grasps, collisions, wrong tool contacts, or IK failure (Kim et al., 17 Jun 2026).

A single RGB view is often insufficient because of occlusion, ambiguity, and depth uncertainty. The larger framework therefore uses a multi-view VLM-based grounding pipeline that lifts 2D semantic predictions into 3D. Reference-view ray voting is introduced specifically for cases in which direct multi-view triangulation is unreliable or weak, especially under partial visibility or inconsistent 2D predictions across views.

Its role is therefore not generic depth estimation. It is a task-conditioned 3D grounding mechanism tied to semantic entities produced by the VLM, such as a grasp point on an object or a functional point on a tool. This suggests that its value lies in preserving semantic correctness when purely geometric consensus is insufficient.

2. Relationship to triangulation within multi-view fusion

The paper uses two complementary 3D lifting mechanisms: triangulation of view-wise VLM groundings and reference-view ray voting (Kim et al., 17 Jun 2026).

Triangulation is described as a RANSAC-style procedure. The VLM is queried across all calibrated views for the same semantic keypoint, and a candidate 3D point is selected by inlier support. This mechanism is best when the VLM produces consistent 2D keypoints across enough views.

Reference-view ray voting is used when the 2D predictions vary too much or triangulation consensus is poor. Instead of relying only on multi-view 2D point agreement, it anchors inference to one reference image and searches along the camera ray defined by the reference-view grounding. Candidate 3D points on that ray are projected into other views, and the VLM is asked which candidate index best matches the semantic description.

A common misconception is to treat ray voting as a replacement for triangulation. The paper states the opposite: they are “complementary parts of the same multi-view fusion approach.” Triangulation is mainly a geometry-first consensus estimator, whereas ray voting is a ray-search semantic consistency estimator. The final 3D point is chosen by triangulation if triangulation has enough support; otherwise the method falls back to ray voting.

Mechanism	Primary basis	Selection condition
Triangulation	Geometry-first consensus estimator	Used if triangulation has enough support
Reference-view ray voting	Ray-search semantic consistency estimator	Used when triangulation consensus is poor

3. Procedure of reference-view ray voting

The procedure begins with reference-frame grounding. Given calibrated multi-view images $I=\{I_v\}_{v=1}^M$ and instruction $l$, the VLM selects a reference view $r$ and predicts the manipulation mode and grounding tuple:

$(r, z, g_z) = \mathcal{P}(I, l). \tag{1}$

A planning prompt is then built, and the VLM outputs primitive-level 2D keypoints and descriptions (Kim et al., 17 Jun 2026).

For each primitive keypoint $p_{t,j}$, a prompt $l''$ is constructed from the language instruction, primitive type, and keypoint description $d_{t,j}$. The VLM predicts view-wise 2D keypoints across all views:

$$p_{t,j}^{(v)} = \mathcal{P}(I_v, l''), \quad v=1,\dots,M. \tag{3}$$

Reference-view ray voting then samples candidate 3D points along the camera ray associated with the reference-view 2D grounding. The paper describes uniform sampling over depth range $[0.5, 2.0]$ m with step size $0.05$ m. If the reference-view pixel defines a ray, each sampled depth yields one candidate:

$l$0

Each sampled 3D candidate is projected into each non-reference view $l$1, and the image is overlaid with numbered markers. The VLM is then queried on the modified image and semantic description, selecting candidate indices

$l$2

Vote counts are accumulated over the selected indices:

$l$3

and the ray-voting estimate is

$l$4

This design uses the VLM not merely as a detector of corresponding pixels, but as a selector of the depth candidate whose cross-view projections best match the intended semantic entity.

4. Mathematical formulation and decision rule

The triangulation branch is scored by inlier support. For each view pair $l$5, a candidate 3D point is triangulated and evaluated by

$l$6

The best triangulated candidate is selected by maximum consensus (Kim et al., 17 Jun 2026).

The final choice between the triangulated result and the ray-voting result is governed by

$l$7

The supplementary gives implementation values: reprojection inlier threshold $l$8 pixels, triangulation acceptance threshold $l$9, and ray sampling over depth $r$0 m in steps of $r$1 m.

The decision rule clarifies the method’s status within the system. Ray voting is not continuously fused with triangulation in a weighted estimator; it is invoked when triangulation does not meet the required consensus threshold. This suggests a modular architecture in which semantic consistency is used to recover from geometric failure modes.

5. Dependence on calibrated multi-view RGB and semantic grounding

The entire pipeline starts from calibrated multi-view RGB. Calibration is necessary because triangulation requires known extrinsics and intrinsics, ray sampling requires the reference camera ray in 3D, projection into neighboring views requires accurate camera geometry, and final 3D keypoints are transformed into the world frame using camera extrinsics (Kim et al., 17 Jun 2026).

The VLM contributes two distinct outputs. First, it performs reference-frame grounding by selecting the reference view and inferring the high-level manipulation mode and task grounding tuple. Second, it provides semantic grounding for each primitive by generating primitive-level 2D keypoints and associated descriptions. The paper emphasizes that these descriptions are crucial: ray voting does not merely ask whether projected candidates land at the same image coordinates, but whether they correspond semantically to entities such as “broom bristle tip,” “pot handle,” “target object body,” or “placement location.”

Accordingly, reference-view ray voting can be understood as a semantic search constrained to a single calibrated camera ray. Geometry defines the hypothesis set; VLM reasoning selects the hypothesis that is most semantically consistent across views.

6. Downstream role in dexterous manipulation and tool-use

Once the 3D keypoints $r$2 are recovered, they drive the rest of the manipulation pipeline (Kim et al., 17 Jun 2026).

For pick-and-place, the lifted keypoints include a grasp point, a waypoint, and a release or location point. The grasp point becomes the pick target, the waypoint supports collision-aware transport, and the release point becomes the placement target. The robot motion generator then constructs a feasible transfer path from the current pose to the lifted 3D goal.

For tool-use, the lifted keypoints include a grasp point on the tool, a functional tip or apply-action point, a target interaction point, and a terminal hold or release pose. These are used to align a stored object-centric atomic action from a Bag of Atomic Actions (BoAA). The library stores a tool-use skill category and a 6D object trajectory,

$r$3

The current scene’s lifted keypoints $r$4 and $r$5 determine a rigid alignment $r$6, and the stored trajectory is transformed as

$r$7

The lifted grasp point is then expanded into a task-conditioned grasp affordance region. The paper describes a multi-view inclusion score over object mesh vertices,

$r$8

with the retained region

$r$9

That affordance region is used to generate grasp candidates, and the resulting grasp candidates and 6D object trajectory are passed to an arm-hand motion generator.

Reference-view ray voting is therefore upstream of grasp synthesis, tool alignment, placement, and motion generation. A plausible implication is that its practical importance derives less from the isolated 3D point estimate than from the sensitivity of all subsequent modules to grounding error.

7. Empirical findings and interpretation

The paper reports that multi-view grounding, including reference-view ray voting, improves both 3D accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines (Kim et al., 17 Jun 2026). In Table 3, compared with the stereo RGB-D baseline, the reported 3D grounding errors are:

Method	$(r, z, g_z) = \mathcal{P}(I, l). \tag{1}$0	$(r, z, g_z) = \mathcal{P}(I, l). \tag{1}$1
Stereo (RGB-D)	16.43 cm	2.72 cm
Ours (2 views)	4.58 cm	1.70 cm
Ours (3 views)	4.60 cm	1.35 cm
Ours (5 views)	4.77 cm	1.94 cm

These results show a strong improvement over single-view stereo grounding, especially for grasp localization. The paper notes diminishing returns from additional views in its setup because wide-baseline pairs already provide strong constraints.

The real-robot results in Table 1 show that the system outperforms or matches the RGB-D baseline on several tasks, with the benefit strongest in cluttered scenes and precise placement tasks. One reported example is “Cluttered Precise Pick-and-Place,” where RGB-D achieves 2/5 and the proposed system achieves 4/5. Table 2 further shows the zero-shot system succeeding where fine-tuned VLA baselines fail completely on the reported tasks.

The paper also states qualitatively that multi-view fusion “significantly reduces grasp localization error and improves apply-action localization,” and that the system is “more reliable under occlusion and viewpoint ambiguity.” While Table 3 is not a direct ablation of ray voting alone, it also shows that adding the refinement stage further improves penetration error. This suggests that the reported performance reflects the interaction of improved 3D grounding from triangulation and ray voting with local collision-aware correction.

Overall, reference-view ray voting is presented as a semantic multi-view 3D lifting method that samples candidate 3D points along a reference camera ray and uses VLM-based cross-view consistency to select the best candidate. Within the full pipeline, it functions as the recovery path when triangulation consensus is insufficient, thereby improving the robustness of zero-shot long-horizon dexterous manipulation in cluttered, ambiguous, and partially occluded scenes (Kim et al., 17 Jun 2026).