RaySt3R (Zero-Shot 3D Object Completion)

Last updated: June 10, 2025

RaySt3R: Zero-Shot 3D Object Completion via Novel View Prediction

The demands of robotics, digital twin reconstruction, and extended reality ° (XR) applications have propelled rapid progress in 3D shape completion °—the task of inferring complete object geometry ° from partial, often single-view, observations. Many established approaches remain limited by their difficulty in maintaining 3D consistency °, high computation requirements, and inability to recover sharp object boundaries. RaySt3R introduces a new methodology, recasting 3D completion as a problem of novel view synthesis and leveraging a transformer-based architecture ° to enable accurate, efficient, and generalizable shape inference from a single RGB-D ° view (Duisterhof et al., 5 Jun 2025 ° ).

Background and Motivation

RaySt3R specifically addresses three principal limitations of previous methods:

1. 3D Consistency and Sharpness: Many grid-based volumetric and canonical mesh approaches generate overly smoothed shape completions, losing fine object boundaries and structure.
2. Computational Expense: Dense volumetric prediction and diffusion-based models ° impose significant runtime and memory costs.
3. Generalization: Models trained exclusively on synthetic data often perform poorly when exposed to the noise, occlusion, and diversity of real-world sensor scenarios.

RaySt3R’s key innovation is reframing shape completion ° as direct prediction of depth and mask information for arbitrary query rays corresponding to new views. By translating shape completion into novel view synthesis, the method sidesteps cubic-scaling computations and directly enables inference of unseen surfaces, drawing on neural rendering ° advances (Duisterhof et al., 5 Jun 2025 ° ).

Foundational Approach

At the heart of RaySt3R is the use of query rays: rather than predicting occupancy or distance values within a voxel grid ° or canonical coordinate system, the model accepts:

• A single, masked RGB-D image ° as input.
• A collection of query rays defining novel viewpoints.

For each query ray (i.e., each pixel of the desired novel view), RaySt3R predicts:

• Depth along the ray.
• Foreground-background mask for the pixel.
• Per-pixel confidence score ° reflecting prediction uncertainty °.

Formally, the input depth map ° $D^{\text{input}}$ is unprojected to a 3D point map $X^{\text{input}}$ using input camera intrinsics ° $K^{\text{input}}$:

$X^{\text{input}}_{i,j} = (K^{\text{input}})^{-1} [i D^{\text{input}}_{i,j},\ j D^{\text{input}}_{i,j},\ D^{\text{input}}_{i,j}]^T$

Query rays for the target view are encoded as:

$R_{i,j} = \left[ \frac{i-c_x}{f_x},\ \frac{j-c_y}{f_y} \right]$

where $(c_x, c_y)$ are the image center coordinates and $(f_x, f_y)$ are focal lengths (Duisterhof et al., 5 Jun 2025 ° ).

Model Architecture

RaySt3R employs a feedforward vision transformer (ViT) architecture, integrating point cloud and image features with ray queries through a hierarchy of attention ° mechanisms (Duisterhof et al., 5 Jun 2025 ° ):

• ViT Backbone (DINOv2 °): Visual features are extracted from the foreground-masked RGB image ° using a frozen DINOv2 ViT, with activations concatenated from multiple intermediate layers ° for rich, multi-scale representation °.
• Point Map / Ray Map Embedding: The unprojected 3D context ° points and the 2D query rays are embedded and independently processed.
• Self-Attention Layers: The context point map and ray map features are separately refined through $L$ layers of self-attention. Non-foreground tokens are replaced with a unique learned embedding.
• Cross-Attention: For each novel view, the ray map embeddings attend to the context point and DINO ° features, fusing appearance, geometry, and positional information ° across perspectives.
• Dense Prediction ° Transformers (DPT ° Heads): Two DPT heads are used atop the fused features: one predicts depth and per-pixel confidence for the query view, the other predicts a binary mask °.
• Multi-view Fusion: Depth and mask predictions ° for many queried views (sampled over a view sphere) are aggregated, suppressing points that are occluded in the input, have low confidence, or are predicted background. The result is a unified, completed 3D shape.

Training Losses:

• Confidence-weighted depth loss ° encourages accuracy where the model is confident, penalizing overconfident errors:

$\mathcal{L}_\text{depth} = \sum_{i,j} M_{i,j}^{\text{gt}} \left( C_{i,j} \left\| d_{i,j} - d_{i,j}^{\text{gt}} \right\|_2 - \alpha \log C_{i,j} \right)$

where $M_{i,j}^{\text{gt}}$ is the ground-truth mask, $d_{i,j}$ and $d_{i,j}^{\text{gt}}$ are predicted and ground-truth depth, and $C_{i,j}$ is the predicted confidence.

• Binary cross-entropy ° mask loss ° applies per pixel.
• Total loss sums the two with a weighting parameter (Duisterhof et al., 5 Jun 2025 ° ).

Methodology: Shape Completion as View Synthesis

RaySt3R operationalizes 3D shape completion in three main stages (Duisterhof et al., 5 Jun 2025 ° ):

1. Novel Depth/Mask Prediction: For each target (query) viewpoint, the model predicts a depth map, per-pixel confidence, and mask, using a single forward pass.
2. Selection and Fusion: Aggregating across many query views (on a sphere or hemisphere around the object), the system retains only those points that pass mask and confidence thresholds ° and are not input-occluded.
3. 3D Completion: The filtered novel-view points are merged, yielding a full 3D shape reconstruction ° that includes both observed and unobserved regions.

This pipeline enables RaySt3R to effectively complete missing structure and recover sharp boundaries without relying on slow volumetric computations or canonical mesh alignment °.

Performance and Empirical Results

Evaluation spans both synthetic and real-world benchmarks (Duisterhof et al., 5 Jun 2025 ° ):

• 3D Chamfer Distance ° (CD): RaySt3R achieves up to 44% lower CD than previous approaches, reflecting both higher accuracy and completeness.
• F1 Score @10mm: Outperforms baselines on all datasets, indicating better precision and recall ° of close matches.
• Runtime: Inference is efficient (<1.2 seconds on RTX 4090 for all query views), significantly faster than diffusion-based methods °.
• Generalization: Despite being trained exclusively on synthetic data (1.1 million scenes, 12 million views), the model performs robustly on real-world datasets including YCB-Video, HOPE, and HomebrewedDB. Performance further improves when synthetic training incorporates mask noise reflecting real-world imperfections.
• Boundary Sharpness: Predicted completions exhibit sharp, mask-informed boundaries, superior to over-smoothed results from volumetric grid approaches.

Ablation studies confirm that each aspect of the architecture—ViT features, cross-attention, confidence prediction, and the query-ray formalism—are essential for optimal results.

Practical Applications

RaySt3R’s structure and completeness lend it to practical applications such as (Duisterhof et al., 5 Jun 2025 ° ):

• Robotics: Grasp planning, safe navigation, and manipulation in cluttered environments, leveraging completed 3D representations ° even from partial object views.
• Digital Twin Reconstruction: Accurate 3D models ° for digital twins, supporting monitoring, simulation, and virtual asset creation.
• Extended Reality (XR): Real-time occlusion handling, object interactions, and collision detection ° benefiting from sharp, actionable object boundaries.

Limitations and Future Directions

• While highly diverse, the synthetic training data may not encompass all real-world sensing artifacts, nor the diversity of challenging materials or lighting.
• Further improvements in generalization could be achieved by integrating real-world data into training.
• Scaling to more expressive transformer architectures ° (e.g., diffusion transformers) and optimizing inference for broader deployment are promising paths.
• Advanced methods for handling extreme occlusion, thin structures, and complex reflectance properties remain areas for development (Duisterhof et al., 5 Jun 2025 ° ).

Speculative Note

Broader trends in the field suggest that RaySt3R’s recasting of shape completion as a novel view synthesis task, enabled by query-ray attention and transformer architectures, may catalyze new research directions at the intersection of neural rendering, robotic perception, and real-time 3D scene understanding ° [citation needed].

Conclusion

RaySt3R introduces a transformer-based, ray-predictive framework for 3D object completion °, offering a principled and empirically validated solution that delivers sharp boundaries, high completion accuracy, and strong real-world generalization—all with efficient inference and without reliance on real data for training. These properties make it practical for robotics, digital twin reconstruction, and XR applications, and mark it as a new state-of-the-art for single-view shape completion (Duisterhof et al., 5 Jun 2025 ° ).