RayletDF: Raylet-Based 3D Reconstruction

• RayletDF is a neural method that predicts 3D surface points using raylet distance fields from raw point clouds or RGB-derived 3D Gaussians.
• It employs a 3D sparse U-Net for feature extraction, an MLP-based raylet distance predictor, and softmax-weighted multi-raylet blending for robust reconstruction.
• Extensive evaluations demonstrate that RayletDF outperforms coordinate-based approaches in cross-domain settings while maintaining competitive computational efficiency.

RayletDF is a generalizable method for 3D surface reconstruction from raw point clouds or pre-estimated 3D Gaussians (e.g., from 3DGS) derived from RGB images. In contrast to coordinate-based methods that are computationally intensive during explicit surface rendering, RayletDF introduces the raylet distance field—a direct technique for predicting surface points from query rays by leveraging localized geometric reasoning within a neural architecture comprising a raylet feature extractor, a raylet distance field predictor, and a multi-raylet blender. This method attains superior surface reconstruction performance and notable cross-domain generalization, reconstructing 3D surfaces in a single forward pass on unseen test data (Wei et al., 13 Aug 2025).

1. Mathematical Definition of Raylet Distance Field

A raylet $l$ is parameterized by a starting point $\mathbf{p}_l\in\mathbb{R}^3$ and a unit direction vector $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$, forming a 6D vector $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$. The scene input is either a point cloud $\mathbf{P}\in\mathbb{R}^{N\times C_0}$ or $N$ Gaussians structured similarly.

The raylet distance field is a map: $$f:\left(\mathbb{R}^{N\times C_0}\times \mathbb{R}^6\right)\;\longrightarrow\;\mathbb{R}$$ which predicts a signed distance $d_l = f(\mathbf{P},l)$ from $\mathbf{p}_l$ along $\mathbf{u}_l$ to the scene surface. Additionally, RayletDF jointly predicts a confidence score $\mathbf{p}_l\in\mathbb{R}^3$0: $\mathbf{p}_l\in\mathbb{R}^3$1 with $\mathbf{p}_l\in\mathbb{R}^3$2 denoting concatenation and $\mathbf{p}_l\in\mathbb{R}^3$3 the local raylet feature.

After $\mathbf{p}_l\in\mathbb{R}^3$4 raylet predictions $\mathbf{p}_l\in\mathbb{R}^3$5 for a single camera ray $\mathbf{p}_l\in\mathbb{R}^3$6: $\mathbf{p}_l\in\mathbb{R}^3$7

2. Neural Architecture and Operational Modules

RayletDF comprises three primary modules:

• Raylet Feature Extractor: Utilizes a 3D sparse U-Net backbone (8 encoder and 8 decoder layers) on the input scene, generating per-point features $\mathbf{p}_l\in\mathbb{R}^3$8 with $\mathbf{p}_l\in\mathbb{R}^3$9. For each raylet $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$0, the $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$1 nearest neighbors $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$2 are identified. Feature vectors are enriched and concatenated:

$\mathbf{u}_l\in S^2\subset\mathbb{R}^3$3

Assembled as

$\mathbf{u}_l\in S^2\subset\mathbb{R}^3$4

yielding a feature of dimension $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$5.

• Raylet Distance Field Predictor: Consumes $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$6 (size $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$7) and processes it through an MLP (1 linear+ReLU to 256 dims, 8×[256→256+ReLU], final linear $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$8 2 outputs $\mathbf{u}_l\in S^2\subset\mathbb{R}^3$9). No nonlinear activation is applied to $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$0; $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$1 is unbounded until softmax.
• Multi-Raylet Blender: For each camera ray, the $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$2 closest surface hypotheses (via intersection with virtual balls or Gaussians) yield start points $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$3. The final depth $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$4 aggregates the $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$5 raylet predictions as above with softmax weights $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$6.

3. Multi-Raylet Aggregation and Outlier Suppression

For each camera ray $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$7, $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$8 surface estimates $l=(\mathbf{p}_l,\mathbf{u}_l)\in\mathbb{R}^6$9 are computed. A softmax over the predicted confidences $\mathbf{P}\in\mathbb{R}^{N\times C_0}$0 assigns normalized weights, fusing the outputs: $\mathbf{P}\in\mathbb{R}^{N\times C_0}$1 This weighted voting mechanism attenuates the influence of outlier raylets, enhancing both robustness and generalization capability by facilitating context-aware surface estimation from multiple local viewpoints.

4. Supervision and Training Methodology

The principal loss function operates on rendered per-ray camera depths: $\mathbf{P}\in\mathbb{R}^{N\times C_0}$2 where $\mathbf{P}\in\mathbb{R}^{N\times C_0}$3 indexes valid rays and $\mathbf{P}\in\mathbb{R}^{N\times C_0}$4 is ground-truth from reference depth maps. No explicit regularization beyond standard weight decay is applied, and confidence scores $\mathbf{P}\in\mathbb{R}^{N\times C_0}$5 are optimized solely for $\mathbf{P}\in\mathbb{R}^{N\times C_0}$6. Optional (not used in main results) normal consistency can be enforced via differentiation of the predicted surface with respect to ray direction.

5. Empirical Performance and Comparative Evaluation

RayletDF is validated on multiple indoor real-world datasets: ScanNet / ScanNet++ (merged), ARKitScenes (mobile LiDAR), and MultiScan (handheld). Both per-ray and mesh-level metrics are reported, including ADE (mean absolute error), RMSE, Abs-Rel, Sq-Rel, $\mathbf{P}\in\mathbb{R}^{N\times C_0}$7, mesh Accuracy, Completion, Chamfer-$\mathbf{P}\in\mathbb{R}^{N\times C_0}$8, Normal Consistency, and F-score@5 cm.

Quantitative Results (Mean ADE in meters)

Setting	RayDF	RayletDF (proposed)
3D Gaussians (ScanNet++→ARKit)	0.183	0.115
ScanNet++→ScanNet++	0.202	0.145
Cross-domain (→MultiScan)	0.326	0.216

Additional mesh-level comparison on ARKitScenes→ScanNet++:

• PGSR TSDF F-score = 0.532
• RayletDF TSDF F-score = 0.566

Ablative studies reveal that omitting the confidence branch $\mathbf{P}\in\mathbb{R}^{N\times C_0}$9 increases ADE from 0.145 to 0.170, and using a single raylet ($N$0) degrades ADE to 0.174. Neighborhood size $N$1 yields optimal performance at $N$2 with marginal sensitivity. Incorporation of a Transformer aggregator does not yield empirical improvement over the MLP+concatenation scheme in the evaluated data regime.

6. Computational Efficiency and Generalization

Rendering speed benchmarks (for 640x480 input) report RayletDF at $N$3 FPS with $N$4, $N$5, peaking at $N$6 GB memory. Reference frame rates for other methods: NeRF $N$7 FPS, InstantNGP $N$8 FPS, 3DGS $N$9 FPS (for photometric rendering, less detailed surfaces).

Training the MLP+U-Net on ScanNet++ requires $$f:\left(\mathbb{R}^{N\times C_0}\times \mathbb{R}^6\right)\;\longrightarrow\;\mathbb{R}$$0 hours, with $$f:\left(\mathbb{R}^{N\times C_0}\times \mathbb{R}^6\right)\;\longrightarrow\;\mathbb{R}$$1 hours for precomputing 3D Gaussians when used. RayletDF does not require per-scene optimization; a single forward pass suffices for full-scene reconstruction. Out-of-domain generalization is evidenced by ADE $$f:\left(\mathbb{R}^{N\times C_0}\times \mathbb{R}^6\right)\;\longrightarrow\;\mathbb{R}$$2 m on ARKit-trained models evaluated on ScanNet, outperforming coordinate-based networks which typically overfit scene-specific content.

7. Strengths, Limitations, and Prospective Directions

RayletDF is characterized by strong generalization to unseen scenes, efficiency due to the limited number of raylets and neighbors per query, and the explicit output of surface intersections and normals. Limitations include lower rendering speed compared to point-splatting photometric approaches (e.g., 3DGS), nontrivial KNN and raylet sampling overhead, and demonstrated results restricted to static indoor environments.

Future avenues include incorporation of learned normals or geometric priors during training, replacement of the MLP aggregator with lightweight point-transformers for greater data efficiency, hierarchical/adaptive raylet sampling to further optimize $$f:\left(\mathbb{R}^{N\times C_0}\times \mathbb{R}^6\right)\;\longrightarrow\;\mathbb{R}$$3 and $$f:\left(\mathbb{R}^{N\times C_0}\times \mathbb{R}^6\right)\;\longrightarrow\;\mathbb{R}$$4, extension to dynamic or outdoor scenes, and joint photometric-geometric (RGB+D) supervision.

RayletDF advances surface-centric 3D reconstruction via localized, raylet-based field definitions and efficient, confidence-driven aggregation, yielding high-precision and cross-domain robustness when reconstructing from point clouds or Gaussian splats (Wei et al., 13 Aug 2025).