Gaussian-Voxel Duet: A Dual-Scaffolding Hybrid Representation for Fast and Accurate Monocular Surface Reconstruction

Abstract: While 3D Gaussian Splatting has achieved remarkable success in photorealistic novel view synthesis, its pursuit of fast and high-fidelity 3D reconstruction has long been constrained by a trade-off between geometric accuracy and optimization efficiency. Methods specialized in image rendering converge quickly at the cost of imperfect geometry caused by superfluous primitives overfitting training views, while methods integrating neural signed-distance field (SDF) for better geometry incur prohibitive training costs. In this paper, we attempt to strike a better trade-off by tethering scaffold-anchored Gaussians to a jointly optimized sparse voxel scaffold. This hybrid Gaussian-Voxel representation explicitly confines anchored Gaussians to a narrow band around surfaces defined by voxelized SDFs, which effectively improves representation efficiency and condenses floating Gaussians without sacrificing geometry quality. An implicit surface tethering loss further pulls individual Gaussian primitives closer to SDF-induced surfaces in a mutually regularized manner for improved reconstruction accuracy. Extensive experiments on diverse real-world indoor scenes from ScanNet++, ScanNetv2, and DeepBlending datasets demonstrate that our method achieves state-of-the-art surface reconstruction quality as well as superior novel view synthesis against leading baselines, while maintaining fast training convergence and real-time rendering. Code will be available at https://github.com/duzh11/VoxelGS.

• The paper introduces a dual-scaffold approach combining 2D Gaussian surfels and sparse SDF voxels to enhance both photometric quality and geometric accuracy.
• The paper employs explicit and implicit tethering mechanisms to keep Gaussian primitives near the surface, effectively eliminating floating artifacts.
• The paper achieves state-of-the-art reconstruction on indoor datasets with a 9× speed improvement compared to global SDF-guided methods.

Gaussian-Voxel Duet: Dual-Scaffolding Hybrid Representation for Fast and Accurate Monocular Surface Reconstruction

Introduction

Monocular multi-view 3D surface reconstruction remains a persistent challenge, particularly when reconciling high-fidelity geometry with rendering efficiency. While recent point-based approaches like 3D Gaussian Splatting (3DGS) achieve fast, realistic novel view synthesis, their geometric reconstructions often suffer from floating artifacts and imprecise surfaces owing to the shape-radiance ambiguity under monocular supervision. Extensions that regularize Gaussians with neural Signed Distance Fields (SDF) improve surface accuracy but introduce substantial computational cost due to dense, global SDF optimization.

"Gaussian-Voxel Duet: A Dual-Scaffolding Hybrid Representation for Fast and Accurate Monocular Surface Reconstruction" (2605.26616) addresses this trade-off. The paper proposes a hybrid dual-scaffold approach that synergistically combines an anchored 2D Gaussian surfel scaffold for appearance with a sparse local SDF voxel scaffold for geometry, coupled via explicit and implicit tethering mechanisms. This design restricts Gaussian primitives to a surface-proximal band defined by the SDF, eliminating outliers without compromising photometric quality. The method consistently achieves state-of-the-art geometric and photometric performance on diverse real-world indoor datasets and demonstrates substantial efficiency gains over prior SDF-guided approaches.

Methodology

The core framework constructs a dual representation of the scene:

• Anchor Scaffold: A set of 2D Gaussian surfels, each anchored to SfM-initialized points, managing several Gaussian attributes conditioned on viewing parameters. This provides compact, locally planar primitives for efficient, coherent appearance modeling.
• Voxel Scaffold: A sparse grid of differentiable voxels localized near the scene surface, each encoding a local SDF and color. Initialization via TSDF fusion of Gaussian-rendered depths concentrates voxels around putative geometry, enabling rapid convergence via local residual SDF updates.

  Figure 1: The end-to-end dual scaffolding pipeline for building the hybrid anchor (2D Gaussian surfel) and voxel (local SDF) representations, followed by explicit and implicit tethering.

Mutual Tethering

To tightly couple the scaffolds, the paper introduces two surface-aware regularization mechanisms:

• Explicit Anchor Tethering: Gaussians are pruned or densified according to their SDF value. Only Gaussians near the surface band ($|f_v(\boldsymbol{x})| < \tau_d$) and with strong photometric gradients are kept or subdivided; outliers are pruned, producing a compact, surface-anchored primitive set.
• Implicit Surface Tethering: An $L_2$ loss minimizes each Gaussian's SDF, directly pulling all remaining primitives onto the SDF surface and providing gradient flow to both Gaussians and the voxel SDF. This mutual regularization leads to dense, geometry-consistent Gaussian distributions.

  Figure 2: Quantitative and qualitative evidence of outlier suppression and geometric alignment via tethering mechanisms, with reduced proportions of distant (floating) Gaussians and tight fit to ground-truth geometry in textureless scenes.

Optimization Objective

Anchor scaffold optimization incorporates photometric, depth, and normal priors, with monocular cues used for warm-up and annealed over time. Voxel scaffold optimization includes color, depth, normal, and Eikonal losses. The total objective is the sum of anchor, voxel, and tethering losses, with negligible sensitivity to hyperparameter choices.

Mesh extraction is direct: Marching Cubes applied to the local SDF, yielding watertight meshes. Unlike multi-stage TSDF fusion pipelines used by baselines, this is efficient and robust.

Experimental Results

Comprehensive experiments on ScanNet++, ScanNetv2, and DeepBlending datasets validate the approach.

Surface Reconstruction

Quantitatively, the method outperforms implicit (MonoSDF, Ash), explicit (2DGS, GeoSVR, PGSR), and SDF-hybrid (GSDF, GS-Pull) baselines on all geometric metrics (accuracy, completeness, precision, recall, F-score).

• On ScanNet++: highest F-score (0.842), recall (0.890), and precision (0.805).
• On ScanNetv2: highest F-score (0.785), with strong robustness to noisy initialization and motion blur.

Qualitatively, reconstructions display superior global and local structure, with clean, artifact-free geometry even in textureless and large-scale scenes.

Figure 3: Superior global and local mesh quality on ScanNet++ versus baselines, evidenced by fewer floating Gaussians and preserved smoothness in challenging regions.

Figure 4: Robustness on ScanNetv2, yielding artifact-free meshes despite low-resolution inputs and noisy SfM points.

Further qualitative and quantitative error analysis demonstrates a substantial reduction in proportion and magnitude of floating Gaussians compared to state-of-the-art alternatives.

Figure 5: Scene-wise error map visualizations on ScanNet++, highlighting precise error localization and strong overall geometric accuracy versus baselines.

Figure 6: Robust surface extraction on ScanNetv2, balancing fine detail preservation and surface smoothness, outperforming patch-based and hybrid competitors.

Novel View Synthesis

The method also achieves strong rendering metrics:

• ScanNet++ (view extrapolation): highest PSNR (23.76), consistently outperforms explicit and hybrid baselines on SSIM and LPIPS.
• DeepBlending (view interpolation): competitive with best baselines.

  Figure 7: High-fidelity NVS results on both extrapolation (ScanNet++) and interpolation (DeepBlending), with baseline methods suffering from ghosting and artifacts.

Efficiency

Gaussian-Voxel Duet yields a significant acceleration in convergence and training time compared to global SDF-guided approaches:

• 9× faster than GSDF on both ScanNet++ and DeepBlending.
• Competitive with pure explicit methods while delivering superior geometry.

Ablation

A detailed ablation confirms contributions from each module:

• Adding voxel scaffold and explicit/implicit tethering systematically improves geometric metrics, reduces average Gaussian-surface distance, and suppresses outliers.

  Figure 8: Ablation study illustration – progressive reduction of floaters and increase in surface detail with each module.

Generalization, Scalability, and Limitations

The method maintains performance over a range of scene complexities and scales.

Figure 9: Dataset overview showing scene scale, image resolution, and evaluation protocols for all tested datasets.

Figure 10: Qualitative scalability demonstration on large-scale scenes, evidencing method’s efficacy under expansive settings.

Preliminary exploration on unbounded scenes is promising but less effective due to SDF’s preference for watertight surfaces; extensions to Unsigned Distance Fields or multiresolution scaffolds are suggested as future directions.

Implications and Future Directions

The Gaussian-Voxel Duet paradigm introduces a compelling direction for efficient, accurate surface reconstruction from monocular images, explicitly bridging efficient point-based modeling with surface-aware geometric priors. By localizing optimization to surface-proximal bands and employing mutual regularization, it achieves a strictly superior trade-off between rendering quality, geometric accuracy, and convergence speed.

Future work could address:

• Unbounded/Outdoor Scenes: Integrating UDF or explicitly handling open surfaces to extend beyond indoor or watertight geometry.
• Hierarchical Grids: Adopting multi-resolution voxel scaffolds for improved detail at varying scales.
• Generalization: Combining per-scene optimization with global, feed-forward predictors for fast, initialization-agnostic setups.

Conclusion

Gaussian-Voxel Duet combines explicit appearance anchors and a sparse, local SDF voxel scaffold via mutual tethering. This yields state-of-the-art monocular surface reconstruction and robust, photorealistic rendering with real-time efficiency, overcoming the key weaknesses of both unregularized 3DGS and slow global SDF models. The dual-scaffold paradigm stands to inform future research on scalable, surface-consistent hybrid representations for both reconstruction and rendering across diverse scene types.