3D Gaussian Flats: Hybrid 2D/3D Photometric Scene Reconstruction (2509.16423v1)

Abstract: Recent advances in radiance fields and novel view synthesis enable creation of realistic digital twins from photographs. However, current methods struggle with flat, texture-less surfaces, creating uneven and semi-transparent reconstructions, due to an ill-conditioned photometric reconstruction objective. Surface reconstruction methods solve this issue but sacrifice visual quality. We propose a novel hybrid 2D/3D representation that jointly optimizes constrained planar (2D) Gaussians for modeling flat surfaces and freeform (3D) Gaussians for the rest of the scene. Our end-to-end approach dynamically detects and refines planar regions, improving both visual fidelity and geometric accuracy. It achieves state-of-the-art depth estimation on ScanNet++ and ScanNetv2, and excels at mesh extraction without overfitting to a specific camera model, showing its effectiveness in producing high-quality reconstruction of indoor scenes.

• The paper introduces a hybrid method that integrates 2D Gaussians on planar surfaces with 3D Gaussians for non-planar areas, resulting in enhanced photorealistic rendering and precise depth estimation.
• It leverages semantic segmentation, RANSAC, and block-coordinate descent to dynamically detect planes and reposition Gaussians, ensuring robust scene densification.
• Experimental evaluations on ScanNet datasets confirm state-of-the-art depth accuracy and competitive rendering quality even in sparse view and textureless scenarios.

3D Gaussian Flats: Hybrid 2D/3D Photometric Scene Reconstruction

Motivation and Problem Statement

Photometric scene reconstruction via neural radiance fields (NeRF) and 3D Gaussian Splatting (3DGS) has achieved high-fidelity novel view synthesis, but these methods exhibit significant limitations when reconstructing flat, textureless surfaces prevalent in indoor environments. Specifically, volume-based approaches tend to produce semi-transparent, geometrically inaccurate representations for such surfaces due to the under-constrained nature of photometric objectives. Conversely, surface-based methods (e.g., 2DGS) yield more accurate geometry but at the expense of rendering quality and flexibility. The central challenge addressed in this work is the simultaneous achievement of photorealistic rendering and accurate geometric reconstruction for both planar and non-planar regions.

Figure 1: 3D Gaussian Flats combine 2D Gaussians on detected planar surfaces with 3D Gaussians elsewhere, yielding photorealistic quality and improved geometry over prior methods.

Hybrid 2D/3D Gaussian Representation

The proposed method introduces a hybrid representation: planar regions are modeled using 2D Gaussian splats constrained to semantically detected planes, while the remainder of the scene is represented with unconstrained 3D Gaussians. This design leverages semantic segmentation masks to dynamically detect and refine planar regions during optimization, enabling the model to adaptively grow planar extents and merge partial planes across views.

Figure 2: Training is split into a warm-up phase (3D Gaussians only) and a planar phase (joint optimization of planes and Gaussians via block-coordinate descent).

Representation Details

• Planes: Each plane is parameterized by a 3D origin and normal, with associated 2D Gaussians defined in the local plane coordinate system.
• 2D Gaussians: Centers and covariances are mapped to world coordinates via rigid transformations, enabling compatibility with standard 3DGS rendering pipelines.
• 3D Gaussians: Freeform primitives with view-dependent color (spherical harmonics) and opacity, as in vanilla 3DGS.

Optimization Strategy

Optimization proceeds via block-coordinate descent:

• Plane parameters are optimized for photometric and mask consistency losses.
• Gaussian parameters (both 2D and 3D) are optimized for photometric, mask, total variation, scale, and opacity regularization.
• Alternating optimization is critical; simultaneous updates cause instability and degraded results.

  Figure 3: Ablation paper demonstrates the necessity of alternating optimization, mask and TV losses, and 2D Gaussian snapping/relocation for depth accuracy.

Dynamic Plane Detection and Gaussian Relocation

Planar regions are detected via RANSAC on the current Gaussian point cloud, using semantic masks and depth constraints. Inliers are snapped from 3D to 2D Gaussians, with merging of planes when spatially and angularly proximate. Densification is achieved by stochastically relocating freeform Gaussians to planes when they are sufficiently close in both normal and tangent directions, controlled by hyperparameters.

Figure 4: Freeform Gaussians are relocated to planes when both perpendicular and parallel distances are small, facilitating densification of planar regions.

Experimental Results

Novel View Synthesis

The method is evaluated on ScanNet++ and ScanNetv2, with comparisons to 3DGS, 3DGS-MCMC, RaDe-GS, 2DGS, and PGSR. Metrics include PSNR, SSIM, LPIPS for image quality, and RMSE, MAE, AbsRel, and $\delta$ thresholds for depth accuracy.

• Depth estimation: The hybrid method achieves state-of-the-art results, with RMSE and AbsRel significantly lower than all baselines.
• Rendering quality: PSNR and SSIM are on par with fully 3D methods, with only minor trade-offs due to geometric constraints.

  Figure 5: Quantitative and qualitative results show significant improvement in predicted depth while maintaining rendering quality comparable to full 3D representations.

  

  Figure 6: On ScanNetv2, the method outperforms baselines in both image and depth quality, even with sparse camera views.

Mesh Extraction

Mesh extraction is performed via unprojection of 2D masks, voxel downsampling, Marching Squares contour extraction, and ear-clipping triangulation. The method generalizes across camera modalities (DSLR and iPhone), unlike prior methods which overfit to specific domains.

• Mesh accuracy: The method yields higher precision, recall, and F1 scores than AirPlanes and PlanarRecon, with fewer spurious plane detections and more complete meshes.

  Figure 7: Consistent mesh extraction across camera types, with fewer inaccurate plane detections and more complete planar meshes than baselines.

Full Mesh and Sparse View Generalization

On ScanNet++ and ETH3D, the method maintains competitive surface reconstruction and rendering quality, even in sparse view scenarios where planar priors are crucial.

Figure 8: Full mesh extraction on ScanNet++ demonstrates competitive surface reconstruction and rendering quality.

Figure 9: Rendering results on ETH3D scenes show superior quality in sparse view settings due to the planar representation.

Figure 10: Full mesh extraction on ETH3D scenes outperforms baselines in both Chamfer distance and F1 score.

Qualitative Analysis

The method produces crisp, photorealistic renderings and accurate depth maps, especially in textureless regions where prior methods fail. Limitations of Gaussian Splatting for textureless surfaces are noted, but the hybrid approach mitigates these issues.

Figure 11: Superior image and depth quality in novel views on ScanNet++ iPhone dataset; limitations for textureless surfaces are noted.

Figure 12: Visualization of output planes on ScanNet++ test views; reconstructed planes align faithfully with ground truth.

Implementation Considerations

• Computational requirements: Training is performed on a single A6000 ADA GPU (46GB), with scene reconstruction taking ~1 hour and mesh extraction ~3 minutes. RANSAC and block-coordinate descent introduce overhead, but future work may reduce this via custom CUDA kernels.
• Hyperparameters: Relocation thresholds ($\sigma_\perp$, $\sigma_\parallel$), regularization weights ($\lambda_\text{mask}$, $\lambda_\text{TV}$, $\lambda_\text{scale}$, $\lambda_\text{opacity}$) are empirically tuned.
• Robustness: The method is robust to random initialization, maintaining performance without reliance on SfM point clouds.
• Domain generalization: Mesh extraction generalizes across camera models, overcoming domain gaps that limit prior methods.

Limitations and Future Directions

• Textureless regions: Initial 3DGS reconstruction may yield insufficient Gaussians in flat areas; adaptive densification strategies could address this.
• Appearance modeling: Spherical harmonics are limited for view-dependent effects; stronger appearance models may improve fidelity.
• Semantic mask quality: Dependence on SAMv2 masks introduces error; improvements in segmentation will directly benefit reconstruction.
• Training efficiency: RANSAC and alternating optimization increase training time; algorithmic and engineering optimizations are warranted.

Conclusion

3D Gaussian Flats present a principled hybrid 2D/3D representation for photometric scene reconstruction, achieving state-of-the-art depth estimation and competitive rendering quality. The method enables accurate planar mesh extraction and generalizes across camera modalities, addressing key limitations of prior volumetric and surface-based approaches. The integration of semantic priors and dynamic plane detection within the optimization loop is shown to be essential for robust, high-quality reconstruction. Future work should focus on improving appearance modeling, densification strategies, and computational efficiency to further advance hybrid scene representations.