VolSplat: Voxel-Aligned 3D Gaussian Splatting

• VolSplat is a 3D reconstruction paradigm that shifts from pixel-aligned to voxel-aligned Gaussian prediction, ensuring adaptive density control and improved geometric consistency.
• It fuses multi-view features using a volumetric grid and refines the data through a sparse 3D U-Net, enhancing robustness against occlusion and texture sparsity.
• Benchmarked on RealEstate10K and ScanNet, VolSplat achieves state-of-the-art rendering metrics, demonstrating superior multi-view fusion and faithful point cloud geometry.

VolSplat is a paradigm in feed-forward 3D Gaussian Splatting for novel view synthesis and 3D reconstruction, characterized by a transition from pixel-aligned Gaussian prediction to voxel-aligned prediction. The approach directly regresses 3D Gaussian primitives from a volumetric grid fused across multiple views, seeking geometric consistency, adaptive density control, and improved robustness to occlusion and texture sparsity. The method demonstrates superior multi-view fusion, more faithful point cloud geometry, and state-of-the-art rendering metrics on benchmarks such as RealEstate10K and ScanNet. VolSplat delineates a scalable framework for real-time 3D reconstruction and sets foundational principles for further research in volumetric scene representations.

1. Paradigm Shift: Voxel-Aligned vs Pixel-Aligned Gaussian Splatting

Conventional feed-forward 3D Gaussian Splatting architectures commonly employ a pixel-aligned Gaussian prediction where each 2D pixel (from rendered or input images) is mapped to a 3D Gaussian through a unidirectional lifting process. This design ties the number of reconstructed Gaussians rigidly to image resolution: with $H \times W$ pixels per image, every image yields exactly $H \times W$ Gaussians. Such pixel-centric mapping induces dependence on the quantity and geometry of source views, causing view-biased Gaussian densities and frequent misalignments across views—especially under conditions of calibration inaccuracy, occlusion, or low surface texture. These limitations often result in "floaters," density artifacts, and poor geometric consistency in multi-view integration.

VolSplat eschews this approach in favor of voxel alignment. Instead of treating the 2D pixel grid as the functional basis for Gaussian placement, it constructs a 3D voxel grid, aggregating features by explicit volumetric fusion and predicts Gaussians for each occupied voxel based on fused features. This strategy fundamentally decouples the 3D representation from the input resolution and enables adaptive, content-dependent Gaussian density, with the number of Gaussians proportional to the complexity and sparsity of the local 3D structure.

2. Pipeline and Mathematical Formulation

The VolSplat pipeline consists of the following components:

1. Multi-view Feature Extraction: Multiple RGB images are processed by a backbone network (e.g., ResNet), augmented by Transformer-based cross-view attention to produce high-dimensional feature maps per view.
2. Depth Prediction and 3D Lifting: For each image, depth estimation is performed using a plane-sweep cost volume. Each 2D pixel $(u, v)$ with predicted depth $D_i(u, v)$ is unprojected to 3D world space as:

$$P_{world} = R_i \cdot (D_i(u, v) \cdot K^{-1} [u \; v \; 1]^{\mathrm T}) + T_i$$

where $K$ is the camera intrinsic matrix, $R_i$ and $T_i$ are the rotation and translation matrices for the $i^\text{th}$ view.

1. Voxelization and Feature Fusion: 3D points are assigned to voxels of size $v_s$:

$$(i, j, k) = (\mathrm{rnd}(x_p / v_s),\: \mathrm{rnd}(y_p / v_s),\: \mathrm{rnd}(z_p / v_s))$$

Features from all points in voxel $S_{i, j, k}$ are aggregated, typically via average pooling:

$$V_{i, j, k} = \frac{1}{|S_{i, j, k}|} \sum_{p \in S_{i, j, k}} f_p$$

where $f_p$ denotes the fused feature of point $p$.

1. Volumetric Refinement: A sparse 3D U-Net refines the voxel features by outputting a residual $R$:

$V' = V + R$

1. Gaussian Parameter Regression: For each occupied voxel, the network predicts offset $\hat{m}_j$, opacity, covariance, and color for its associated Gaussian. The center is parameterized as:

$$\mu_j = r \cdot \sigma(\hat{m}_j) + \mathrm{Center}_j$$

Here, $r$ is a hyperparameter (typically $3 v_s$), $\sigma$ is the sigmoid function, and $\mathrm{Center}_j$ is the centroid of voxel $j$.

The result is a set of 3D Gaussians whose locations, opacities, covariances, and appearance attributes are predicted in a manner directly coupled to the 3D fusion of multi-view input, not individual 2D pixels.

3. Addressing Limitations of Pixel Alignment

VolSplat is designed to overcome several explicit shortcomings found in pixel-aligned prediction:

• View-Biased Density: Pixel alignment leads to fixed Gaussian numbers per image/view, yielding excessive density in simple regions and under-sampling in complex ones. Voxel alignment in VolSplat grants adaptive control, allocating more Gaussians in intricate regions while minimizing those in homogenous areas.
• Feature Matching Errors and Occlusion: 2D-based feature matching is sensitive to errors in calibration and is brittle when occlusions or low-texture surfaces prevent reliable correspondence. The 3D grid-based fusion in VolSplat leverages full multi-view evidence, suppressing misalignments and occlusion-related errors.
• Fidelity and Multi-view Consistency: The volumetric fusion and adaptive density enable VolSplat to produce Gaussian point clouds that exhibit improved geometric consistency across views and scenes. Errors such as "floaters" or surface discontinuities are effectively mitigated by enforcing 3D spatial continuity.

4. Quantitative Benchmarks and Experimental Performance

Performance results substantiating the VolSplat design are presented in evaluations on the RealEstate10K and ScanNet datasets. Table-based comparison (using PSNR, SSIM, and LPIPS metrics) demonstrates:

Method	PSNR (RE10K)	SSIM (RE10K)	LPIPS (RE10K)
pixelSplat	<30.0	<0.93	>0.08
MVSplat	<30.5	<0.94	>0.08
TranSplat	<30.8	<0.94	>0.08
VolSplat	31.30	0.941	0.075

Similar improvements are shown on ScanNet (PSNR: 28.41 for VolSplat). Furthermore, cross-dataset generalization tests (training on RealEstate10K, evaluation on ACID) confirm robustness in unseen scenarios.

5. Applications and Scalability

The voxel-aligned VolSplat framework offers advantages for diverse applications:

• Real-Time Large-Scale Reconstruction: The feed-forward and adaptive nature allows volumetric scene reconstructions to scale efficiently, supporting virtual reality, robotics perception, and rapid environment mapping.
• Digital Twin Creation: The capacity for detailed, faithful Gaussian density control is well-suited for high-fidelity digital twin visualization, where geometric detail and multi-view consistency are critical.
• Augmented and Mixed Reality: The approach's inherent robustness to viewpoint variation makes it suitable for interactive rendering scenarios.
• Research Advancement: By demonstrating a scalable multi-view fusion and volumetric prediction paradigm, VolSplat catalyzes further research into hybrid 2D–3D fusion, adaptive volumetric representations, and non-Lambertian surface modeling.

6. Supplementary Resources

The authors provide reproducibility infrastructure:

• Video Demonstrations: Detailed side-by-side comparisons of VolSplat and pixel-aligned methods for qualitative assessment.
• Open Codebase and Trained Models: Source code and pretrained models are publicly released, facilitating immediate utilization and adaption within the research community.
• Project Website: Additional materials accessible through https://lhmd.top/volsplat.

7. Implications and Future Directions

A plausible implication is that the shift to voxel-aligned prediction not only enhances current 3D Gaussian Splatting pipelines but establishes a conceptual foundation for future improvements in feed-forward 3D scene understanding. Adaptive control over point cloud density, robustness to occlusions, and geometric consistency are likely to play pivotal roles in forthcoming volumetric rendering, scene completion, and interactive synthesis frameworks. The demonstrated scalability suggests applicability to increasingly large and complex spatial datasets, aligning with the trajectory toward real-time, high-fidelity 3D content pipelines.