FSFSplatter: Rapid Sparse-View 3D Reconstruction

• FSFSplatter is a surface reconstruction framework that unifies dense Gaussian initialization and transformer-based multi-view encoding to generate 3D scenes from sparse RGB inputs.
• It employs a self-splitting Gaussian head and contribution-based pruning to refine geometric details and maintain consistency even with limited views.
• Empirical evaluations show improved accuracy with lower Chamfer Distance and LPIPS errors, achieving high-fidelity reconstructions in approximately three minutes per scene.

FSFSplatter is a surface reconstruction and novel view synthesis framework that enables rapid and accurate generation of 3D scenes directly from sparse, uncalibrated RGB images. Distinguished by its integration of end-to-end dense Gaussian initialization, transformer-driven multi-view encoding, differentiable camera parameter estimation, and geometry-driven scene optimization, FSFSplatter circumvents the limitations of classical multi-stage pipelines that require dense calibrated views. The approach is designed to avoid error accumulation and overfitting inherent in sparse-view scenarios, achieving high-fidelity surface reconstruction and novel view synthesis within approximately three minutes per scene.

1. Foundation and Objective

FSFSplatter operationalizes surface reconstruction via Gaussian Splatting—a methodology where 3D scenes are represented and rendered through collections of Gaussian primitives. Traditionally, Gaussian Splatting presupposes dense camera coverage with extensively calibrated parameters. FSFSplatter departs from this by formulating an end-to-end pipeline capable of working with minimal overlapping views. The method does not rely on external multi-stage subsystems such as sequential point cloud extraction, separate pose estimation, or iterative surface recovery. Instead, it replaces this cascade with a unified architecture that simultaneously infers camera parameters, initializes a semi-dense Gaussian representation, and enhances geometric consistency using transformer-based encoding.

FSFSplatter’s central goal is to reliably reconstruct detailed surfaces and synthesize novel viewpoints from free sparse RGB imagery (i.e., setups with few images and unconstrained camera poses).

2. Multi-View Encoding and Initialization

The encoding process relies on a large Transformer backbone pre-initialized with DINOv2 features and weights from VGGT. Sparse RGB images are transformed into high-dimensional tokens, from which multiple outputs are regressed in a single forward pass:

• Camera Parameters: Scale-consistent intrinsics and extrinsics are inferred, obviating the need for external calibration.
• Depth Maps: Estimated by a DPT Head, depth predictions serve as geometric priors for scene initialization and supervision.
• Initial Semi-Dense Gaussians: Feature and depth maps are back-projected to produce a semi-dense point cloud. This serves as input for the scene densification stage.

A “self-splitting Gaussian head” or patch densification module refines the initialization. Each Gaussian primitive $G_P$ is split into $N$ sub-primitives by an encoder-decoder mechanism:

$$\Delta G_{N\cdot P} = \{ \Delta X^3, \Delta R^4, \Delta S^2, \Delta O^1, \Delta SH^{48} \} = D(\text{concat}( \mathcal{F}_P, SH^{48}_P, R^4_P, S^2_P, \mathcal{F}_P ))$$

$\text{Densify}(G_P) = G_P + \Delta G_{N\cdot P}$

This function ensures geometric consistency and local detail even in the presence of sparse input views.

3. Geometry-Driven Scene Optimization

Upon initial densification, the scene typically consists of many Gaussian primitives, including redundant or ambiguous ones (“floaters”). FSFSplatter addresses this through contribution-based pruning:

• Contribution Calculation: For each primitive $n$, contribution $C_n$ is evaluated over all rasterized views using the composite blending equation:

$$C_n = \sum_{p \in P_n} \left( \alpha^p \cdot \prod_{q < p} (1 - \alpha^q) \right) / |P_n|$$

Primitives with low opacity or negligible contribution are eliminated, preserving only geometrically salient components.

The pipeline additionally integrates supervision mechanisms to mitigate overfitting:

• Depth Supervision: Depth maps predicted by the transformer are regularized against rendered depths using $L_1$, SSIM, depth ranking, and smoothness losses. The depth ranking loss is specifically formulated to address scale ambiguity:

$$\mathcal{L}_{\text{rank}} = \sum_{p_1, p_2 \in P} \sigma( \text{sgn}( S(D_{\text{est}}, p_1, p_2) ) \cdot S(D_{\text{re}}, p_1, p_2) + m )$$

where $S(D, p_1, p_2) = D(p_1) - D(p_2)$.

• Multi-View Feature Supervision: A U-Net extracts high-dimensional features from the original images, and multi-view consistency losses are used to preserve finer geometric details.
• Differentiable Camera Parameter Optimization: Camera intrinsics and extrinsics are updated via a backward-propagatable rasterization process. Loss gradients flow not only into Gaussian attributes but also into camera parameter tensors, providing independent optimization for each scene even under sparse conditions.

4. Quantitative Performance and Comparative Analysis

FSFSplatter has been empirically validated on the DTU (object-level) and Replica (scene-level) benchmarks. When compared to prior art including 3DGS, 2DGS, CF-3DGS (pose-free), and FreeSplatter (free-sparse-view), it demonstrates:

Metric	FSFSplatter	Competing Methods
Chamfer Distance (CD)	Lower	Higher
LPIPS Error	46–73% lower	Baseline
PSNR, SSIM	Higher	Lower

Notably, the method retains competitive or superior performance even when baseline comparisons are given ground truth camera parameters, emphasizing FSFSplatter's robustness to sparse, uncalibrated inputs. End-to-end dense Gaussian generation (“Ours(wo Opt.)”) also yields stronger results than competing techniques, even without per-scene optimization.

Computation is rapid: ~3 minutes per scene.

5. Application Domains and Implications

FSFSplatter’s architecture renders it suitable for multiple domains requiring real-time or near-real-time 3D scene understanding, especially where only sparse multi-view imagery is feasible. Key applications include:

• Robotics and Autonomous Driving: Enabling robust 3D scene reconstruction from limited sensor viewpoints.
• Virtual and Augmented Reality: Facilitating quick digitization of real-world environments with minimal effort.
• Mobile Photography: Supporting high-fidelity 3D modeling from limited consumer-grade inputs.
• General 3D Modeling: For any workflow prioritizing geometric accuracy and visual realism with minimal capture requirements.

A plausible implication is the reduction in hardware and data acquisition requirements for high-quality mesh and scene generation in resource-constrained or dynamic settings.

6. Methodological Formulas and Algorithmic Pipeline

Several formal equations underscore FSFSplatter’s methodology:

• Gaussian Splatting:

$u = W \cdot H(s_1, s_2, 1, 1)^\top$

Projects Gaussian coordinates onto pixel locations.

• Alpha Blending:

$$C = \sum_i (c_i \cdot \alpha_i \cdot \prod_{j < i} (1 - \alpha_j))$$

With opacity:

$$\alpha_n = \sigma_n \cdot \exp[ -\frac{1}{2} (p - \mu_n)^\top \Sigma_n^{-1} (p - \mu_n) ]$$

• Contribution-Based Pruning:

$$C_n = \sum_{p \in P_n} \left( \alpha^p \cdot \prod_{q < p} (1 - \alpha^q) \right) / |P_n|$$

• Dense Gaussian Densification: As above in Section 2.

These equations are central to FSFSplatter’s pipeline, providing geometric and photometric consistency, dense point representation, and camera differentiability.

7. Resources and Implementation

FSFSplatter provides open-source code via https://github.com/saliteta/splat-distiller.git and further documentation and resources at https://splat-distiller.pages.dev/. These repositories include architectural details, ablation studies, and visualization material conducive to further research reproduction and extension.

In summary, FSFSplatter establishes a unified, transformer-guided framework for rapid, sparse-view surface reconstruction and novel view synthesis. Its contribution-based pruning, differentiable camera optimization, and depth-guided supervision collectively advance the state-of-the-art, particularly for applications necessitating high-fidelity results from limited input data (Zhao et al., 3 Oct 2025).