PF3plat: Pose-Free 3D Gaussian Splatting

• PF3plat is a pose-free, feed-forward 3D Gaussian splatting framework that enables novel view synthesis from unposed images using modular depth and pose refinement.
• It integrates pretrained monocular depth estimation and visual correspondences to achieve robust coarse alignment followed by learnable fine adjustments for accurate geometry.
• The framework leverages a geometry confidence mechanism and efficient single-pass rendering to deliver state-of-the-art performance on benchmarks like RealEstate10K.

PF3plat is a pose-free, feed-forward 3D Gaussian splatting framework for novel view synthesis from sets of unposed images. Addressing the limitations of traditional 3D Gaussian Splatting (3DGS) in pose-free or sparsely overlapping, unposed scenarios, PF3plat introduces modular depth and pose refinement, geometry-aware parameterization, and efficient single-pass inference. The framework eliminates the need for dense views, accurate camera poses, or substantial image overlap, enabling high-fidelity 3D reconstruction and view synthesis by leveraging pretrained networks for coarse geometric alignment and introducing learnable refinement modules and geometry confidence estimation (Hong et al., 2024).

1. Pipeline Architecture and Data Flow

PF3plat operates on $N$ unposed RGB images $\{I_i\}_{i=1}^{N}$ of fixed spatial resolution $H \times W$. The principal outputs of the pipeline are:

• Predicted per-image depth maps $D_i \in \mathbb{R}^{H \times W}$
• Estimated camera poses $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$
• A pixel-aligned 3D Gaussian field $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$

The processing stages are as follows:

1. Coarse Alignment:
   • Pretrained monocular depth estimation network (e.g., UniDepth) outputs $D_i$.
   • Visual correspondence network (e.g., LightGlue) computes pairwise matches $M_{ij}$ and confidences $C_{ij}$ between images.
   • Robust PnP (Perspective-n-Point) with RANSAC, followed by transformation synchronization, yields initial absolute camera poses $P_i$.
2. Fine Alignment:
   • Depth refinement module (transformer-based) predicts pixel-wise depth offset $\{I_i\}_{i=1}^{N}$0 from depth-net features $\{I_i\}_{i=1}^{N}$1, producing refined depths $\{I_i\}_{i=1}^{N}$2.
   • Pose refinement module takes as input $\{I_i\}_{i=1}^{N}$3, where $\{I_i\}_{i=1}^{N}$4 are Plücker coordinates derived from $\{I_i\}_{i=1}^{N}$5. The output is rotation offset $\{I_i\}_{i=1}^{N}$6 and translation offset $\{I_i\}_{i=1}^{N}$7, yielding refined pose $\{I_i\}_{i=1}^{N}$8.
3. 3D Gaussian Field Construction:
   • For each pixel $\{I_i\}_{i=1}^{N}$9, computes center $H \times W$0, covariance $H \times W$1, opacity $H \times W$2, and color coefficients $H \times W$3, the latter three conditioned on geometry confidence.
4. Geometry Confidence and Splat Parameter Prediction:
   • Multi-view stereo (MVS) cost volumes $H \times W$4 are constructed; guidance cost volumes $H \times W$5 are used for conditioning; the resulting processed cost $H \times W$6 yields pixelwise geometry confidence $H \times W$7.
   • The final Gaussian parameter prediction layers take as input $H \times W$8.
5. Rendering:
   • The set of predicted Gaussians is rendered via a rasterization-based 3DGS renderer for novel view synthesis.

2. Mathematical Formulation of Pixel-Aligned 3D Gaussian Splatting

For each input image $H \times W$9 and pixel $D_i \in \mathbb{R}^{H \times W}$0, the associated 3D Gaussian is parameterized by:

• Center:

$D_i \in \mathbb{R}^{H \times W}$1

where $D_i \in \mathbb{R}^{H \times W}$2 is the intrinsic calibration of image $D_i \in \mathbb{R}^{H \times W}$3.

• Covariance:

$D_i \in \mathbb{R}^{H \times W}$4

• Opacity:

$D_i \in \mathbb{R}^{H \times W}$5

• Color coefficients:

$D_i \in \mathbb{R}^{H \times W}$6, using spherical harmonics of order $D_i \in \mathbb{R}^{H \times W}$7

The volume radiance field is defined by

$D_i \in \mathbb{R}^{H \times W}$8

where $D_i \in \mathbb{R}^{H \times W}$9 encodes the directional color, and the final rendered color at each ray is obtained by integrating $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$0 along that ray.

3. Coarse and Fine Alignment Mechanisms

Coarse Alignment

• Monocular Depth Estimation:

UniDepth, pretrained and frozen, provides initial depth maps $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$1 per image.

• Visual Correspondences:

LightGlue produces $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$2, $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$3 as keypoint pairs and match confidences between image pairs.

• Pose Estimation:

RANSAC-based PnP solvers and transformation synchronization yield initial $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$4. No learning is performed at this stage; all operations are algorithmic, and initialization is robust to weak supervision.

Fine Alignment (Learnable Refinement)

• Depth Refinement:

Input $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$5 (features from UniDepth) are processed via transformer $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$6 and MLP projection $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$7:

$P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$8

• Pose Refinement:

Form Plücker coordinate $P_i = [R_i|t_i] \in \mathbb{R}^{3 \times 4}$9 from refined depth/pose. Jointly with $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$0 and pose token, fed into transformer $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$1, yielding global token $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$2:

$G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$3

The refined pose is updated as $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$4, $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$5.

4. Geometry Confidence Mechanism

After multi-view cost volume processing, the geometry confidence per pixel $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$6 is given by:

$G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$7

High $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$8 implies peaked support in the multi-view depth distribution and thus reliable estimated geometry; low values indicate ambiguity or poor correspondence. The predicted Gaussian parameters $G = \{\mu_i(p), \Sigma_i(p), \sigma_i(p), c_i(p)\}$9 are conditioned on $D_i$0 by concatenating it with $D_i$1 prior to the parameter prediction layers. This enables spatially adaptive parameter estimation, directly reflecting uncertainty in geometric alignment.

5. Loss Functions and Optimization

PF3plat is supervised using a composite multi-objective loss:

• Reconstruction loss:

$D_i$2

• 2D–3D correspondence loss:

For projected 3D Gaussian centers $D_i$3 and corresponding keypoints $D_i$4,

$D_i$5

where $D_i$6 is the Huber loss.

• 3D–3D agreement loss:

$D_i$7

• Total loss:

$D_i$8

with $D_i$9.

6. Experimental Protocol and Ablative Evaluation

Experiments are conducted on RealEstate10K, ACID, and DL3DV datasets, with camera intrinsics known for all cases. Training uses one A6000 GPU, Adam optimizer (learning rate $M_{ij}$0), batch size 5, and 40,000 iterations.

Key quantitative findings and ablations on RealEstate10K:

Method	PSNR	SSIM	LPIPS	Rot. Err.	Trans. Err.
Baseline (coarse align + MVSplat)	20.14	0.694	0.281	2.776°	10.043°
Full PF3plat	22.347	0.763	0.205	1.965°	10.113°
w/o Depth refinement	21.963	0.759	0.208	2.240°	9.701°
w/o Pose refinement	21.519	0.737	0.222	2.349°	12.123°
w/o Geometry confidence	21.239	0.737	0.223	2.303°	9.179°
w/o Pretrained UniDepth	16.132	0.511	0.405	6.990°	21.328°

PF3plat surpasses previous pose-free methods (e.g., DBARF, FlowCAM, CoPoNeRF) by $M_{ij}$12 dB PSNR on RealEstate10K/ACID and $M_{ij}$23 dB on DL3DV, with lower average pose errors.

Further analyses indicate:

• Fully fine-tuning the depth network leads to divergence; restricting learning to light-weight refinement modules is essential for stable training.
• Exclusion of either 2D–3D or 3D–3D consistency losses dramatically impairs convergence, especially for wide-baseline or sparsely overlapping inputs.
• PF3plat generalizes to varying input counts ($M_{ij}$3 views), achieving $M_{ij}$426 dB PSNR and absolute trajectory error $M_{ij}$5 m for multi-view inputs.
• Zero-shot evaluation (e.g., RealEstate10K $M_{ij}$6 DL3DV) achieves PSNR $M_{ij}$720 dB, indicating robust generalization.

7. Context, Limitations, and Extensions

PF3plat demonstrates that reliable 3D view synthesis is possible from unposed and sparsely overlapping images using a unified feed-forward approach. The reliance on frozen, pretrained depth and correspondence models (notably UniDepth and LightGlue) is integral; ablation shows catastrophic degradation without pretrained depth features. All refinement is restricted to lightweight modules to prevent overfitting and instability. The geometry confidence mechanism provides adaptive control over the spatial reliability of splatting parameters, allowing PF3plat to handle weakly aligned or ambiguous regions robustly.

The framework is extensible to additional inputs, new pretrained backbones, or alternative cost volume construction techniques. The core design enables efficient single-pass inference suitable for real-time novel view synthesis at scale. The approach represents a state-of-the-art solution in pose-free feed-forward 3D Gaussian splatting (Hong et al., 2024).