Pixel-Aligned 1DoF Gaussian Splatting (PAGaS)

• The paper demonstrates that PAGaS refines per-pixel depth by optimizing a single depth parameter per Gaussian, leading to improved Chamfer and F1-scores on benchmark datasets.
• PAGaS is a method that leverages geometric priors and pixel-aligned Gaussian representations to achieve high-fidelity depth maps while maintaining computational efficiency.
• The approach integrates an occlusion-aware 3DGS rasterizer with carefully designed loss functions to enhance depth accuracy in challenging 3D reconstruction tasks.

Pixel-Aligned 1DoF Gaussian Splatting (PAGaS) extends recent advances in Gaussian Splatting (GS) techniques by introducing a constrained 1-degree-of-freedom refinement for per-pixel depth estimation in multi-view stereo (MVS). Unlike full 3DGS systems, which often optimize over position, covariance, color, and opacity for each Gaussian, PAGaS restricts each pixel-aligned Gaussian to optimizing depth as its sole parameter, leveraging geometric priors while achieving pixel-accurate depth refinements. By tightly relating the spatial and scale parameters of Gaussians to image back-projection, this method enhances depth map fidelity on challenging 3D reconstruction benchmarks while remaining computationally efficient (Recasens et al., 24 Apr 2026).

1. Formulation and Theoretical Foundations

PAGaS represents each pixel $i$ in a target view image as a single 3D Gaussian $g_i(p)$, parameterized as

$$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$

with $\mu_i \in \mathbb{R}^3$ denoting the 3D center, and $\Sigma_i \in \mathbb{R}^{3 \times 3}$ the covariance. The 3D center is constrained by the camera geometry:

$\mu_i = C + d_i \cdot K^{-1}[u_i \ v_i \ 1]^\top$

where $C$ is the camera center, $K$ the intrinsic matrix, $(u_i, v_i)$ the pixel coordinates, and $d_i$ the variable depth.

Covariance is not learned in a general form but enforced to be spherical,

$g_i(p)$0

where the scale $g_i(p)$1 is set such that the Gaussian projection in image space covers approximately one pixel. The Euclidean depth is given as

$g_i(p)$2

and the scale as

$g_i(p)$3

where $g_i(p)$4 are principal point offsets and $g_i(p)$5 are focal lengths. Each Gaussian is fixed in opacity and color ($g_i(p)$6, $g_i(p)$7 sampled directly from the reference RGB image), with no per-Gaussian rotation or appearance modeling.

2. Rendering, Loss Functions, and Optimization

Rendering of the per-pixel Gaussians follows a forward splatting mechanism:

$g_i(p)$8

with occluded Gaussians ignored using adaptive radius and depth thresholds. The method uses an occlusion-aware 3DGS rasterizer suited to tightly constrained 1DoF splats.

The loss function for depth refinement involves:

• Photometric Consistency: Enforcing rendered-to-observed color agreement,

$g_i(p)$9

with $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$0. Disoccluded pixels in context views are excluded by warping masks via initial depths.

• Edge-aware Regularization: Pixel weights $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$1 penalize over-smoothing at color edges,

$$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$2

• Normal Smoothness: Penalizes angle differences in surface normals,

$$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$3

with $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$4. Per-pixel normals $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$5 are computed by finite-differencing the evolving depth map.

The full loss is $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$6. Optimization occurs view-by-view, using up to $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$7 nearest context images measured by camera orientation. Scalar depths $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$8 are refined using Adam across a two-level image pyramid (half and full resolution), with a schedule: initial learning rate $$g_i(p) = \exp\left[-(p-\mu_i)^\top \Sigma_i^{-1} (p-\mu_i)\right]$$9, step decay on plateau, and termination at $\mu_i \in \mathbb{R}^3$0.

3. Implementation Overview

Initialization of each view uses an external MVS pipeline's coarse depth (e.g., Colmap, MVSAnywhere, 2DGS, or PGSR), as $\mu_i \in \mathbb{R}^3$1, and $\mu_i \in \mathbb{R}^3$2 from the associated RGB value. Camera intrinsics/extrinsics are assumed known (from SfM or Colmap). Images are normalized to $\mu_i \in \mathbb{R}^3$3 and optionally foreground-masked.

The learning schedule employs Adam with $\mu_i \in \mathbb{R}^3$4, following 200 steps at half resolution (100 for DTU), then 100 steps at full resolution. Processing time per view is approximately 10 seconds on a single 11GB GPU. After per-view depth refinement, final meshes are produced by fusing all view depths into a TSDF grid, extracting surfaces using Marching Cubes.

4. Empirical Evaluation

PAGaS is validated on several datasets and metrics:

• DTU (object scans): Chamfer distance (mm, lower is better).
• Tanks & Temples (TnT): F1-score (%, higher is better).
• BlendedMVS: Only qualitative results due to low resolution.
• ActorsHQ: High-resolution human scans, qualitative results only.

Quantitative improvements over baselines were observed:

• DTU Chamfer (Table 2):

Baseline	Original	PAGaS-refined
MVSAnywhere	1.97	1.85
2DGS	0.75	0.72
PGSR	0.54	0.51

• TnT F1-score (Table 3):

Baseline	Original	PAGaS-refined
MVSAnywhere	0.16	0.18
2DGS	0.26	0.28
PGSR	0.40	0.40

Qualitative results demonstrate recovery of fine-scale geometric details, such as roof tiles, fabric folds, hair strands, and, in high-resolution cases, eyelashes and shoe features. Mesh outputs preserve per-pixel geometric refinements after fusion.

5. Algorithmic Outline

The process for a single view can be summarized as:

1. Input: Images $\mu_i \in \mathbb{R}^3$5, known intrinsics/extrinsics $\mu_i \in \mathbb{R}^3$6, and initial depths $\mu_i \in \mathbb{R}^3$7 from a baseline.
2. For each target view $\mu_i \in \mathbb{R}^3$8:
   • Select $\mu_i \in \mathbb{R}^3$9 nearest context views $\Sigma_i \in \mathbb{R}^{3 \times 3}$0 (by view direction).
   • Initialize pixel depths $\Sigma_i \in \mathbb{R}^{3 \times 3}$1, colors $\Sigma_i \in \mathbb{R}^{3 \times 3}$2.
   • Construct image pyramid (half, full resolution).
   • For each scale:
     • Compute $\Sigma_i \in \mathbb{R}^{3 \times 3}$3 and $\Sigma_i \in \mathbb{R}^{3 \times 3}$4.
     • Render using occlusion-aware 3DGS.
     • Compute loss terms $\Sigma_i \in \mathbb{R}^{3 \times 3}$5.
     • Backpropagate gradients on $\Sigma_i \in \mathbb{R}^{3 \times 3}$6, update with Adam, and adjust learning rate.
     • Proceed to next scale after convergence or learning rate thresholds.
   • Save refined depth $\Sigma_i \in \mathbb{R}^{3 \times 3}$7.
3. After all views: Fuse depths into a TSDF for mesh extraction via Marching Cubes.

6. Strengths, Limitations, and Directions

PAGaS is memory and computationally efficient, introducing only one scalar (depth) per pixel, dispensing with appearance, anisotropic covariance, or opacity optimization. The method requires no pretraining or SfM points and can refine both traditional and learning-based MVS outputs in a post-processing framework at full image resolution. Processing time is low and hardware requirements modest.

Limitations include fixed, opaque colors incapable of modeling view-dependent effects or transparency, reliance on high-quality initial depths (failure with random initialization), and sensitivity to exposure inconsistencies across views. Mitigations include exposure modeling; proposed extensions involve joint exposure/intrinsic parameter optimization, hybrid 1DoF+view-dependent color modeling (e.g., low-order spherical harmonics), integration of learned depth priors or uncertainty modeling, and adaptation for dynamic scenes via temporal coupling.

PAGaS provides a principled, efficient mechanism for high-fidelity per-pixel multi-view depth refinement, offering tangible improvements on established MVS benchmarks while retaining methodological and computational simplicity (Recasens et al., 24 Apr 2026).