Pixel-Aligned Gaussian Training

• The paper introduces a feed-forward framework that predicts per-pixel 3D Gaussian primitives, bypassing traditional per-scene optimization.
• PGT maps 2D pixels to 3D Gaussians by fusing multi-view features and lifting depth predictions into world coordinates with learned anisotropic covariances.
• Empirical results show that PGT achieves high reconstruction quality and real-time performance, making it ideal for dynamic and efficiency-critical applications.

Pixel-Aligned Gaussian Training (PGT) is a feed-forward technique for novel view synthesis that directly maps each input image pixel to a corresponding 3D Gaussian primitive, enabling efficient and generalizable 3D Gaussian Splatting (3DGS) without per-scene optimization. Unlike traditional scene-specific optimization of unstructured Gaussian clouds, PGT leverages deep learning to predict all parameters of these pixel-aligned Gaussians in a single network pass, supporting real-time and high-quality reconstruction across diverse settings (Park et al., 21 Dec 2025, Zhou et al., 2024).

1. Theoretical Formulation of Pixel-Aligned Gaussians

PGT treats every pixel in each source image as a predictor for a unique 3D Gaussian primitive. Given $N$ input images $\{ I_i \}_{i=1}^N$, each of size $H \times W$, the system generates $N \cdot H \cdot W$ primitives per scene. The $j$-th pixel in the $i$-th view yields a Gaussian

$$G_{i,j}(x) = \mathcal{N}(x; \mu_{i,j}, \Sigma_{i,j}),$$

where the parameters are:

• $\mu_{i,j} \in \mathbb{R}^3$: Gaussian mean (center) in world coordinates
• $\Sigma_{i,j} \in \mathbb{R}^{3 \times 3}$: positive-definite covariance matrix (generally anisotropic)
• $c_{i,j} \in \mathbb{R}^3$: RGB color
• $\{ I_i \}_{i=1}^N$0: scalar opacity

All parameters for $\{ I_i \}_{i=1}^N$1 are regressed per-pixel, using only local and multi-view fused image features, ensuring that primitives are pixel-aligned (Park et al., 21 Dec 2025, Zhou et al., 2024).

In approaches such as GPS-Gaussian+, these parameters are defined through dense Gaussian parameter maps:

$\{ I_i \}_{i=1}^N$2

where $\{ I_i \}_{i=1}^N$3 is the 3D position, $\{ I_i \}_{i=1}^N$4 the color (copied from the source image), $\{ I_i \}_{i=1}^N$5 the rotation (unit quaternion), $\{ I_i \}_{i=1}^N$6 the anisotropic scale, and $\{ I_i \}_{i=1}^N$7 the opacity (Zhou et al., 2024).

2. Network Architecture and Feature Regression

PGT implementations employ advanced encoder-decoder architectures for efficient per-pixel parameter regression:

• EcoSplat uses a Vision Transformer (ViT): Each input view is tokenized and encoded with a shared ViT backbone. Decoder blocks allow for multi-view fusion via cross-attention, producing per-pixel feature tensors $\{ I_i \}_{i=1}^N$8. Two separate multi-layer perceptron (MLP) "heads" regress the Gaussian parameters: the 'center head' $\{ I_i \}_{i=1}^N$9 predicts $H \times W$0 from concatenated decoder outputs, and the 'parameter head' $H \times W$1 predicts $H \times W$2 from features and a shallow CNN map for fine detail (Park et al., 21 Dec 2025).
• GPS-Gaussian+ leverages a U-Net with stereo attention: The network contains a shared 2D encoder for both source views, a bottleneck with epipolar attention (for multi-view feature sharing), an auxiliary depth encoder (for predicted stereo depth), and a U-Net-style decoder yielding a dense per-pixel feature volume. Small, specialized heads regress rotation, scale, opacity, and depth residual via appropriate activations (normalization, softplus, sigmoid, tanh) (Zhou et al., 2024).

This rigorous per-pixel regression mechanism is fully differentiable and designed to generalize across large datasets without per-scene fine-tuning.

3. Lifting 2D Pixels to 3D Gaussians

The process of mapping a 2D pixel to a 3D Gaussian (termed 'lifting') is key:

• Depth Prediction: Initial coarse depths are produced via stereo matching (e.g., RAFT-stereo). A learned residual map $H \times W$3 refines the depth for each pixel:

$H \times W$4

where $H \times W$5 is a scale hyperparameter and $H \times W$6 a learned head from the per-pixel feature volume $H \times W$7 (Zhou et al., 2024).

• Unprojection: The final depth $H \times W$8 is unprojected to 3D using the camera matrix $H \times W$9:

$N \cdot H \cdot W$0

forming the mean of the corresponding Gaussian.

• Covariance Construction: The covariance $N \cdot H \cdot W$1 is composed from the learned rotation $N \cdot H \cdot W$2 and scale $N \cdot H \cdot W$3 as:

$N \cdot H \cdot W$4

providing anisotropy in Gaussian shape (Zhou et al., 2024).

This design ensures that each Gaussian models a physically meaningful 3D volumetric region, facilitating differentiable rendering.

4. Training Objectives, Losses, and Supervision

PGT relies on reconstruction-driven losses:

• Photometric Rendering Loss: Differentiable splatting renders novel target views, which are compared against ground-truth images using combined mean squared error (MSE) and perceptual losses (e.g., LPIPS):

$N \cdot H \cdot W$5

(Park et al., 21 Dec 2025).

• Depth and Geometry Regularization: With depth supervision, an exponentially weighted L1 loss penalizes disparity prediction errors. In absence of depth, a Chamfer distance penalty enforces geometric consistency between left/right unprojected point clouds (Zhou et al., 2024).
• Stereo Attention and Cost Volume: Epipolar attention layers augment per-pixel features with context from the corresponding stereo view, strengthening geometric reliability for highly accurate depth estimation (Zhou et al., 2024).
• Rendering Loss for GPS-Gaussian+:

$N \cdot H \cdot W$6

All losses are fully differentiable and designed for end-to-end training. Supervision is predominantly based on rendered image quality; no explicit depth, semantic, or geometric annotations are required unless available.

5. Empirical Performance and Implementation Details

Two major implementations provide empirical evidence for PGT effectiveness:

• EcoSplat: PGT achieves PSNR ≈ 24 dB, SSIM ≈ 0.82, and LPIPS ≈ 0.18 on RealEstate10K validation splits. Ablation studies show that omitting PGT (using only importance-aware pruning) significantly degrades reconstruction (PSNR drop >1.5 dB at moderate, >18 dB at extreme compaction), confirming its necessity as a robust base for efficiency-controllable rendering (Park et al., 21 Dec 2025).
• GPS-Gaussian+: On scenes rendered from as few as two views, source-view processing requires ~30 ms and each render ~1.9 ms on a single RTX 3090, supporting ≥25 FPS for high-resolution applications without per-scene optimization. Training is performed on human and human-scene datasets, using 100k iterations of AdamW (learning rate $N \cdot H \cdot W$7) (Zhou et al., 2024).

This suggests PGT is directly suitable for real-time, generalizable novel-view synthesis in both static and dynamic scenes.

6. Relationship to Existing Methods and Research Context

PGT represents a paradigm shift for 3D Gaussian Splatting by enabling pixel-to-primitive feed-forward prediction:

• Contrast with Per-scene Optimization: Traditional methods optimize an unstructured Gaussian cloud per scene, requiring lengthy optimization and offering limited generalization. PGT, in contrast, learns globally valid mappings, renders efficiently in a single pass, and adapts to arbitrary image and scene statistics (Park et al., 21 Dec 2025, Zhou et al., 2024).
• Integration with Differentiable Rendering: By casting splatting and geometric prediction as differentiable operations, PGT connects advances in neural radiance fields (NeRFs), stereo matching, and transformer-based multi-view fusion, producing a unified framework for scene reconstruction.
• Foundation for Efficiency-aware Pruning: In two-stage frameworks (e.g., EcoSplat), the PGT stage provides a dense, highly redundant primitive set which is later pruned and fine-tuned to meet computational budgets via importance-aware Gaussian Finetuning (IGF) (Park et al., 21 Dec 2025).

7. Applications, Limitations, and Impact

PGT-powered models are applicable in efficiency-critical or real-time settings, including:

• Free-viewpoint video rendering
• Real-time human-scene rendering with sparse and dense cameras
• High-resolution novel view synthesis for dynamic or unconstrained scenes

A plausible implication is that PGT, by avoiding explicit scene optimization, may be limited in extremely underconstrained environments—scene generalization hinges on the diversity and coverage of the training data (Park et al., 21 Dec 2025, Zhou et al., 2024). Further research may address scaling to ultra-sparse views and exploring hybrid losses or architectures for edge-case reconstruction.

---

References:

• EcoSplat: Efficiency-controllable Feed-forward 3D Gaussian Splatting from Multi-view Images (Park et al., 21 Dec 2025)
• GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views (Zhou et al., 2024)