Pixel-Aligned Gaussians: Rendering & Reconstruction

• Pixel-aligned Gaussians are explicit, per-pixel parametric primitives that map 2D features to 3D geometric representations via depth regression and unprojection.
• They enable efficient differentiable rendering, achieving competitive PSNR in novel view synthesis, video generation, and SLAM through structured encoder-based pipelines.
• Challenges such as view-bias, occlusion misalignment, and quadratic scaling are addressed via adaptive allocation, hierarchical splatting, and transformer-based refinements.

Pixel-aligned Gaussians are explicit, per-pixel parametric primitives used in feed-forward representations for high-fidelity image, 3D scene, and video synthesis. Each Gaussian is anchored ("aligned") to a specific pixel in one or more calibrated source images, typically via image-space feature extraction followed by depth regression and geometric unprojection. These representations are central to modern differentiable rendering, single- or multi-view reconstruction, video synthesis, SLAM, and neural image compression. The pixel-aligned formulation offers a highly structured interface between 2D convolutional architectures and 3D geometric reconstruction but imposes specific trade-offs regarding adaptivity, multi-view consistency, memory efficiency, and downstream application flexibility.

1. Parametric Formulation and Pixel-to-Gaussian Lifting

The canonical pixel-aligned Gaussian representation assigns a unique 2D (for image-plane splatting) or 3D (for geometry) Gaussian to each pixel of an input image. Each primitive typically carries parameters for center $\mu$ (either in $\mathbb{R}^2$ or $\mathbb{R}^3$), covariance $\Sigma$ (anisotropic or isotropic), color (RGB or spherical harmonics), and opacity (density). For 3D variants, unprojection relies on per-pixel depth (either predicted or observed):

• Given a pixel $u=(u_x,u_y)$ and depth $d(u)$, the 3D Gaussian mean is computed as

$$\mu(u) = R \left( d(u) K^{-1} [u_x, u_y, 1]^T \right) + T,$$

with camera intrinsics $K$ and pose $(R,T)$ (Wang et al., 23 Sep 2025, Fei et al., 2024, Shen et al., 2024).

• The covariance $\Sigma$ can be factorized by diagonal scales $\mathbb{R}^2$0 and rotation $\mathbb{R}^2$1 (e.g., via quaternions), e.g., $\mathbb{R}^2$2 (Almeida et al., 2 Jan 2026, Fei et al., 2024, Shen et al., 2024).

Augmentations include per-pixel color (RGB or SH coefficients), density or opacity $\mathbb{R}^2$3, view-dependent head features, and in dynamics applications, per-Gaussian velocity and acceleration (Almeida et al., 2 Jan 2026).

2. Network Architectures and Training Pipelines

The pixel-aligned paradigm is typically realized via a 2D encoder (often a U-Net or ResNet supplemented with cross-view attention for multi-view settings), a per-pixel MLP head that regresses Gaussian parameters, and a geometric lifting stage to map pixels to Gaussians in 2D or 3D. For single-view video synthesis, pipelines use a U-Net encoder-decoder on the input image and estimated depth map, potentially supplementing with features from pretrained vision transformers (e.g., DINOv2) before per-pixel regression of Gaussian parameters (Almeida et al., 2 Jan 2026).

In multi-view settings, shared image encoders extract features per input view, which are fused (often by cross-attention or cost-volume aggregation). Each pixel then produces a Gaussian via regression, and per-pixel depth is typically predicted using a dedicated depth head (Wang et al., 23 Sep 2025, Fei et al., 2024).

Training objectives may include reconstruction loss on rendered target views (typically a weighted sum of L2 and LPIPS), and in dynamic scenarios, photometric and perceptual loss terms on predicted future frames, along with depth consistency and variational autoencoder (VAE) losses for dynamics (Almeida et al., 2 Jan 2026).

3. Rendering via Gaussian Splatting

Pixel-aligned Gaussians support differentiable rasterization known as "Gaussian splatting." In 2D, this consists of rendering colors as a normalized sum of the top-K Gaussians at each pixel coordinate, weighted by density. In 3D settings, each Gaussian is projected into the target image plane as an elliptical 2D Gaussian; colors and opacities are composited via volume rendering or normalized splatting:

• For $\mathbb{R}^2$4 Gaussians $\mathbb{R}^2$5, the rendered color at pixel $\mathbb{R}^2$6 is

$\mathbb{R}^2$7

where $\mathbb{R}^2$8 (Wang et al., 23 Sep 2025, Zhang et al., 2024).

For 4D (dynamic) scenes, as in Pixel-to-4D, Gaussian parameters are advanced in time using per-object velocities/accelerations and group assignments from instance segmentation. Rendering in novel views is achieved by reprojecting the evolved Gaussians and splatting (Almeida et al., 2 Jan 2026).

4. Applications Across Vision and Graphics

Pixel-aligned Gaussians have broad adoption:

• Single-view and multi-view 3D reconstruction: Assigning one or more Gaussians per pixel and training to synthesize held-out views, with pipelines ranging from Splatter Image (one Gaussian per pixel) to hierarchical expansions allowing k>1 children for occlusion recovery (Shen et al., 2024, Fei et al., 2024).
• Video generation: Via single-image-to-video synthesis with physically consistent camera motion and explicit object dynamics; by leveraging per-pixel alignment, synthetic sequences are both temporally and geometrically consistent (Almeida et al., 2 Jan 2026).
• Depth refinement and MVS: Extremely constrained 1DoF pixel-aligned formulations enabling fast, fine-grained geometric post-processing (e.g., PAGaS, SGAD-SLAM), fixing Gaussian means along camera rays with a single depth parameter per pixel (Recasens et al., 24 Apr 2026, Hu et al., 22 Mar 2026).
• Image compression: Content-adaptive 2D splatting delivers low-bitrate, real-time image reconstruction with competitive visual fidelity to block-based and implicit-field baselines (Zhang et al., 2024).
• SLAM: Fast RGB-D SLAM via pixel-aligned Gaussians with per-pixel depth adjustability and robust metric tracking (Hu et al., 22 Mar 2026).
• Novel view synthesis: Feed-forward architectures such as MVSplat and FreeSplat use pixel-aligned Gaussians for image-based 3D reconstruction and rendering (Wang et al., 23 Sep 2025).

5. Advantages, Limitations, and Mitigations

The principal strengths of pixel-aligned Gaussians comprise:

• Simplicity of mapping pixel-local features to 3D primitives.
• Hardware- and batch-friendly rendering pipelines due to regular arrangement.
• Explicit, differentiable structure suitable for feed-forward, encoder-based learning and large-scale training.
• Real-time or near-real-time inference for image, geometry, and SLAM settings.

However, several limitations have been systematically identified:

Limitation	Source/Details	Quantitative Evidence
View-bias & fixed density	Number and arrangement cannot adapt to scene complexity	PSNR: Pixel-aligned MVSplat 26.4 dB vs. VolSplat 31.3 dB (Wang et al., 23 Sep 2025)
Misalignment under occlusion	Noisy depth, low texture induce "floaters," artifacts	Poor geometry in weak-texture regions (Wang et al., 23 Sep 2025)
Sparse multi-view inconsistency	No unified 3D support across views	Degradation as input views decrease (Wang et al., 23 Sep 2025, Fei et al., 2024)
Quadratic scaling at high-res	O(HW) primitives for H×W images—poor scalability	4K: Pixel-aligned 9.4M vs. LGTM’s 147K (Lao et al., 26 Mar 2026)
Redundancy/poor generalization	Uniform per-pixel; redundancy when views increase	Pixel-aligned method PSNR drops as views increase, CGA method improves (Fei et al., 2024)

Mitigations proposed include:

• Cascade Gaussian Adapter (CGA) and transformer-based refinement for adaptive allocation (Fei et al., 2024).
• Hierarchical splatting with per-pixel child Gaussians (Shen et al., 2024).
• Voxel-aligned predictions (VolSplat) for improved cross-view consistency and scene-adaptive density (Wang et al., 23 Sep 2025).
• Efficient per-primitive textures to decouple geometry from output resolution (LGTM) (Lao et al., 26 Mar 2026).

6. Quantitative Benchmarks

Pixel-aligned pipelines yield competitive performance across diverse tasks, but their effectiveness is tightly coupled to scene complexity, input data quality, and system scale.

• Novel view synthesis: Pixel-aligned MVSplat (6-view input) achieves 26.4 dB PSNR, while adaptive (voxel-aligned) VolSplat achieves 31.3 dB (Wang et al., 23 Sep 2025).
• 3D reconstruction: On RealEstate10K/ACID, PixelGaussian yields consistently improving PSNR as reference view count increases (e.g., 26.72→26.85 dB for 2→4 views), while pixel-aligned baselines degrade sharply (MVSplat 26.25→20.24 dB) (Fei et al., 2024).
• Compression: Image-GS outperforms neural implicit and block codecs at similar memory budgets, e.g., PSNR 32.19 dB at 0.244 bpp vs. SIREN 27.48 dB (Zhang et al., 2024).
• SLAM: SGAD-SLAM achieves PSNR/SSIM of 44.87/0.998 on Replica (vs. VTGS-SLAM 43.34/0.996), mapping at 0.9 s per frame (Hu et al., 22 Mar 2026).
• Scalability: LGTM (non-pixel-aligned) supports feed-forward 4K rendering at practical resource cost, while pixel-aligned methods are prohibitively expensive beyond HD (Lao et al., 26 Mar 2026).

7. Extensions, Generalizations, and Outlook

Active research generalizes pixel-aligned Gaussians from strict one-to-one mappings to dynamic, scene-adaptive strategies by analyzing geometric complexity and allocating primitives where needed. Transformer-based refinement, multi-stage cascading, and hybrid approaches (e.g., hierarchical structures with child Gaussians per pixel) address redundancy and occlusion limitations (Fei et al., 2024, Shen et al., 2024).

Pixel-aligned principles also underlie minimalist post-processing for depth refinement and plug-and-play modules for MVS baselines (PAGaS), as well as content-adaptive 2D representations (Image-GS) for fast neural image coding (Recasens et al., 24 Apr 2026, Zhang et al., 2024). These design patterns suggest pixel alignment will persist as a structural inductive bias linking image-space processing to explicit, explicit geometric reasoning in neural rendering, while hybrid and adaptive variants will increasingly dominate as applications scale to complex dynamic scenes, higher resolution, and strong generalization requirements.