Pixel-Aligned Gaussian Representation

• Pixel-Aligned Gaussian Representation is a technique that maps image pixels to explicit parametrized Gaussian primitives in 2D/3D space, enabling continuous scene reconstruction.
• It employs learned regressors and differentiable rendering pipelines to achieve efficient image compression, novel view synthesis, and super-resolution.
• Adaptive pooling and redundancy management optimize performance, evidenced by improved PSNR scores and reduced BD-rate in experimental benchmarks.

The pixel-aligned Gaussian representation is a parametric paradigm in which image pixels are mapped to explicit Gaussian primitives defined in 2D or 3D space. These Gaussians serve as basis functions for image reconstruction, continuous super-resolution, novel view synthesis, geometric inference, mapping, and compression. Each Gaussian is parameterized by its center, covariance, color or radiance attributes, and often an opacity term. The pixel-aligned formulation explicitly ties the distribution or attributes of Gaussians to the pixel grid of one or more input images, facilitating interpretable, dense, and highly parallelizable operations that align with modern differentiable rendering and vision pipelines. This approach underpins a suite of recent advances in differentiable splatting, real-time neural rendering, rate-efficient image representation, and geometry refinement.

1. Mathematical Formulation of Pixel-Aligned Gaussian Representation

In the general setting, each input pixel is associated with a parameterized Gaussian. In 2D, such a primitive can be written as: $$G_i(x) = w_i \exp\left(-\frac{1}{2}(x-\mu_i)^\top \Sigma_i^{-1} (x-\mu_i)\right)$$ where $\mu_i\in\mathbb{R}^2$ is the center, $\Sigma_i\in\mathbb{R}^{2\times2}$ the covariance (usually positive semidefinite via Cholesky decomposition), and $w_i$ the color or amplitude vector (Liang et al., 30 Dec 2025, Peng et al., 9 Mar 2025).

For 3D, as in multi-view synthesis: $$G_i(X) = \exp\left(-\frac{1}{2}(X - \mu_i)^\top \Sigma_i^{-1}(X - \mu_i)\right)$$ with $\mu_i\in\mathbb{R}^3$ and $\Sigma_i\in\mathbb{R}^{3\times3}$; attributes such as color $c_i\in\mathbb{R}^3$ and opacity $\alpha_i\in[0,1]$ are attached (Zhou et al., 2024, Wang et al., 23 Sep 2025, Fei et al., 2024, Zhang et al., 20 Mar 2025).

Rendering is accomplished by projecting these Gaussians into the image plane, compositing colors/opacity using alpha blending or weighted sums.

2. Construction and Regression of Pixel-Aligned Gaussians

a. 2D Scenario (Super-Resolution, Compression)

Given an LR image $I_{LR}$ of size $\mu_i\in\mathbb{R}^2$0, a learned encoder produces a feature map. Each pixel emits $\mu_i\in\mathbb{R}^2$1 Gaussians (typically, $\mu_i\in\mathbb{R}^2$2. MLP heads regress for each Gaussian:

• Mean $\mu_i\in\mathbb{R}^2$3 (possibly offset from the pixel center)
• Covariance $\mu_i\in\mathbb{R}^2$4 (parametrized or mixed from a learned prior/dictionary)
• Color or amplitude $\mu_i\in\mathbb{R}^2$5

The continuous reconstructed signal is

$\mu_i\in\mathbb{R}^2$6

with $\mu_i\in\mathbb{R}^2$7 (Peng et al., 9 Mar 2025, Liang et al., 30 Dec 2025).

b. 3D Scenario (Multi-View Splatting, Geometric Modeling)

Each pixel in each input view is lifted to a 3D Gaussian by combining image position with a depth estimate: $\mu_i\in\mathbb{R}^2$8 or, for constrained setups, restricting to per-ray (1DoF) models with only depth as a free parameter (Recasens et al., 24 Apr 2026, Hu et al., 22 Mar 2026).

Covariance is often parameterized as: $\mu_i\in\mathbb{R}^2$9 where $\Sigma_i\in\mathbb{R}^{2\times2}$0 is a scaling matrix and $\Sigma_i\in\mathbb{R}^{2\times2}$1 parameterized by quaternions (Zhou et al., 2024, Fei et al., 2024, Zhang et al., 20 Mar 2025).

Opacity $\Sigma_i\in\mathbb{R}^{2\times2}$2 and color $\Sigma_i\in\mathbb{R}^{2\times2}$3 (RGB or radiance, possibly as spherical harmonics) are regressed via decoders from latent feature codes.

3. Algorithmic and Network Architectures

Pixel-aligned Gaussian systems employ a variety of network backbones:

• 2D Image Backbones: UNet, ResNet, SwinIR, or HAT backbones extract features per input view (Liang et al., 30 Dec 2025, Peng et al., 9 Mar 2025, Wang et al., 23 Sep 2025).
• Depth Estimation Modules: Cost-volume or stereo-based modules predict per-pixel depths for 3D lifting (Zhou et al., 2024, Wang et al., 23 Sep 2025, Fei et al., 2024, Zhang et al., 20 Mar 2025).

The regression heads operate pointwise or with local context (often 1×1 or 3×3 convolution), with architectural enhancements including:

• Epipolar Attention: Cross-view attention along epipolar lines for improved stereo (Zhou et al., 2024).
• Cascade Adapters and Pruning: Dynamic splitting and pruning of Gaussians based on geometric complexity metrics, deformable attention, or context-aware hypernetworks (Fei et al., 2024).
• Gaussian Graph Networks: Message passing between view-aligned Gaussian groups via explicit graph constructions with binary correspondences, followed by pooling/merging in 3D (Zhang et al., 20 Mar 2025).

Standard splatting-based differentiable rasterizers project each Gaussian to a 2D ellipse, compositing colors using alpha blending in front-to-back order or normalized weighting (Wang et al., 23 Sep 2025, Fei et al., 2024, Peng et al., 9 Mar 2025).

4. Applications: Rendering, Super-Resolution, Compression, and Geometry

Pixel-aligned Gaussian representations support a broad range of applications:

Domain	Key Methodologies	Representative Papers
Multi-view Rendering	Pixelwise 3D Gaussian Splatting	(Zhou et al., 2024, Wang et al., 23 Sep 2025)
Super-Resolution	Pixel-to-Gaussian 2D Splatting	(Peng et al., 9 Mar 2025)
Image Compression	Structure-guided 2DGS Allocation	(Liang et al., 30 Dec 2025)
Geometry/Depth Refinement	Pixel-aligned 1DoF Gaussians	(Recasens et al., 24 Apr 2026, Hu et al., 22 Mar 2026)
SLAM/Mapping	Ray-aligned Depth-Optimized 3DGS	(Hu et al., 22 Mar 2026)

• Novel View Synthesis: Given multiple RGB views, pixel-aligned Gaussians can synthesize new viewpoints via explicit 3D scene representation (Zhou et al., 2024, Fei et al., 2024, Wang et al., 23 Sep 2025).
• Continuous Super-Resolution: Upsample LR images to arbitrary scales using explicit 2D Gaussian mixtures for real-time continuous signal reconstruction (Peng et al., 9 Mar 2025).
• Compression: Allocate Gaussians and quantize their parameters efficiently according to image structure, achieving a 43.44% BD-rate reduction on the Kodak dataset versus baselines (Liang et al., 30 Dec 2025).
• Geometry/Depth Refinement: Optimize pixel-aligned Gaussians with 1DoF (depth) for detailed, robust per-view stereo depth maps (Recasens et al., 24 Apr 2026).
• SLAM: Use optimized per-pixel 3D Gaussians for robust, efficient tracking and mapping in real-time RGBD SLAM systems (Hu et al., 22 Mar 2026).

5. Efficiency, Scalability, and Redundancy Management

A fundamental challenge of pixel alignment is redundancy. With $\Sigma_i\in\mathbb{R}^{2\times2}$4 pixels across $\Sigma_i\in\mathbb{R}^{2\times2}$5 views, naively allocating one Gaussian per pixel explodes memory and computation, especially as the number of views grows. Methods address this by:

• Pooling/Merging (Post-hoc): Merge and prune Gaussians that represent coincident or overlapping 3D locations post-message passing (Zhang et al., 20 Mar 2025).
• Dynamic Adaptation: Cascade adapters split and prune based on local complexity and view aggregation, keeping the total Gaussian count sublinear in view count (Fei et al., 2024).
• 1DoF Constraints: Restrict Gaussian degrees of freedom to reduce redundant parameterization (e.g., only optimizable along the pixel’s ray) (Recasens et al., 24 Apr 2026).
• Structure Guidance: Initialize and quantize Gaussians preferentially in regions of high gradient or semantic complexity, allocating parameter precision where detail warrants (Liang et al., 30 Dec 2025).

Quantitatively, GGN (Zhang et al., 20 Mar 2025) uses ~102 K Gaussians for 4 views of RealEstate10K (227 FPS) versus 786 K for pixelSplat (110 FPS); PixelGaussian (Fei et al., 2024) grows only from 188 K to 278 K Gaussians from 2 to 6 views, maintaining or improving PSNR; structure-guided allocation in image compression achieves >1000 FPS with sharp edge retention (Liang et al., 30 Dec 2025, Peng et al., 9 Mar 2025).

6. Limitations, Variants, and Comparison to Alternative Paradigms

Pixel-aligned Gaussian representation presents multiple limitations and tradeoffs:

• Fixed Density Bias: Gaussian count and spatial density are tied to pixel grid, over-representing flat or low-detail regions and under-representing geometric complexity (Wang et al., 23 Sep 2025).
• View-Dependent Artifacts: Each view instantiates its own Gaussian map, leading to duplicated or misaligned Gaussians and inconsistent geometry without explicit inter-view fusion (Zhang et al., 20 Mar 2025, Fei et al., 2024).
• Lack of 3D Neighborhood Context: Unless augmented with cross-Gaussian communication (e.g., graph networks), no explicit scene-wide geometric regularization is enforced (Zhang et al., 20 Mar 2025, Wang et al., 23 Sep 2025).
• Redundancy Explosion with Views: The naive approach scales linearly with the product of image size and view count unless pruned.

Voxel-aligned alternatives (e.g., VolSplat (Wang et al., 23 Sep 2025)) address these issues by predicting Gaussians on a 3D voxel grid, leading to superior consistency and control over scene-adaptive Gaussian density. A plausible implication is that for applications demanding strict 3D regularity and scene-adaptive efficiency, voxel-/geometry-aligned strategies may supplant pixel alignment.

7. Representative Experimental Findings and Quantitative Benchmarks

• Rendering quality: For 4 views on RealEstate10K, GGN achieves PSNR 24.76 dB with only 102 K Gaussians vs. MVSplat's 20.86 dB and 262 K Gaussians (Zhang et al., 20 Mar 2025). On ACID, GGN reaches PSNR 26.46 dB.
• Scaling: GGN’s Gaussian count increases only modestly with the number of input views (from ~100 K to ~150 K from 4 to 16 views), whereas pixelwise methods can reach millions of Gaussians, collapsing rendering speed (Zhang et al., 20 Mar 2025, Fei et al., 2024).
• Super-resolution: Pixel-to-Gaussian improves PSNR by up to 0.9 dB over the best INR baseline on Urban100 ×4 (28.22 dB vs. 27.42 dB), with sampling at 1 ms per output scale (Peng et al., 9 Mar 2025).
• Compression: Structure-guided 2DGS achieves 43.44% BD-rate reduction on Kodak images, 29.91% on DIV2K, at rates >1000 FPS (Liang et al., 30 Dec 2025).
• Geometry: PAGaS improves mean Chamfer distance on DTU from 0.75 mm to 0.72 mm (over baseline 2DGS) and increases F1 from 0.26 to 0.28 on Tanks & Temples (Recasens et al., 24 Apr 2026).
• SLAM: SGAD-SLAM achieves PSNR 44.87, SSIM 0.998, and tracking ATE RMSE 0.16 cm on Replica, with robust depth estimation under high corruption (Hu et al., 22 Mar 2026).

8. Outlook and Synthesis

Pixel-aligned Gaussian representations provide a tractable, interpretable bridge between dense pixel observations and explicit parametric scene modeling, supporting scalable, differentiable vision and graphics with real-time inference. Innovations in redundancy reduction, adaptive allocation, and inter-Gaussian communication have addressed many early limitations of view-tied density bias and geometric inconsistency. Ongoing work seeks further gains through scene-adaptive (voxelic/hybrid) alignment, geometric pooling, and learned structure adaptation. Comparative studies confirm that while pixel-aligned methods remain competitive in speed and quality for view-limited, feed-forward rendering and image processing, their long-term scalability for large-scale 3D mapping may depend on continued hybridization with spatially adaptive frameworks.

Key references: (Recasens et al., 24 Apr 2026, Zhang et al., 20 Mar 2025, Liang et al., 30 Dec 2025, Zhou et al., 2024, Fei et al., 2024, Peng et al., 9 Mar 2025, Wang et al., 23 Sep 2025, Hu et al., 22 Mar 2026)