Orthogonal Projection-Aware Gaussian Splatting

• The paper introduces an orthogonal projection-aware Gaussian splatting framework that treats projection geometry as a primary component to overcome reconstruction ambiguities.
• It integrates orthogonal plane decomposition, multi-view constraints, and tailored rasterization to stabilize geometry and prevent overfitting across projection views.
• Empirical results across modalities show improvements in PSNR, SSIM, and efficiency, validating the method’s benefits in single-view reconstruction and multi-view synthesis.

Searching arXiv for the cited works and closely related Gaussian splatting papers. Searching for GeoGS3D and related orthogonal/projection-aware Gaussian splatting work. Orthogonal Projection-Aware Gaussian Splatting designates a family of Gaussian-splatting formulations in which projection geometry is treated as a first-class component of representation, conditioning, rasterization, or optimization. In recent work, the term covers several distinct but related ideas: orthogonal plane decomposition for extracting 3D-aware latent structure from a single image; multi-view constraints that prevent Gaussians from overfitting a single projection plane; and rendering formulations that replace or augment standard perspective, image-plane splatting with orthographic line integrals, tangent-plane footprints, or equal-area spherical rasterization (Feng et al., 2024, Jia et al., 11 Aug 2025, Chen et al., 6 Aug 2025, Zhu et al., 29 Jun 2026, Steiner et al., 17 Apr 2025, Tu et al., 15 May 2025). The unifying premise is that conventional 3DGS becomes fragile when projection is handled only through local image-plane approximations, especially under single-view ambiguity, wide fields of view, heterogeneous camera models, or imaging systems whose forward physics are not perspective.

1. Conceptual scope and failure modes

A central motivation is that single-view 3D reconstruction is fundamentally under-constrained: many 3D shapes can explain the same 2D image because of projective ambiguity, occlusions, and the absence of multi-view geometric constraints. In that setting, methods that rely purely on one perspective view tend to produce geometry collapse, multi-view inconsistency, and missing details in unobserved regions (Feng et al., 2024). A related pathology appears even in multi-view optimization: a Gaussian’s shortest axis, or effective normal, can align with the current projection plane so that geometry looks plausible in one view while becoming biased in nearby views. The multi-view normal and distance guidance work makes this failure mode explicit, showing that normals and distances fitted in a single view can be projection-plane aligned rather than surface-aligned in 3D (Jia et al., 11 Aug 2025).

Projection awareness is therefore not a single mechanism. In some systems it means injecting orthographic structure into the latent representation before splatting. In others it means replacing the screen-space approximation itself. UniTriSplat formalizes a broader camera-model version of the same problem: existing 3DGS frameworks rely on camera-specific rasterization and therefore suffer from inconsistent solid-angle sampling across perspective, fisheye, and omnidirectional cameras (Zhu et al., 29 Jun 2026). AAA-Gaussians and VRSplat identify the same weakness from the rasterization side, attributing aliasing, projection artifacts, and view inconsistencies to the simplification of treating splats as 2D entities rather than evaluating them in full 3D or on ray-orthogonal support (Steiner et al., 17 Apr 2025, Tu et al., 15 May 2025).

This suggests that “orthogonal projection-aware” is best understood as a design principle rather than a single renderer. The principle is to constrain Gaussian primitives using geometry that remains stable under viewpoint change or under the physical projection model of the sensing system.

2. Orthogonal-plane priors and single-view reconstruction

The clearest latent-space formulation appears in GeoGS3D, which uses orthogonal or orthographic decomposition onto the canonical planes $XY$, $XZ$, and $YZ$ to recover 3D-aware features from a single input image (Feng et al., 2024). The orthographic projection operators are

$$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$

These projections define a tri-plane representation $T=\{T_{xy},T_{xz},T_{yz}\}$, with a generic mapping

$$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$

In GeoGS3D, a ViT encodes the reference image $I_{\mathrm{ref}}$ into a latent $h$. Two decoders then produce image-plane features $F_{xy}$ via self-attention and orthogonal-plane features $F_{yz}$ and $XZ$0 via cross-attention with a learnable query embedding $XZ$1. The fused geometry condition is

$XZ$2

where $XZ$3 denotes channel concatenation (Feng et al., 2024).

The resulting geometric condition is not itself a renderer. Rather, it conditions a latent diffusion pipeline that synthesizes multi-view-consistent images under sampled relative poses $XZ$4. The denoising objective is

$XZ$5

with the orthogonal geometry condition $XZ$6 concatenated with the CLIP-based semantic condition $XZ$7 and the noisy target latent (Feng et al., 2024). The output is a set of synthesized novel views that are then used to drive a Gaussian-splatting reconstruction stage.

Within this formulation, orthogonal planes are neither a purely architectural convenience nor a volumetric field in the EG3D sense. They function as a geometric prior that disentangles axes, improves multi-view consistency, and stabilizes downstream Gaussian initialization, densification, and color fitting. GeoGS3D’s ablations identify this component as the most critical: removing the orthogonal-plane geometry condition on GSO drops PSNR from 22.98 to 20.79 with no CLIP, to 18.37 with CLIP only, and to 17.05 when both geometry and CLIP are removed (Feng et al., 2024).

3. Projection models and rendering formulations

At the representation level, many of these methods still begin from the standard 3DGS parameterization: each Gaussian has a mean $XZ$8, covariance $XZ$9, opacity $YZ$0, and color $YZ$1, with

$YZ$2

Under perspective rendering, screen-space covariance is obtained through the Jacobian of projection, $YZ$3, followed by front-to-back alpha compositing (Feng et al., 2024). Under an orthographic camera, by contrast, $YZ$4 is constant with respect to depth, foreshortening vanishes, and footprint size is depth-invariant (Feng et al., 2024, Steiner et al., 17 Apr 2025).

CryoGS makes this distinction physically explicit. In cryo-EM, image formation is not perspective and not alpha composited; it is a parallel-beam line integral of the 3D electrostatic potential along the beam axis, modulated by the contrast transfer function. The orthographic projection operator is

$YZ$5

with in-plane shift applied after projection (Chen et al., 6 Aug 2025). For a Gaussian mixture density, the orthographic marginal remains Gaussian, with projected parameters

$YZ$6

CryoGS retains the normalization term $YZ$7 so that projection preserves mass. It then rasterizes the sum of projected Gaussians in real space and applies CTF modulation by FFT on an FFT-aligned coordinate grid. In this setting, orthogonal projection awareness is not a regularizer; it is the forward model itself (Chen et al., 6 Aug 2025).

A related distinction appears in CT. GaSpCT adapts the standard differentiable GS rasterizer to CT novel-view synthesis, but the paper states that it uses a pinhole camera approximation and front-to-back alpha compositing, not explicit Beer–Lambert line integrals (Nikolakakis et al., 2024). The same source also formulates what an orthogonal or parallel-beam Gaussian splatting renderer for CT would require: common ray direction, linear detector projection, constant Jacobian, and multiplicative transmittance accumulation rather than opacity blending (Nikolakakis et al., 2024). This is an important boundary case: projection-aware geometry and projection-aware physics are not identical.

UniTriSplat generalizes the projection problem further by moving rasterization from the image plane to the unit sphere. Using HEALPix, it defines $YZ$8 equal-area pixels with solid angle $YZ$9, and projects Gaussians into spherical radian space before camera-specific resampling (Zhu et al., 29 Jun 2026). The method is intrinsically projection-aware for central cameras and provides an explicit orthographic adaptation: either a distant-pinhole approximation, or replacement of radial depth sorting by along-ray sorting $$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$0 when strictly parallel rays are required (Zhu et al., 29 Jun 2026).

4. Cross-view alignment, attention, and optimization

Projection-aware Gaussian splatting is also an optimization strategy. In GeoGS3D, once multi-view images have been synthesized, Gaussian parameters are optimized with image-level losses and an epipolar attention mechanism that restricts correspondences to epipolar lines (Feng et al., 2024). Between calibrated views, the source-view weight map is

$$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$1

and the feature update is

$$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$2

The stated purpose is to reduce correspondence search space, promote geometric consistency, and regularize Gaussian parameter optimization (Feng et al., 2024).

GeoGS3D also introduces Gaussian Divergence Significance, or GDS, to accelerate adaptive density control. For Gaussians with means $$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$3 and covariances $$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$4,

$$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$5

Split or clone operations are applied only to Gaussians with large positional gradients and large GDS relative to nearest neighbors, with $$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$6 computation via $$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$7-NN pairing (Feng et al., 2024).

The multi-view normal and distance guidance formulation addresses a different failure mode: depth and normals that are locally consistent in one view but not across views. It introduces multi-view distance reprojection regularization $$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$8, which aligns plane-to-camera distances across nearby views, and multi-view normal enhancement $$(x_{xy}, y_{xy}) = (X, Y), \qquad (x_{xz}, z_{xz}) = (X, Z), \qquad (y_{yz}, z_{yz}) = (Y, Z).$$9, which fits local planes from depth neighborhoods and penalizes disagreement after rotation into the neighboring frame (Jia et al., 11 Aug 2025). Because both losses are evaluated in camera or world coordinates rather than solely on the image plane, they directly counter projection-plane alignment bias.

AAA-Gaussians and VRSplat extend projection awareness into rasterization itself. AAA-Gaussians evaluates Gaussian contribution in full 3D, adds an adaptive 3D smoothing filter $T=\{T_{xy},T_{xz},T_{yz}\}$0, uses perpendicular-area amplitude normalization, performs stable view-space bounding, and promotes tile-based culling to 3D with screen-space planes (Steiner et al., 17 Apr 2025). VRSplat combines StopThePop resorting with Optimal Projection, which constructs the footprint on a tangent plane orthogonal to the viewing ray rather than on the image plane. Tile culling is then performed on that optimal plane, and the maximum-density point along an edge is computed in closed form by

$T=\{T_{xy},T_{xz},T_{yz}\}$1

The result is a projection-aware pipeline designed to suppress wide-FoV distortions, cloud-like floaters, and temporal popping in VR (Tu et al., 15 May 2025).

5. Empirical behavior across modalities

The empirical literature shows that projection awareness improves both geometric stability and task-specific accuracy, but the gains depend on which aspect of projection is being corrected.

In single-view object reconstruction, GeoGS3D reports on Objaverse and Google Scanned Objects. On Objaverse, it obtains PSNR 23.97, SSIM 0.921, and LPIPS 0.113; on GSO, PSNR 22.98, SSIM 0.899, and LPIPS 0.146. For single-image reconstruction on GSO it reports CLIP Sim 80.0, CD 0.0232, and Avg. time 70 s. The same study attributes a major runtime benefit to GDS, with reconstruction time reduced to $T=\{T_{xy},T_{xz},T_{yz}\}$2 s from $T=\{T_{xy},T_{xz},T_{yz}\}$3 min without GDS and speedups of up to $T=\{T_{xy},T_{xz},T_{yz}\}$4 faster convergence, while view count ablations show CD improving from 0.0552 to 0.0233 when synthesized views increase from 4 to 16, with negligible gain at 32 views (Feng et al., 2024).

For multi-view scene reconstruction, the normal-and-distance guidance method reports on DTU and Mip-NeRF360. In the downsampled DTU setting it reports best mean CD 0.51 versus PGSR(DS) 0.52, with similar runtime of $T=\{T_{xy},T_{xz},T_{yz}\}$5 h. On Mip-NeRF360 it reports highest average SSIM 0.844, lowest average LPIPS 0.176, and outdoor PSNR best at 24.92. Its ablations further report w/o MDRR: CD 0.54; w/o MNE: CD 0.49; Ours: CD 0.48 (Jia et al., 11 Aug 2025).

For universal-camera rendering, UniTriSplat reports perspective Mip-NeRF360 results of HSSIM 0.806 with PSNR 27.57 dB and SSIM 0.848; on ScanNet++ fisheye data it is best in all metrics with PSNR 29.75, SSIM 0.928, HSSIM 0.883, and LPIPS 0.179; on 360Roam it reports PSNR 21.82, SSIM 0.747, LPIPS 0.353, HSSIM 0.724, and shortest training time 102.1 min (Zhu et al., 29 Jun 2026). These results support the claim that uniform spherical rasterization improves cross-camera behavior rather than only one camera family.

In physically grounded imaging, cryoGS evaluates on EMPIAR-10028, EMPIAR-10049, EMPIAR-10076, and EMPIAR-10180 using published poses and per-particle CTFs. With $T=\{T_{xy},T_{xz},T_{yz}\}$6 anisotropic Gaussians, it reports FSC 0.143 resolutions of approximately 3.80 Å, 2.49 Å, 3.30 Å, and 4.26 Å respectively, and states that it consistently outperforms voxel backprojection and cryoDRGN across frequency ranges while converging in $T=\{T_{xy},T_{xz},T_{yz}\}$7 epochs and reaching $T=\{T_{xy},T_{xz},T_{yz}\}$8–$T=\{T_{xy},T_{xz},T_{yz}\}$9 higher FPS than cryoDRGN at typical $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$0 (Chen et al., 6 Aug 2025). In CT projection synthesis, GaSpCT reports on 20 PPMI brain scans with 360 DRR views at $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$1 steps and detector resolution $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$2. Using 50% training views, it reports PSNR $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$3, SSIM $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$4, and LPIPS $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$5, with graceful degradation to PSNR 42.03 at 25%, 38.5 at 10%, and 34.01 at 5% views (Nikolakakis et al., 2024).

For immersive rendering, VRSplat reports 72+ FPS and average stereo frame times of 8.14 ms on Deep Blending, 12.36 ms on Mip-NeRF 360 Indoor, 10.47 ms on Mip-NeRF 360 Outdoor, and 9.22 ms on Tanks and Temples at $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$6 per eye on an NVIDIA RTX 4090 with a Meta Quest 3. A controlled user study with 25 participants reports statistically significant preference for VRSplat over Mini-Splatting baselines, with artifacts and preference at $$f(X)=g\big(T_{xy}(X,Y),\,T_{xz}(X,Z),\,T_{yz}(Y,Z)\big).$$7 in both comparisons (Tu et al., 15 May 2025).

6. Limitations, misconceptions, and open problems

A common misconception is that orthogonal projection-aware Gaussian splatting always implies an orthographic camera. The literature shows three different meanings. In GeoGS3D, orthogonal structure enters through the latent tri-plane decomposition, while synthesis and rendering remain perspective (Feng et al., 2024). In VRSplat, the relevant plane is the tangent plane orthogonal to the viewing ray, not a global orthographic camera (Tu et al., 15 May 2025). In cryoGS, by contrast, orthographic projection is the imaging physics and replaces perspective alpha compositing altogether (Chen et al., 6 Aug 2025).

Another misconception is that projection awareness alone resolves all reconstruction errors. The papers collectively identify substantial residual limitations. GeoGS3D uses a fixed number of generated views, focuses on single objects, and remains sensitive to input viewpoint and non-Lambertian effects (Feng et al., 2024). Multi-view normal and distance guidance depends on reliable homographies over small baselines; large baselines, severe occlusions, textureless regions, or poor initial SfM can destabilize the losses (Jia et al., 11 Aug 2025). UniTriSplat incurs overhead from HEALPix indexing and sphere-to-image interpolation, and its default derivation remains central-camera based, requiring explicit adaptation for strict orthographic rays (Zhu et al., 29 Jun 2026). GaSpCT uses a pinhole approximation instead of exact CT detector physics and does not explicitly model Beer–Lambert attenuation, beam hardening, scatter, or Poisson photon statistics (Nikolakakis et al., 2024). CryoGS assumes known poses, a homogeneous target structure, and the standard projection approximation, leaving joint pose recovery and heterogeneity unresolved (Chen et al., 6 Aug 2025). AAA-Gaussians still relies on local linearization for footprint estimation and can face numerical issues for degenerate covariances or extreme distortion (Steiner et al., 17 Apr 2025). VRSplat notes that approximate hierarchical depth sorting can still flicker under complex occlusion relationships or extreme depth discontinuities (Tu et al., 15 May 2025).

The broader trajectory is nonetheless clear. Recent work is moving away from a single image-plane approximation of Gaussian footprints and toward formulations in which geometry, rasterization, and optimization are explicitly conditioned by the projection model. A plausible implication is that future systems will further separate scene representation from camera parameterization, while integrating stronger physical forward models when the sensing modality demands them.