Learning Stable Canonical Worlds for Novel View Synthesis and Beyond

Abstract: Feed-forward Gaussian splatting (FFGS) facilitates real-time novel view synthesis, yet current methods often remain tied to view-dependent predictions. As more input views are added, they may accumulate noisy or redundant evidence instead of converging to a stable scene representation. In this paper, we introduce CanonicalGS, a feed-forward pipeline that maps cluttered multi-view observations into a stable, scene-centric representation. CanonicalGS first extracts view-centric evidence from depth, semantic features, and uncertainty estimates, and then aggregates this evidence in a canonical latent world using uncertainty-aware fusion. By emphasizing reliable observations while suppressing uncertain or redundant ones, CanonicalGS produces representations that scale more effectively for novel view synthesis and transfer to downstream visual perception tasks. Experiments show up to a $2.5$ dB improvement in peak signal-to-noise ratio for synthesizing novel views and an $11\%$ gain in semantic segmentation accuracy.

• The paper introduces a novel approach that transforms view-centric data into stable, canonical scene representations using reliability-guided aggregation.
• It employs a 2D→2.5D→3D pipeline with GP decoding, achieving up to a 2.5 dB PSNR increase and enhanced performance in segmentation tasks.
• The methodology demonstrates robustness to depth noise and scalability with additional views, offering practical benefits for real-time rendering and perception.

CanonicalGS: Stable Scene-Centric Representation for Feed-Forward Gaussian Splatting

Motivation and Context

Feed-forward Gaussian splatting (FFGS) has enabled real-time novel view synthesis using compact, explicit 3D representations, but prior work often remains tied to view-dependent reconstruction, resulting in unstable representations as additional input views accumulate. This instability manifests as redundant or noisy scene hypotheses, fundamentally limiting downstream perception and scalability. CanonicalGS addresses this by establishing a representation-first pipeline, orienting FFGS toward canonicalization as articulated in Marr's classical vision theory: transforming cluttered, view-centric input into a stable, scene-centric latent world, then decoding this world into Gaussian primitives (GPs) for rendering and perception. By integrating uncertainty-aware fusion and reliability-guided aggregation, CanonicalGS enforces that additional views increase scene evidence rather than introduce conflicting hypotheses.

Methodology

CanonicalGS employs a 2D$\rightarrow$2.5D$\rightarrow$3D pipeline, instantiated as: (1) view-centric evidence formation, (2) scene-centric evidence aggregation, and (3) GP decoding.

1. View-Centric Evidence Formation: Each input view undergoes dense feature extraction via a DINO-v2 backbone, generating patch-level descriptors. Depth and uncertainty are estimated using cost-volume and UNet-inspired modules, providing both geometric cues and pixel-wise reliability. Positional uncertainty arises from probabilistic depth distribution, while appearance uncertainty is inferred via semantic features.
2. Scene-Centric Evidence Aggregation: Input images are rasterized onto a shared voxel lattice in scene space, mitigating sampling irregularity and pose dependence. Aggregation is guided by reliability and feature-consistency with a local representative in each voxel, ensuring unreliable or inconsistent evidence is suppressed before the construction of the canonical latent world. The resulting fields encode accumulated reliability and reliability-weighted feature descriptors.
3. GP Decoding: The aggregated scene fields are decoded into Gaussian primitives. Opacity is constrained to be non-decreasing with aggregated reliability, ensuring primitives become more robust as support increases. Remaining attributes (geometry, SH appearance) are predicted from the reliability-weighted feature field, anchored at the representative scene point.

   Figure 1: CanonicalGS pipeline overview: input images are converted into view-centric evidence, aggregated into a scene-centric latent representation, and decoded to Gaussian primitives; color indicates reliability $R_{\mathrm{scene}}$.

Experimental Analysis

Novel View Synthesis

CanonicalGS is systematically compared to state-of-the-art FFGS baselines including DepthSplat, MVSplat, FreeSplat, ZPressor, and Gaussian-space merging variants. Evaluation spans both indoor (RE10K) and outdoor (DL3DV) datasets, using challenging target-view protocols where extrapolation is required. CanonicalGS consistently outperforms all baselines, with up to a $2.5$ dB PSNR increase and improved SSIM and LPIPS. Notably, its metrics improve monotonically as the number of input views increases, in contrast to pixel-aligned methods where stability degrades beyond two views. Qualitative results demonstrate robust consolidation of appearance and geometry, suppressing view-dependent artifacts and ghosting.

Figure 2: Qualitative synthesis on DL3DV; CanonicalGS produces more stable and contextually refined renderings with increasing input views.

Representation Stability

CanonicalGS features remain stable as input view counts grow, measured via cosine similarity to a dense ($12$-view) reference and linear-probe semantic segmentation. Competing baselines exhibit view-conditioned drift, while CanonicalGS monotonically approaches the canonical reference, demonstrating that additional views consolidate rather than perturb the latent scene.

Figure 3: Left: feature stability improves with more input views; Right: superior linear-probe segmentation from CanonicalGS splatted features.

Scene-centric aggregation preserves semantic structure, yielding spatially coherent predictions that outperform pixel-aligned and latent fusion alternatives on segmentation tasks.

Figure 4: CanonicalGS segmentation masks exhibit spatial coherence, evidencing reliability-guided aggregation’s efficacy.

Robustness and Scalability

CanonicalGS demonstrates strong robustness to depth noise; reliability-guided aggregation suppresses uncertain geometric evidence prior to GP decoding, resulting in smaller PSNR and LPIPS degradation relative to direct pixel-aligned methods.

Level-of-detail control is achieved by subsampling the decoded GP set. CanonicalGS degrades gracefully and avoids hollow artifacts, offering practical compression for scalable applications.

Figure 5: GP subsampling enables smooth quality-compactness tradeoff without introducing hollow regions, supporting scalable rendering.

Architectural Ablations

Ablation studies show that removing reliability or feature-consistency weighting substantially degrades novel view synthesis and representation stability. Aggregated reliability is essential for both scene-centric consolidation and decoding constraints; average merging or unconstrained decoding results in performance collapse, highlighting the necessity of CanonicalGS’s architectural design.

Increasing the number of training views further improves performance, demonstrating that CanonicalGS is not locked to sparse-view regimes and can exploit dense evidence.

Implications and Future Directions

CanonicalGS enforces architectural pressure to organize evidence in scene space prior to rendering, producing representations that scale with increasing input. This fundamentally improves both reconstructive and perceptual tasks, facilitating downstream applications in semantic segmentation, spatial reasoning, and embodied scene understanding. Practical benefits include inference efficiency, compactness, and cross-dataset generalization.

Current limitations include dependence on pose and depth quality, susceptibility to occlusion and dynamic scenes, and fixed voxel lattice granularity. Future work should pursue joint canonicalization—refining pose, depth, visibility, and aggregation in tandem using feedback from the reliability field. Richer uncertainty modeling (e.g., occlusion likelihood, epistemic confidence) may further distinguish unsupported or contradictory regions. Adaptive and hierarchical scene representations could enable room-scale or outdoor-scale deployment, while integration with language-level supervision might extend CanonicalGS features toward open-vocabulary semantic maps and embodied planning.

Conclusion

CanonicalGS establishes a paradigm shift in feed-forward Gaussian splatting, prioritizing canonicalization and uncertainty-aware evidence aggregation for stable, scene-centric representation. Its architectural innovations yield substantial improvements in view synthesis, semantic segmentation, robustness, and scalability. CanonicalGS’s approach lays groundwork for future feed-forward pipelines that efficiently and reliably integrate multi-view observations, supporting broader vision and spatial intelligence tasks.

(2606.23027)