CanonicalGS: Stable 3D Gaussian Splatting

• CanonicalGS is a feed-forward 3D Gaussian splatting method that constructs a stable, scene-centric latent world from posed multi-view observations.
• It employs uncertainty-aware and consistency-aware fusion on a voxelized canonical scene space to integrate view-centric evidence robustly.
• The method decodes the canonical latent world into Gaussian primitives, yielding improvements of up to 2.5 dB in PSNR and an 11% gain in semantic segmentation accuracy.

CanonicalGS is a feed-forward 3D Gaussian splatting method for novel view synthesis that constructs a stable, scene-centric representation before decoding renderable Gaussian primitives. Rather than predicting Gaussians directly from view-aligned evidence, it first maps posed multi-view observations into a canonical latent world defined in shared scene coordinates, then decodes that latent world into 3D Gaussian primitives for differentiable Gaussian splatting. The method is motivated by the observation that many feed-forward Gaussian splatting pipelines remain effectively view-dependent: as more input views are added, they may accumulate noisy or redundant evidence instead of converging to a stable scene estimate. CanonicalGS addresses this through uncertainty-aware, consistency-aware fusion in a voxelized canonical scene space, yielding improved multi-view scalability for novel view synthesis and stronger transfer to downstream perception tasks. The reported results show up to a $2.5$ dB improvement in peak signal-to-noise ratio and an $11\%$ gain in semantic segmentation accuracy (Xu et al., 22 Jun 2026).

1. Concept and problem formulation

CanonicalGS is framed as a criticism of prior feed-forward Gaussian splatting (FFGS) systems that are fast at inference but remain tightly coupled to view-aligned or pixel-aligned predictions. In that regime, additional views do not necessarily stabilize the reconstructed scene; they can instead introduce redundant primitive hypotheses, conflicting geometric estimates, and view-conditioned clutter. CanonicalGS proposes that a feed-forward scene representation should improve by converging as more valid views are observed, rather than by merely producing more primitives (Xu et al., 22 Jun 2026).

Its central representational object is the canonical latent world, described as a stable, shared scene/world representation rather than a collection of independent per-view predictions. The input is an unordered set of posed RGB views,

$\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$

where $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$ is an image and $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$ is the known perspective projection. The overall feed-forward pipeline is written as

$$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$

followed by Gaussian decoding,

$$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$

Here $\mathcal{E}$ denotes view-centric evidence, while $\mathcal{Z}$ denotes the fused scene-centric latent representation (Xu et al., 22 Jun 2026).

The “canonical” designation in this paper refers to a shared scene/world coordinate system induced by the known camera geometry and the voxelized latent scene lattice. It does not introduce a separate learned canonical object frame with explicit deformation into observed states. In that respect, CanonicalGS differs from canonicalized dynamic Gaussian representations and from latent token methods such as TokenGS, whose “canonical coordinate space” refers to global 3D means in a shared frame rather than to a latent scene warped across time (Ren et al., 16 Apr 2026). A plausible implication is that CanonicalGS is best understood as a world-aligned canonicalization of evidence, not as a deformation-based canonical scene model.

2. View-centric evidence extraction

CanonicalGS begins by extracting dense per-view evidence independently, before any cross-view fusion. For each input image $I^{(i)}$, a pretrained DINO-v2 backbone produces patch-level visual features, and a lightweight upsampling decoder restores dense feature maps at image resolution. This yields

$11\%$0

These features are used jointly for depth estimation, appearance representation, and uncertainty prediction (Xu et al., 22 Jun 2026).

Depth is estimated through a multi-view cost-volume mechanism. For a reference view $11\%$1, with sampled depth hypotheses $11\%$2, a pixel $11\%$3 is hypothesized at depth $11\%$4, back-projected to 3D, and reprojected into source view $11\%$5: $11\%$6 The pairwise matching score is the normalized feature correlation

$11\%$7

and the reference-view score volume averages over all other inputs,

$11\%$8

A UNet-like module processes the feature map and score volume to predict a discrete depth probability volume,

$11\%$9

Expected depth and positional uncertainty are then defined as

$\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$0

$\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$1

This $\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$2 term is a depth-distribution uncertainty derived from the predicted depth posterior (Xu et al., 22 Jun 2026).

CanonicalGS also predicts an appearance uncertainty from the dense feature map,

$\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$3

The final per-pixel reliability is the product of two exponentially decayed uncertainties,

$\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$4

The per-view evidence set is therefore

$\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$5

In the paper’s formulation, semantic features are not provided through a separate branch; rather, DINO-v2 features supply dense descriptors that are treated as semantically informative (Xu et al., 22 Jun 2026).

3. Canonical latent world

The second stage lifts the per-view evidence into a shared 3D scene representation. For each pixel $\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$6 in view $\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$7, the expected depth induces a 3D point

$\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$8

These lifted observations are rasterized onto a fixed voxel lattice $\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},$9 in shared world coordinates. The paper emphasizes that aggregation is performed on the voxelized scene lattice rather than directly on an irregular point set, so that the representation is not tied to incidental sampling density or view-dependent observation patterns (Xu et al., 22 Jun 2026).

The canonical latent world is

$I^{(i)} \in \mathbb{R}^{H \times W \times 3}$0

where $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$1 is a feature vector at voxel $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$2 and $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$3 is an accumulated reliability scalar. Spatial alignment is purely geometric: observations from different views that back-project into the same voxel are grouped together. If

$I^{(i)} \in \mathbb{R}^{H \times W \times 3}$4

then $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$5 contains all observations assigned to voxel $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$6 (Xu et al., 22 Jun 2026).

This makes CanonicalGS a hybrid explicit representation. Fusion happens on a dense voxelized latent world, while rendering happens from explicit Gaussian primitives decoded afterward. In that sense it is adjacent to canonicalized 3D Gaussian pipelines that normalize scene geometry into a shared bounded coordinate system before latent processing, such as Can3Tok, but the design objective is different: Can3Tok is a scene-level VAE for canonicalized 3DGS latent modeling, whereas CanonicalGS is a feed-forward multi-view reconstruction system built around uncertainty-aware scene fusion (Gao et al., 2 Aug 2025).

The default volume resolutions given in the implementation are $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$7 on RE10K and $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$8 on DL3DV (Xu et al., 22 Jun 2026).

4. Uncertainty-aware and consistency-aware fusion

The central technical contribution of CanonicalGS is its fusion rule inside the canonical voxel lattice. Rather than averaging all evidence in a voxel, the method first selects a representative observation and then weights the remaining observations by both reliability and feature consistency (Xu et al., 22 Jun 2026).

For each nonempty voxel $I^{(i)} \in \mathbb{R}^{H \times W \times 3}$9, the representative observation is the most reliable one: $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$0 Each observation $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$1 then receives weight

$\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$2

with $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$3. The scene reliability and scene feature fields are then

$\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$4

$\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$5

This construction has two coupled effects. First, uncertain observations are suppressed through $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$6. Second, observations whose features are inconsistent with the voxel’s representative evidence are also suppressed through the nonnegative cosine similarity gate. The result is not presented as a probabilistic posterior estimator; it is a deterministic fusion mechanism in which reliable, mutually consistent observations strengthen the latent scene state, while uncertain or contradictory ones contribute little (Xu et al., 22 Jun 2026).

The ablation results support the importance of both components. On DL3DV, the reference configuration reports $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$7 in PSNR/SSIM/LPIPS, whereas removing reliability gives $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$8, removing similarity gives $\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$9, and average merging gives $$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$0 (Xu et al., 22 Jun 2026). This indicates that the canonical world is not merely a coordinate transformation; its performance depends materially on uncertainty-aware and consistency-aware evidence consolidation.

5. Gaussian decoding, rendering, and training

Once the canonical latent world has been formed, CanonicalGS decodes it into a Gaussian set

$$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$1

where $$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$2 is the Gaussian mean, $$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$3 the covariance, $$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$4 the opacity, and $$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$5 the spherical harmonic appearance coefficients (Xu et al., 22 Jun 2026).

A distinctive design choice is that opacity is decoded from accumulated reliability through a monotone mapping. First,

$$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$6

then

$$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$7

The monotonicity constraint is implemented using nonnegative weights in the MLP head. This enforces the inductive bias that opacity should increase or saturate as scene support becomes stronger, rather than decrease (Xu et al., 22 Jun 2026).

The remaining Gaussian attributes are decoded from scene features: $$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$8 and the mean is anchored at the representative 3D point,

$$\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},$$9

The rendering stage is standard differentiable Gaussian splatting; the paper does not restate the full rasterization equations, but the novelty is explicitly placed in the representation and fusion stages rather than in a new renderer (Xu et al., 22 Jun 2026).

Training proceeds in two stages. First, the depth-related modules are pretrained by distilling Depth Anything V2 with an affine-invariant depth loss following Ranftl et al. Second, the full model is fine-tuned end-to-end using MSE and LPIPS, with

$$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$0

No separate semantic supervision, uncertainty calibration loss, consistency regularization loss, or end-to-end depth loss is introduced in the described final objective (Xu et al., 22 Jun 2026).

The implementation details reported include input resolution $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$1, RE10K training for 300,000 steps with batch size $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$2 on 4 NVIDIA RTX A6000 GPUs, DL3DV fine-tuning for 100,000 steps, and AdamW optimization. The learning rates are $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$3 for the ViT backbone, upsampling head, and UNet, and $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$4 for the remaining parameters, with cosine annealing, 2,000 warm-up steps, and minimum learning rate $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$5 (Xu et al., 22 Jun 2026).

6. Empirical behavior, comparisons, and limitations

CanonicalGS is evaluated primarily on RE10K and DL3DV, with ACID used for zero-shot transfer. The main evaluation varies the number of input views from $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$6 to $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$7, renders 50 target views per scene, and uses a target-view span that is 20% larger than the input-view span (Xu et al., 22 Jun 2026).

On RE10K, CanonicalGS improves as more views are added:

• $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$8 views: 24.22 / 0.840 / 0.164
• $$\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).$$9 views: 24.70 / 0.853 / 0.154
• $\mathcal{E}$0 views: 24.82 / 0.857 / 0.149
• $\mathcal{E}$1 views: 25.22 / 0.861 / 0.145

By contrast, several reported baselines degrade with more views, including MVSplat from 22.52 to 19.68 PSNR, DepthSplat from 24.16 to 22.00, and FreeSplat from 21.74 to 20.07 (Xu et al., 22 Jun 2026). On DL3DV, CanonicalGS likewise improves from 19.26 / 0.592 / 0.331 at $\mathcal{E}$2 views to 20.21 / 0.676 / 0.269 at $\mathcal{E}$3 views (Xu et al., 22 Jun 2026). The paper presents this as evidence that the canonical world stabilizes rather than destabilizes when observation count increases.

In the bounded-view RE10K protocol, CanonicalGS reports 27.36 / 0.886 / 0.114, compared with 26.84 / 0.878 / 0.122 for DepthSplat and 26.41 / 0.871 / 0.132 for FreeSplat (Xu et al., 22 Jun 2026). In zero-shot transfer from RE10K to ACID, CanonicalGS reports 28.47 / 0.859 / 0.140 with 46.5M parameters, exceeding the listed zero-shot baselines (Xu et al., 22 Jun 2026).

The representation is also evaluated beyond rendering. The paper measures feature stability by cosine similarity to a 12-view reference representation and reports stronger stability than competing feed-forward representations as views are added. For semantic transfer, it freezes rendered features, trains a linear semantic probe, and evaluates over 150 classes using pseudo-labels from Mask2Former; the abstract reports an 11% gain in semantic segmentation accuracy (Xu et al., 22 Jun 2026). Because the main training loop does not include semantic supervision, this suggests that the canonical latent world preserves scene information useful beyond photometric rendering.

Runtime and memory statistics are also reported. On DL3DV with $\mathcal{E}$4 images, $\mathcal{E}$5 input views, $\mathcal{E}$6 target views, and batch size $\mathcal{E}$7, CanonicalGS reports 555.6 FPS, 9.98 GB GPU memory, 172.3K Gaussian primitives, and 19.78 PSNR (Xu et al., 22 Jun 2026). The efficiency argument is that evidence is merged before Gaussian decoding, reducing the need to generate and then prune redundant primitives.

The paper explicitly identifies several limitations. CanonicalGS assumes known camera projections, depends on reasonably accurate poses and depth quality, remains vulnerable to heavy occlusion and pose errors, is not designed for dynamic objects, and uses a fixed dense voxel lattice that may be less suitable for large-scale scenes (Xu et al., 22 Jun 2026). The uncertainty representation is also limited to a scalar reliability signal. The authors suggest richer uncertainty modeling, joint canonicalization of depth, pose, visibility, and scene evidence, and replacement of the fixed voxel lattice by adaptive sparse or hierarchical scene fields (Xu et al., 22 Jun 2026).

Relative to adjacent work, CanonicalGS is most closely contrasted with view-centric feed-forward Gaussian splatting systems such as PixelSplat, MVSplat, and DepthSplat, and more broadly with token-based scene-centric predictors such as TokenGS. TokenGS also argues against pixel-aligned Gaussian prediction and instead predicts a scene-level set of primitives from learned tokens, but its main structural move is token-based decoupling from image resolution and view count, whereas CanonicalGS places the scene bottleneck in a voxelized canonical world with uncertainty-aware fusion (Ren et al., 16 Apr 2026). A plausible synthesis is that both methods reject direct view-conditioned Gaussian decoding, but they do so through different canonicalization strategies: CanonicalGS through explicit world-aligned evidence fusion, and TokenGS through scene-level latent tokens.

CanonicalGS is therefore best characterized as a representation-first feed-forward Gaussian splatting system. Its distinctive claim is not that 3DGS rendering itself is altered, but that robust novel view synthesis and downstream scene understanding benefit from constructing a stable, reliability-weighted canonical scene representation before any Gaussian primitives are emitted (Xu et al., 22 Jun 2026).