---
title: 'CanonicalGS: Stable 3D Gaussian Splatting'
url: https://www.emergentmind.com/topics/canonicalgs
type: topic
---

# CanonicalGS: Stable 3D Gaussian Splatting

CanonicalGS is a feed-forward 3D Gaussian splatting method for novel view synthesis that constructs a **stable, scene-centric representation** before decoding renderable Gaussian primitives. Rather than predicting Gaussians directly from view-aligned evidence, it first maps posed multi-view observations into a **canonical latent world** defined in shared scene coordinates, then decodes that latent world into 3D Gaussian primitives for differentiable Gaussian splatting. The method is motivated by the observation that many feed-forward Gaussian splatting pipelines remain effectively view-dependent: as more input views are added, they may accumulate noisy or redundant evidence instead of converging to a stable scene estimate. CanonicalGS addresses this through uncertainty-aware, consistency-aware fusion in a voxelized canonical scene space, yielding improved multi-view scalability for novel view synthesis and stronger transfer to downstream perception tasks. The reported results show up to a \(2.5\) dB improvement in peak signal-to-noise ratio and an \(11\%\) gain in semantic segmentation accuracy [2606.23027].

## 1. Concept and problem formulation

CanonicalGS is framed as a criticism of prior **feed-forward Gaussian splatting (FFGS)** systems that are fast at inference but remain tightly coupled to view-aligned or pixel-aligned predictions. In that regime, additional views do not necessarily stabilize the reconstructed scene; they can instead introduce redundant primitive hypotheses, conflicting geometric estimates, and view-conditioned clutter. CanonicalGS proposes that a feed-forward scene representation should improve by **converging** as more valid views are observed, rather than by merely producing more primitives [2606.23027].

Its central representational object is the **canonical latent world**, described as a stable, shared scene/world representation rather than a collection of independent per-view predictions. The input is an unordered set of posed RGB views,
\[
\mathcal{D}=\{(I^{(i)},\Pi^{(i)})\}_{i=1}^{N},
\]
where \(I^{(i)} \in \mathbb{R}^{H \times W \times 3}\) is an image and \(\Pi^{(i)}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}\) is the known perspective projection. The overall feed-forward pipeline is written as
\[
\mathcal{Z} = \operatorname{Agg}\!\left(\mathcal{E} \right),\qquad \mathcal{E} = \left\{\operatorname{Ext}\left(I^{(i)},\Pi^{(i)}\right)\right\}_{i=1}^{N},
\]
followed by Gaussian decoding,
\[
\mathcal{G} = \operatorname{Dec}\left(\mathcal{Z}\right).
\]
Here \(\mathcal{E}\) denotes **view-centric evidence**, while \(\mathcal{Z}\) denotes the fused **scene-centric** latent representation [2606.23027].

The “canonical” designation in this paper refers to a shared scene/world coordinate system induced by the known camera geometry and the voxelized latent scene lattice. It does **not** introduce a separate learned canonical object frame with explicit deformation into observed states. In that respect, CanonicalGS differs from canonicalized dynamic Gaussian representations and from latent token methods such as TokenGS, whose “canonical coordinate space” refers to global 3D means in a shared frame rather than to a latent scene warped across time [2604.15239]. A plausible implication is that CanonicalGS is best understood as a **world-aligned canonicalization of evidence**, not as a deformation-based canonical scene model.

## 2. View-centric evidence extraction

CanonicalGS begins by extracting dense per-view evidence independently, before any cross-view fusion. For each input image \(I^{(i)}\), a pretrained **DINO-v2 backbone** produces patch-level visual features, and a lightweight upsampling decoder restores dense feature maps at image resolution. This yields
\[
F^{(i)}_\mathrm{view} \in \mathbb{R}^{H\times W\times L}.
\]
These features are used jointly for depth estimation, appearance representation, and uncertainty prediction [2606.23027].

Depth is estimated through a multi-view cost-volume mechanism. For a reference view \(i\), with sampled depth hypotheses \(\{d_k\}_{k=1}^{K}\), a pixel \(m\) is hypothesized at depth \(d_k\), back-projected to 3D, and reprojected into source view \(j\):
\[
m^{(i,j)}_k = \Pi^{(j)}\!\left(\left(\Pi^{(i)}\right)^{-1}(m,d_k)\right).
\]
The pairwise matching score is the normalized feature correlation
\[
S_k^{(i,j)}(m) = \frac{ \left\langle F^{(i)}_\mathrm{view}(m), F^{(j)}_\mathrm{view}(m^{(i,j)}_k) \right\rangle }{\sqrt{L}},
\]
and the reference-view score volume averages over all other inputs,
\[
S_k^{(i)}(m)=\frac{1}{N-1}\sum_{j\neq i}S_k^{(i,j)}(m).
\]
A **UNet-like** module processes the feature map and score volume to predict a discrete depth probability volume,
\[
P^{(i)} = \operatorname{softmax} \left( \operatorname{UNet}\left(F^{(i)}_\mathrm{view},S^{(i)}\right) \right)\in\mathbb{R}^{H\times W\times K}.
\]
Expected depth and positional uncertainty are then defined as
\[
D^{(i)}(m)=\sum_{k=1}^{K} P_k^{(i)}(m)\,d_k,
\]
\[
U^{(i)}_\mathrm{pos}(m) = \sqrt{\sum_{k=1}^{K} P_k^{(i)}(m)\left(d_k-D^{(i)}(m)\right)^2}.
\]
This \(U_\mathrm{pos}\) term is a depth-distribution uncertainty derived from the predicted depth posterior [2606.23027].

CanonicalGS also predicts an appearance uncertainty from the dense feature map,
\[
U^{(i)}_{\mathrm{app}}(m)= \operatorname{ReLU} \left( \operatorname{MLP}\left(F^{(i)}_\mathrm{view}(m)\right) \right).
\]
The final per-pixel reliability is the product of two exponentially decayed uncertainties,
\[
R_{\mathrm{view}}^{(i)} = \exp\left(-U^{(i)}_{\mathrm{pos}}\right) \odot \exp\left(-U^{(i)}_{\mathrm{app}}\right).
\]
The per-view evidence set is therefore
\[
\mathcal{E} =\left\{F^{(i)}_\mathrm{view},D^{(i)},R_{\mathrm{view}}^{(i)}\right\}_{i=1}^N.
\]
In the paper’s formulation, semantic features are not provided through a separate branch; rather, DINO-v2 features supply dense descriptors that are treated as semantically informative [2606.23027].

## 3. Canonical latent world

The second stage lifts the per-view evidence into a shared 3D scene representation. For each pixel \(m\) in view \(i\), the expected depth induces a 3D point
\[
x^{(i)}(m) = \left(\Pi^{(i)}\right)^{-1} \!\left(m,D^{(i)}(m)\right).
\]
These lifted observations are rasterized onto a fixed voxel lattice \(\mathcal{V}\) in shared world coordinates. The paper emphasizes that aggregation is performed on the voxelized scene lattice rather than directly on an irregular point set, so that the representation is not tied to incidental sampling density or view-dependent observation patterns [2606.23027].

The canonical latent world is
\[
\mathcal{Z} = \{F_{\mathrm{scene}}, R_{\mathrm{scene}}\},
\]
where \(F_{\mathrm{scene}}(v)\) is a feature vector at voxel \(v\) and \(R_{\mathrm{scene}}(v)\) is an accumulated reliability scalar. Spatial alignment is purely geometric: observations from different views that back-project into the same voxel are grouped together. If
\[
\Omega(v)=\{(i,m)\mid x^{(i)}(m)\in v\},
\]
then \(\Omega(v)\) contains all observations assigned to voxel \(v\) [2606.23027].

This makes CanonicalGS a hybrid explicit representation. Fusion happens on a dense voxelized latent world, while rendering happens from explicit Gaussian primitives decoded afterward. In that sense it is adjacent to canonicalized 3D Gaussian pipelines that normalize scene geometry into a shared bounded coordinate system before latent processing, such as Can3Tok, but the design objective is different: Can3Tok is a scene-level VAE for canonicalized 3DGS latent modeling, whereas CanonicalGS is a feed-forward multi-view reconstruction system built around uncertainty-aware scene fusion [2508.01464].

The default volume resolutions given in the implementation are \([1024, 1024, 512]\) on RE10K and \([768, 768, 384]\) on DL3DV [2606.23027].

## 4. Uncertainty-aware and consistency-aware fusion

The central technical contribution of CanonicalGS is its fusion rule inside the canonical voxel lattice. Rather than averaging all evidence in a voxel, the method first selects a representative observation and then weights the remaining observations by both **reliability** and **feature consistency** [2606.23027].

For each nonempty voxel \(v\), the representative observation is the most reliable one:
\[
(i^\star,m^\star) = \operatorname*{arg\,max}_{(i,m)\in\Omega(v)} R_{\mathrm{view}}^{(i)}(m).
\]
Each observation \((i,m)\in \Omega(v)\) then receives weight
\[
W^{(i)}(m,v) = R_{\mathrm{view}}^{(i)}(m) \left[ \frac{ \left\langle F^{(i)}_\mathrm{view}(m), F^{(i^\star)}_\mathrm{view}(m^\star)\right\rangle }{ \|F^{(i)}_\mathrm{view}(m)\|\,\|F^{(i^\star)}_\mathrm{view}(m^\star)\| } \right]_+,
\]
with \([\cdot]_+=\max(\cdot,0)\). The scene reliability and scene feature fields are then
\[
R_{\mathrm{scene}}(v) =  \sum_{(i,m)\in\Omega(v)} W^{(i)}(m,v),
\]
\[
F_\mathrm{scene}(v) = \frac{ \sum_{(i,m)\in\Omega(v)} W^{(i)}(m,v)F^{(i)}_\mathrm{view}(m) }{ R_{\mathrm{scene}}(v)+\epsilon }.
\]

This construction has two coupled effects. First, uncertain observations are suppressed through \(R_{\mathrm{view}}^{(i)}\). Second, observations whose features are inconsistent with the voxel’s representative evidence are also suppressed through the nonnegative cosine similarity gate. The result is not presented as a probabilistic posterior estimator; it is a deterministic fusion mechanism in which reliable, mutually consistent observations strengthen the latent scene state, while uncertain or contradictory ones contribute little [2606.23027].

The ablation results support the importance of both components. On DL3DV, the reference configuration reports \(19.78 / 0.665 / 0.284\) in PSNR/SSIM/LPIPS, whereas removing reliability gives \(18.65 / 0.592 / 0.314\), removing similarity gives \(19.20 / 0.623 / 0.298\), and average merging gives \(18.58 / 0.590 / 0.306\) [2606.23027]. This indicates that the canonical world is not merely a coordinate transformation; its performance depends materially on uncertainty-aware and consistency-aware evidence consolidation.

## 5. Gaussian decoding, rendering, and training

Once the canonical latent world has been formed, CanonicalGS decodes it into a Gaussian set
\[
\mathcal{G}=\{\mu(v),\Sigma(v),\alpha(v),h(v)\}_{v\in\mathcal{V}},
\]
where \(\mu(v)\) is the Gaussian mean, \(\Sigma(v)\) the covariance, \(\alpha(v)\) the opacity, and \(h(v)\) the spherical harmonic appearance coefficients [2606.23027].

A distinctive design choice is that opacity is decoded from accumulated reliability through a monotone mapping. First,
\[
r(v)=1-\exp(-R_{\mathrm{scene}}(v)),
\]
then
\[
\alpha(v)=\phi(r(v)) = \operatorname{Sigmoid}\left(\operatorname{MLP}\left(r(v)\right)\right), \qquad \frac{\partial \phi(r)}{\partial r}\ge 0.
\]
The monotonicity constraint is implemented using nonnegative weights in the MLP head. This enforces the inductive bias that opacity should increase or saturate as scene support becomes stronger, rather than decrease [2606.23027].

The remaining Gaussian attributes are decoded from scene features:
\[
\delta(v),\ \Sigma(v),\ h(v) = \psi\!\left(F_{\mathrm{scene}}(v)\right),
\]
and the mean is anchored at the representative 3D point,
\[
\mu(v)=x^{(i^\star)}(m^\star)+\delta(v).
\]
The rendering stage is standard differentiable Gaussian splatting; the paper does not restate the full rasterization equations, but the novelty is explicitly placed in the representation and fusion stages rather than in a new renderer [2606.23027].

Training proceeds in two stages. First, the depth-related modules are pretrained by distilling **Depth Anything V2** with an affine-invariant depth loss following Ranftl et al. Second, the full model is fine-tuned end-to-end using **MSE** and **LPIPS**, with
\[
\lambda_{\mathrm{LPIPS}} = 0.05.
\]
No separate semantic supervision, uncertainty calibration loss, consistency regularization loss, or end-to-end depth loss is introduced in the described final objective [2606.23027].

The implementation details reported include input resolution \(256\times256\), RE10K training for **300,000 steps** with batch size \(2\) on **4 NVIDIA RTX A6000 GPUs**, DL3DV fine-tuning for **100,000 steps**, and **AdamW** optimization. The learning rates are \(10^{-6}\) for the ViT backbone, upsampling head, and UNet, and \(10^{-4}\) for the remaining parameters, with cosine annealing, **2,000 warm-up steps**, and minimum learning rate \(10^{-8}\) [2606.23027].

## 6. Empirical behavior, comparisons, and limitations

CanonicalGS is evaluated primarily on **RE10K** and **DL3DV**, with **ACID** used for zero-shot transfer. The main evaluation varies the number of input views from \(2\) to \(8\), renders **50 target views** per scene, and uses a target-view span that is **20% larger than the input-view span** [2606.23027].

On RE10K, CanonicalGS improves as more views are added:
- \(2\) views: **24.22 / 0.840 / 0.164**
- \(4\) views: **24.70 / 0.853 / 0.154**
- \(6\) views: **24.82 / 0.857 / 0.149**
- \(8\) views: **25.22 / 0.861 / 0.145**

By contrast, several reported baselines degrade with more views, including **MVSplat** from **22.52** to **19.68** PSNR, **DepthSplat** from **24.16** to **22.00**, and **FreeSplat** from **21.74** to **20.07** [2606.23027]. On DL3DV, CanonicalGS likewise improves from **19.26 / 0.592 / 0.331** at \(2\) views to **20.21 / 0.676 / 0.269** at \(8\) views [2606.23027]. The paper presents this as evidence that the canonical world stabilizes rather than destabilizes when observation count increases.

In the bounded-view RE10K protocol, CanonicalGS reports **27.36 / 0.886 / 0.114**, compared with **26.84 / 0.878 / 0.122** for DepthSplat and **26.41 / 0.871 / 0.132** for FreeSplat [2606.23027]. In zero-shot transfer from RE10K to ACID, CanonicalGS reports **28.47 / 0.859 / 0.140** with **46.5M** parameters, exceeding the listed zero-shot baselines [2606.23027].

The representation is also evaluated beyond rendering. The paper measures feature stability by cosine similarity to a **12-view reference** representation and reports stronger stability than competing feed-forward representations as views are added. For semantic transfer, it freezes rendered features, trains a linear semantic probe, and evaluates over **150 classes** using pseudo-labels from **Mask2Former**; the abstract reports an **11% gain in semantic segmentation accuracy** [2606.23027]. Because the main training loop does not include semantic supervision, this suggests that the canonical latent world preserves scene information useful beyond photometric rendering.

Runtime and memory statistics are also reported. On DL3DV with \(256\times256\) images, \(4\) input views, \(50\) target views, and batch size \(1\), CanonicalGS reports **555.6 FPS**, **9.98 GB** GPU memory, **172.3K** Gaussian primitives, and **19.78** PSNR [2606.23027]. The efficiency argument is that evidence is merged **before** Gaussian decoding, reducing the need to generate and then prune redundant primitives.

The paper explicitly identifies several limitations. CanonicalGS assumes **known camera projections**, depends on reasonably accurate poses and depth quality, remains vulnerable to heavy occlusion and pose errors, is not designed for **dynamic objects**, and uses a fixed dense voxel lattice that may be less suitable for large-scale scenes [2606.23027]. The uncertainty representation is also limited to a scalar reliability signal. The authors suggest richer uncertainty modeling, joint canonicalization of depth, pose, visibility, and scene evidence, and replacement of the fixed voxel lattice by **adaptive sparse or hierarchical scene fields** [2606.23027].

Relative to adjacent work, CanonicalGS is most closely contrasted with view-centric feed-forward Gaussian splatting systems such as PixelSplat, MVSplat, and DepthSplat, and more broadly with token-based scene-centric predictors such as TokenGS. TokenGS also argues against pixel-aligned Gaussian prediction and instead predicts a scene-level set of primitives from learned tokens, but its main structural move is token-based decoupling from image resolution and view count, whereas CanonicalGS places the scene bottleneck in a voxelized canonical world with uncertainty-aware fusion [2604.15239]. A plausible synthesis is that both methods reject direct view-conditioned Gaussian decoding, but they do so through different canonicalization strategies: CanonicalGS through explicit world-aligned evidence fusion, and TokenGS through scene-level latent tokens.

CanonicalGS is therefore best characterized as a **representation-first** feed-forward Gaussian splatting system. Its distinctive claim is not that 3DGS rendering itself is altered, but that robust novel view synthesis and downstream scene understanding benefit from constructing a stable, reliability-weighted canonical scene representation before any Gaussian primitives are emitted [2606.23027].

Source: https://www.emergentmind.com/topics/canonicalgs