SplatWeaver: Adaptive Gaussian Scene Synthesis

• SplatWeaver is a generalizable framework that predicts camera poses and allocates a variable number of 3D Gaussian primitives based on scene complexity.
• It employs cardinality Gaussian experts with pixel-level routing and a wavelet-based high-frequency prior to concentrate resources on intricate regions.
• Empirical results demonstrate enhanced rendering quality and efficiency, achieving higher PSNR/SSIM with notably fewer Gaussians compared to prior methods.

SplatWeaver is a generalizable novel view synthesis framework that predicts camera parameters and a compact set of 3D Gaussian primitives from a handful of unposed input images in a single feed-forward pass, without requiring per-scene optimization (Wan et al., 8 May 2026). Its defining feature is dynamic, region-dependent allocation of Gaussian primitives: instead of imposing a fixed per-pixel or per-voxel primitive budget, it learns to place Gaussians densely in complex regions and sparsely in smooth ones. In the formulation reported for the method, this adaptive allocation is implemented through cardinality Gaussian experts, pixel-level routing, a wavelet-derived high-frequency prior, and local neighbor aggregation before standard forward 3D Gaussian splatting (Wan et al., 8 May 2026).

1. Problem setting and representational objective

SplatWeaver addresses the setting in which a model receives $N$ uncalibrated images $\{I_n\}_{n=1}^N$ of a scene and must estimate both camera poses $\{p_n\}$ and a renderable Gaussian scene representation in one pass (Wan et al., 8 May 2026). The predicted scene is a set of anisotropic Gaussians

$$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$

where the parameters correspond to position, scale, rotation, opacity, and color spherical-harmonics, respectively (Wan et al., 8 May 2026).

The motivating limitation is the fixed-budget strategy used by prior generalizable splatting methods. In that regime, every pixel or voxel receives the same number of Gaussian primitives. The paper argues that this uniform allocation over-represents smooth areas while under-fitting edges, textured regions, fine structures, and complex geometry (Wan et al., 8 May 2026). It further distinguishes adaptive allocation from pruning or early truncation: those strategies can reduce total primitive count, but they do not reallocate capacity adaptively across the scene (Wan et al., 8 May 2026).

This framing places SplatWeaver within feed-forward 3D Gaussian Splatting for zero-shot novel view synthesis, but with a representational objective that is explicitly complexity-aware. A plausible implication is that the method treats primitive count itself as a learnable, spatially varying latent variable rather than a globally fixed design choice.

2. Cardinality Gaussian experts and pixel-level routing

The core mechanism is a set of $M+1$ cardinality Gaussian experts $\{E_e\}_{e=0}^M$, where expert $E_e$ is specialized to predict exactly $e$ hidden Gaussians from a pixel feature, with $e\in\{0,1,\dots,M\}$ and, in practice, $M=3$ (Wan et al., 8 May 2026). The null expert $\{I_n\}_{n=1}^N$0 produces no Gaussians and thereby enforces sparsity where appropriate (Wan et al., 8 May 2026).

For a pixel at spatial location $\{I_n\}_{n=1}^N$1 with feature $\{I_n\}_{n=1}^N$2, the router network $\{I_n\}_{n=1}^N$3 produces logits $\{I_n\}_{n=1}^N$4, converted to soft-assignment weights

$\{I_n\}_{n=1}^N$5

with temperature $\{I_n\}_{n=1}^N$6 (Wan et al., 8 May 2026). During training, the expert selection is sampled as a one-hot $\{I_n\}_{n=1}^N$7 with a straight-through estimator, while the soft weights $\{I_n\}_{n=1}^N$8 are retained for regularization (Wan et al., 8 May 2026).

Once an expert $\{I_n\}_{n=1}^N$9 is selected, it predicts

$\{p_n\}$0

that is, exactly $\{p_n\}$1 3D positions $\{p_n\}$2 and latent features $\{p_n\}$3 (Wan et al., 8 May 2026). This expert design operationalizes primitive cardinality as a discrete routing decision. The paper’s qualitative interpretation is “dense-where-complex, sparse-where-smooth,” with low-cardinality experts tending toward flat walls and sky regions, and high-cardinality experts tending toward edges, textured fabrics, and foliage (Wan et al., 8 May 2026).

A common misconception is that this is merely a variable-output decoder. In the reported formulation, the more specific mechanism is a mixture-of-experts routing paradigm in which each expert is trained to specialize in a fixed primitive count, and the router determines which count is appropriate at each spatial location (Wan et al., 8 May 2026).

3. High-frequency prior and routing regularization

To bias routing toward textured and structurally detailed regions, SplatWeaver computes a per-pixel high-frequency energy map using a single-level discrete wavelet transform:

$\{p_n\}$4

where $\{p_n\}$5 denotes upsampling by factor $\{p_n\}$6 (Wan et al., 8 May 2026).

The method introduces a small frequency prior guidance module $\{p_n\}$7 that ingests $\{p_n\}$8 and produces an attention mask that modulates encoder features:

$\{p_n\}$9

The resulting frequency-aware feature $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$0 is then supplied to the router (Wan et al., 8 May 2026). The stated purpose is to encourage more Gaussian primitives in fine structures, complex geometry, and textured regions while suppressing redundant primitives in smooth areas (Wan et al., 8 May 2026).

Routing is additionally stabilized by a ranking-based cross-entropy loss. Pixels are sorted by $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$1, with the top $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$2 assigned to expert $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$3, the next $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$4 to $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$5, the next $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$6 to $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$7, and the remainder to $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$8 (Wan et al., 8 May 2026). Using one-hot soft labels $$\mathcal{G}=\bigl\{(\mu^{(g)},s^{(g)},q^{(g)},\alpha^{(g)},c^{(g)})\bigr\}_{g=1}^G,$$9 and smoothed labels

$M+1$0

the loss is

$M+1$1

A global budget regularizer,

$M+1$2

discourages the total number $M+1$3 of predicted Gaussians from exceeding a fraction $M+1$4 of pixels (Wan et al., 8 May 2026). During the second half of training, $M+1$5 is turned off so that the model can discover the routing strategy under the budget constraint more freely (Wan et al., 8 May 2026).

This design distinguishes two roles for the prior. First, the wavelet-derived signal provides a structural bias. Second, the regularization regime prevents unstable or degenerate expert selection early in training. The ablations summarized in the paper indicate that removing frequency guidance leads to uneven Gaussian clustering and softened details (Wan et al., 8 May 2026).

4. Architecture and data flow

The network begins by tokenizing each input image with DINOv2 and processing the tokens with a multi-view geometry transformer, VGGT, which estimates poses $M+1$6 and produces pixel-level features $M+1$7 (Wan et al., 8 May 2026). A DPT-style decoder then upsamples these to full-resolution $M+1$8 latent maps (Wan et al., 8 May 2026).

In parallel, discrete-wavelet high-frequency maps are computed and passed through the guidance module $M+1$9, which modulates the latent features to obtain $\{E_e\}_{e=0}^M$0 (Wan et al., 8 May 2026). The router $\{E_e\}_{e=0}^M$1 operates on $\{E_e\}_{e=0}^M$2 and selects one cardinality expert $\{E_e\}_{e=0}^M$3, which predicts $\{E_e\}_{e=0}^M$4 hidden Gaussians $\{E_e\}_{e=0}^M$5 for that location (Wan et al., 8 May 2026).

Each hidden Gaussian is linked back to the corresponding projected pixel feature

$\{E_e\}_{e=0}^M$6

forming triplets $\{E_e\}_{e=0}^M$7 (Wan et al., 8 May 2026). A fast K-NN lookup identifies the $\{E_e\}_{e=0}^M$8 spatial neighbors of each Gaussian, and a point-transformer attention module aggregates neighbor information:

$\{E_e\}_{e=0}^M$9

$E_e$0

A residual MLP head applied to $E_e$1 decodes the final Gaussian parameters: scale $E_e$2, rotation $E_e$3, opacity $E_e$4, and color spherical-harmonics $E_e$5 (Wan et al., 8 May 2026). The complete set $E_e$6 is then rendered by standard forward 3D Gaussian splatting (Wan et al., 8 May 2026).

The architectural significance lies in how primitive allocation and primitive parameterization are separated. Routing determines cardinality at the pixel level, while neighbor-conditioned decoding refines the predicted Gaussians using local spatial context. The paper reports that this neighbor-conditioned decoding sharpens high-frequency structures further in ablations (Wan et al., 8 May 2026).

5. Training objective and supervisory signals

Beyond routing and budget regularization, SplatWeaver uses camera-pose distillation and sparse depth supervision from a pre-trained VGGT (Wan et al., 8 May 2026). The pose term $E_e$7 is a Huber loss, and the sparse depth term $E_e$8 is an MSE loss (Wan et al., 8 May 2026).

Image reconstruction is optimized with

$E_e$9

The full objective is

$e$0

with

$e$1

All of these terms are explicitly specified in the reported formulation (Wan et al., 8 May 2026).

The supervisory design makes clear that SplatWeaver is not only a renderer but also a joint pose-and-geometry predictor. The presence of pose distillation and sparse depth supervision suggests that stable feed-forward novel view synthesis from uncalibrated images is treated as a multi-objective estimation problem rather than purely an image reconstruction problem.

6. Empirical performance, qualitative behavior, and interpretive boundaries

Across three zero-shot benchmarks—DL3DV, RealEstate10K, and Mip-NeRF 360—and view counts $e$2, SplatWeaver is reported to consistently outperform prior feed-forward splatting methods while using $e$3–$e$4 fewer Gaussians (Wan et al., 8 May 2026). In the 16-view DL3DV setting, it achieves PSNR $e$5, SSIM $e$6, and LPIPS $e$7 with approximately $e$8 Gaussians, compared with AnySplat at $e$9 and $e\in\{0,1,\dots,M\}$0 Gaussians (Wan et al., 8 May 2026). On RealEstate10K with 16 views, it reaches $e\in\{0,1,\dots,M\}$1 with $e\in\{0,1,\dots,M\}$2K Gaussians, versus AnySplat’s $e\in\{0,1,\dots,M\}$3 at $e\in\{0,1,\dots,M\}$4M (Wan et al., 8 May 2026). The paper also reports an “extreme-efficiency” variant, SplatWeaver†, using only $e\in\{0,1,\dots,M\}$5K Gaussians while matching or exceeding prior art (Wan et al., 8 May 2026).

Qualitative comparisons attribute the performance gains to the allocation pattern itself. Flat walls and sky regions are assigned few Gaussians, often through experts $e\in\{0,1,\dots,M\}$6 or $e\in\{0,1,\dots,M\}$7, whereas edges, textured fabrics, and foliage receive higher-cardinality experts (Wan et al., 8 May 2026). The paper describes the resulting renderings as having crisper geometry and more faithful textures, with examples including sharper window frames and finer surface details (Wan et al., 8 May 2026).

The main interpretive boundary is that SplatWeaver is not presented as a generic increase in model size or a simple post hoc pruning strategy. Its central claim is that adaptive primitive cardinality is the decisive mechanism. The paper explicitly argues that uniform allocation wastes capacity in smooth regions and under-serves high-frequency content, while pruning-only schemes control count without scene-adaptive redistribution (Wan et al., 8 May 2026). This suggests that SplatWeaver’s contribution is best understood as a representational allocation strategy layered onto feed-forward Gaussian scene prediction, rather than merely a more efficient implementation of fixed-budget splatting.