Token-Based 3D Gaussian Splatting

• The paper introduces token-based methods that align and fuse neural tokens to decode explicit 3D Gaussian primitives for compact, efficient scene reconstruction.
• Token-based 3D Gaussian splatting leverages cross-attentional feature alignment and dual-flow decoding to disentangle geometric structure from appearance across views.
• Both TokenSplat and GlobalSplat demonstrate competitive novel view synthesis and pose estimation performance while reducing redundancy compared to dense, pixel-aligned approaches.

Token-based 3D Gaussian splatting refers to a family of feed-forward methods for scene reconstruction and novel view synthesis that replace traditional pixel/voxel-aligned primitives with a set of neural tokens, which are aligned, grouped, and fused across multiple unposed views prior to decoding explicit 3D Gaussian primitives. These approaches, exemplified by TokenSplat and GlobalSplat, enable globally consistent, compact 3D representations, support joint camera pose estimation in pose-free settings, and facilitate real-time inference while mitigating the redundancy endemic to dense pixel-based pipelines (Li et al., 28 Feb 2026, Itkin et al., 16 Apr 2026). The core advance is the direct alignment of features or tokens corresponding to the same semantic scene points across views within the latent feature space, followed by fine-grained decoding into spatially distributed Gaussians suitable for physically motivated differentiable rendering.

1. Architectural Foundations and Token Alignment

Token-based 3D Gaussian splatting is predicated on learning and manipulating abstract tokens—vector embeddings corresponding to image patches or learned scene latents—rather than traditional dense per-pixel or voxel parametric primitives. In TokenSplat (Li et al., 28 Feb 2026), a shared Vision Transformer (ViT) encoder converts each input RGB image $I_i$ into a sequence of image tokens $t_i^I \in \mathbb{R}^{L\times D}$, optionally augmented with camera intrinsic codes.

Tokens corresponding across different views are aligned directly in latent space using cross-attentional modules. TokenSplat employs a reference-view decoder to aggregate information into a canonical scene representation. The Asymmetric Dual-Flow Decoder (ADF-Decoder) further disentangles camera pose and semantic scene features by maintaining parallel token streams: an image token stream and a learnable camera token for each view. The ADF-Decoder enforces directionally-constrained attention—self-attention among image tokens, camera token attending to its local image tokens, and carefully gated cross-view attention—enabling disambiguation of geometric and appearance information without iterative optimization or bundle adjustment.

In GlobalSplat (Itkin et al., 16 Apr 2026), a small set of global scene tokens $\{l_j\in\mathbb{R}^d\}_{j=1}^M$ serve as the sole communication bottleneck between all multi-view inputs. Each scene token cross-attends to all per-view patch embeddings and then self-attends to other scene tokens, driving the concurrent learning of spatial and contextual correspondences before any explicit geometry is instantiated.

2. Mathematical Basis: 3D Gaussian Primitives and Differentiable Rendering

Both approaches produce explicit 3D Gaussian primitives parameterized as follows:

• Center: $\mu_g \in \mathbb{R}^3$
• Covariance: $$\Sigma_g = R(r_g)\,\mathrm{diag}(s_g^2)\,R(r_g)^\top$$ with rotation quaternion $r_g$ and scale $s_g$
• Opacity (density amplitude): $\alpha_g > 0$
• Appearance: SH coefficients $c_g\in\mathbb{R}^{3\times (k+1)^2}$ (typically $k=3$ or $t_i^I \in \mathbb{R}^{L\times D}$0)

The Gaussian basis function at point $t_i^I \in \mathbb{R}^{L\times D}$1 is

$t_i^I \in \mathbb{R}^{L\times D}$2

Volume rendering along ray $t_i^I \in \mathbb{R}^{L\times D}$3 yields

$t_i^I \in \mathbb{R}^{L\times D}$4

with density $t_i^I \in \mathbb{R}^{L\times D}$5. In practice, rendering proceeds via pre-integrating each Gaussian’s contribution along the ray: $t_i^I \in \mathbb{R}^{L\times D}$6 where $t_i^I \in \mathbb{R}^{L\times D}$7 is the ray integral for Gaussian $t_i^I \in \mathbb{R}^{L\times D}$8 and $t_i^I \in \mathbb{R}^{L\times D}$9 are real-valued spherical harmonics basis functions (Li et al., 28 Feb 2026, Itkin et al., 16 Apr 2026).

3. Token-Guided Prediction and Fusion of Gaussians

Central to token-based splatting is aggregating tokens that correspond to the same physical points in the scene, fusing their features, and decoding these into a compact, non-redundant set of Gaussians:

• In TokenSplat, the Token-aligned Gaussian Prediction (TAGP) module predicts, for each per-view token, a coarse 3D position and a fusion confidence. Tokens within a spatial neighborhood ($\{l_j\in\mathbb{R}^d\}_{j=1}^M$0-ball) are grouped; their features and positions are fused using softmax-weighted averaging:

$\{l_j\in\mathbb{R}^d\}_{j=1}^M$1

Each fused token is decoded via a multi-scale fusion of decoder features, yielding a one-to-many mapping to sets of explicit 3D Gaussians (Li et al., 28 Feb 2026).

• GlobalSplat formulates token fusion via global scene tokens, which, after cross-view alignment, decode their states into $\{l_j\in\mathbb{R}^d\}_{j=1}^M$2 candidate Gaussians per token. A coarse-to-fine curriculum merges these candidates into a reduced number of splats per token as training proceeds. Weighted merging by latent gating scores ensures that only the most semantically meaningful and non-redundant points generate explicit primitives (Itkin et al., 16 Apr 2026).

This aggregation suppresses overlapping and redundant Gaussians, which frequently debilitate pixel- or voxel-aligned feed-forward pipelines.

4. Differentiable Pose Estimation with Tokenized Factorization

TokenSplat supports fully feed-forward camera pose estimation from unposed multi-view images. Each non-reference view features a learnable camera token $\{l_j\in\mathbb{R}^d\}_{j=1}^M$3, which, after several ADF-Decoder blocks, is mapped by a small MLP into predicted pose parameters as a unit dual quaternion. The ADF-Decoder’s design maintains strict causal flows: image tokens are modulated by camera tokens via pre/post-modulation, but cross-talk is directionally constrained to prevent entangling viewpoint cues with semantic scene content. The resulting factorization supports stable joint optimization of geometry and extrinsics in a single inference pass, with pose loss terms enforcing MSE on extrinsics and dual-quaternion alignment (Li et al., 28 Feb 2026).

GlobalSplat presupposes known camera parameters and focuses on geometric-appearance factorization within global tokens, using camera codes in input embeddings but not updating camera pose.

5. Training Objectives and Losses

Token-based splatting methods rely on composite objectives:

• Image rendering loss: $\{l_j\in\mathbb{R}^d\}_{j=1}^M$4, supervising appearance and realism of novel view synthesis.
• Pose loss (TokenSplat): MSE on extrinsics plus dual-quaternion alignment.
• Self-supervised consistency loss (GlobalSplat): Cross-subset rendered opacity and depth consistency.
• Regularization: Priors on frustum membership, Gaussian scales, opacity, SH coefficients.
• Coarse-to-fine curriculum (GlobalSplat): Exposure of increasing splat multiplicity per token, from 1 to 8, to avoid representation bloat and achieve stable, compact decodings.

The total loss is a weighted sum of these components, with careful stage scheduling and hyperparameterization supporting convergence and performance (Li et al., 28 Feb 2026, Itkin et al., 16 Apr 2026).

6. Empirical Performance and Practical Impact

Token-based 3D Gaussian splatting attains competitive or state-of-the-art results (PSNR, SSIM, LPIPS, ATE) on benchmarks such as RealEstate10K and ScanNet. Notable empirical findings:

• Novel-view synthesis (RE10K, 8 views): TokenSplat achieves PSNR = 26.15 dB, SSIM = 0.858, LPIPS = 0.135, exceeding prior pose-free baselines (PSNR ≈ 25.0 dB, SSIM ≈ 0.83, LPIPS ≈ 0.16).
• Pose estimation: TokenSplat attains RPE–r = 0.458° (8 views), ATE = 0.012, representing a significant improvement over previous pose-free pipelines.
• Compactness: GlobalSplat yields PSNR = 28.57 (SSIM = 0.885, LPIPS = 0.138) with only 16 K Gaussians (4 MB asset), comparable to competing systems using >20× more primitives.
• Inference speed: GlobalSplat reports inference in 77.9 ms per scene on A100, with peak memory 1.79 GB.
• Stability in long sequences: Token-level fusion in TokenSplat prevents performance degradation due to Gaussian redundancy when aggregating many views; this issue afflicts pixel-aligned approaches (Li et al., 28 Feb 2026, Itkin et al., 16 Apr 2026).

Empirical ablation studies confirm the necessity of global attention stages, token-level fusion, dual-branch architectures, and cross-view alignment for optimal compactness and quality. Omission of these components yields degraded performance and/or increased representation redundancy.

7. Significance and Theoretical Implications

Token-based 3D Gaussian splatting establishes a new principle of "align first, decode later" in 3D scene reconstruction from multi-view images. By fusing cross-view tokens prior to explicit geometry, these methods minimize representation size while maximizing the expressivity and coherence of the 3D asset. This architecture natively prevents bloat otherwise induced by dense, view-aligned primitive allocation and demonstrates robust scalability as the number of input images grows.

A plausible implication is that the token-based methodology, with its abstraction of cross-view semantic correspondence, could generalize to other 3D generative or inverse problems where geometric alignment and compactness under minimal supervision are critical. The explicit disentanglement of pose and semantics via token specialization, and curriculum-based exposure to increasingly complex geometric structures, suggest fruitful directions for scene-level modeling, especially in pose-free and resource-constrained scenarios (Li et al., 28 Feb 2026, Itkin et al., 16 Apr 2026).