Can3Tok: Scene-Level 3D Tokenization VAE

• Can3Tok is a VAE that tokenizes scene-level 3D Gaussian Splatting data using canonical normalization and semantic filtering to manage tens of thousands of Gaussians.
• It employs a transformer-based encoder–decoder with cross-attention for compressing irregular 3D data, ensuring fast encoding and precise reconstruction.
• Empirical validation on DL3DV-10K shows reduced failure rates and improved FID scores, demonstrating enhanced scalability and accuracy compared to prior methods.

Searching arXiv for Can3Tok and closely related references to ground the article. Can3Tok is the first variational autoencoder (VAE) that operates directly on scene-level 3D Gaussian Splatting (3DGS) representations, learning a compact, structured latent embedding that captures both semantic and spatial information for entire scenes. It introduces a canonical 3D tokenization scheme, a transformer-based encoder–decoder with cross-attention to compress tens of thousands of Gaussians, and a scene processing pipeline centered on scale normalization, allowing convergence and generalization on thousands of unbounded and scale-inconsistent scenes from DL3DV-10K. The learned latent supports feedforward generation, including text-to-3DGS and image-to-3DGS, by decoding canonical 3DGS and then de-canonicalizing to world coordinates (Gao et al., 2 Aug 2025).

1. Problem setting and motivation

Can3Tok is designed for scene-level 3D generation rather than the more common object-level setting. Object-level generative models assume bounded, canonical spaces, often a normalized unit cube with consistent scale, whereas scene-level 3DGS is unbounded, SfM-initialized, non-metric, and highly inconsistent in global scale across scenes. In addition, Gaussian primitives vary in scale, geometry, and appearance, and reconstructions contain floaters and hallucinations caused by limited views and distant content. These properties make unified latent representation learning difficult: without normalization, the encoder receives batches with radically different global scales and Gaussian radii; learned latents collapse or drift; self-attention over tens of thousands of Gaussians is computationally infeasible; and noise degrades high-frequency detail in reconstructions (Gao et al., 2 Aug 2025).

The method is motivated by empirical failures of prior 3D VAEs on this regime. PointNet VAE, PointTransformer, and L3DG fail to converge on even a few hundred scenes and exhibit zero generalization on DL3DV-10K, whereas Can3Tok converges across thousands and generalizes. This establishes the central problem Can3Tok addresses: scalable latent representation learning for unbounded scene-level 3DGS rather than bounded object-centric geometry (Gao et al., 2 Aug 2025).

At the representation level, a 3DGS scene is modeled as a set of anisotropic Gaussians with center $x \in \mathbb{R}^3$, orientation $r \in \mathbb{R}^4$ as a unit quaternion, anisotropic scale $s \in \mathbb{R}^3$, opacity $\alpha \in \mathbb{R}$, view-independent color $c \in \mathbb{R}^3$, and optionally view-dependent spherical harmonics coefficients $c_h \in \mathbb{R}^h$. The covariance is written as

$$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$

Under a pinhole camera, the projected mean and covariance follow

$$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$

and rendering proceeds by ordered alpha compositing,

$I(u) = \sum_i T_i(u)\, w_i(u)\, c_i,$

with per-Gaussian footprint and transmittance defined in the standard 3DGS manner (Gao et al., 2 Aug 2025).

2. Canonical 3D tokenization pipeline

The core preprocessing step is canonical normalization of scene geometry and scales. For Gaussian centers $\{x_i\}$, Can3Tok computes the scene center

$r \in \mathbb{R}^4$0

and a global scale

$r \in \mathbb{R}^4$1

It then applies the mapping

$r \in \mathbb{R}^4$2

Rotations $r \in \mathbb{R}^4$3, opacity $r \in \mathbb{R}^4$4, colors $r \in \mathbb{R}^4$5, and spherical harmonics coefficients are kept unchanged. In the more general form with a canonical rotation $r \in \mathbb{R}^4$6, the mapping becomes

$r \in \mathbb{R}^4$7

Because cameras undergo the same similarity transform, rendering from the normalized 3DGS with transformed cameras produces the same images as rendering from the original scene (Gao et al., 2 Aug 2025).

Canonicalization is paired with semantic-aware filtering. Can3Tok runs LangSAM on a central frame to segment “the most salient region,” picks one Gaussian inside the mask, and grows a set via K-NN in 3D until reaching a fixed budget $r \in \mathbb{R}^4$8, which is $r \in \mathbb{R}^4$9K in practice. This suppresses floaters and noisy, poorly supervised regions while preserving clean and salient scene partitions (Gao et al., 2 Aug 2025).

Token construction then attaches structured features to each Gaussian. For each primitive, the input concatenates Fourier positional encodings of the Gaussian center and the nearest canonical voxel center,

$s \in \mathbb{R}^3$0

together with raw parameters

$s \in \mathbb{R}^3$1

The input volume resolution is $s \in \mathbb{R}^3$2, and the positional encoding uses $s \in \mathbb{R}^3$3 Fourier channels per position. The per-Gaussian token therefore has dimension $s \in \mathbb{R}^3$4, and the full scene becomes a matrix

$s \in \mathbb{R}^3$5

This “dual positional encoding” injects structured anchors into the irregular 3D continuum and improves detail reconstruction (Gao et al., 2 Aug 2025).

A plausible implication is that Can3Tok’s notion of tokenization is not merely serialization of scene primitives; it is a canonicalization-and-anchoring procedure that makes irregular 3DGS data compatible with transformer latent modeling.

3. Encoder, decoder, and variational objective

The encoder receives $s \in \mathbb{R}^3$6, where $s \in \mathbb{R}^3$7, and first maps it to keys and values with a linear layer. Compression is achieved by cross-attention with a learnable canonical query

$s \in \mathbb{R}^3$8

where $s \in \mathbb{R}^3$9. The query is initialized as regular voxel grids plus descriptors, so the compressed tokens inherit geometric priors. Eight self-attention blocks, using multi-head attention with $\alpha \in \mathbb{R}$0 heads of dimension $\alpha \in \mathbb{R}$1 and FlashAttention, operate on the compressed representation. Two linear layers then produce the bottleneck mean $\alpha \in \mathbb{R}$2 and log-variance $\alpha \in \mathbb{R}$3, with the latent tensor shaped like Stable Diffusion’s latent, namely $\alpha \in \mathbb{R}$4 (Gao et al., 2 Aug 2025).

The VAE reparameterization is

$\alpha \in \mathbb{R}$5

The decoder expands $\alpha \in \mathbb{R}$6 with a linear layer, applies $\alpha \in \mathbb{R}$7 self-attention blocks, and uses an MLP tail to map back into continuous 3DGS parameter space, producing an output that corresponds index-wise to the input Gaussian set (Gao et al., 2 Aug 2025).

The training objective combines parameter-space reconstruction and standard VAE regularization. Let $\alpha \in \mathbb{R}$8 denote the full Gaussian parameters used as input channels. The reconstruction term is

$\alpha \in \mathbb{R}$9

and the complete objective is

$c \in \mathbb{R}^3$0

with

$c \in \mathbb{R}^3$1

and

$c \in \mathbb{R}^3$2

No additional regularizers are required if the data pipeline is used; training relies on canonicalization, semantic filtering, and random $c \in \mathbb{R}^3$3 rotations for augmentation (Gao et al., 2 Aug 2025).

The architectural constraint is computational as much as statistical. Self-attention on $c \in \mathbb{R}^3$4K tokens is $c \in \mathbb{R}^3$5 and infeasible, so Can3Tok compresses $c \in \mathbb{R}^3$6 by cross-attention with cost $c \in \mathbb{R}^3$7, then applies self-attention on the compressed sequence at cost $c \in \mathbb{R}^3$8. This compression mechanism is critical both for convergence and for memory feasibility (Gao et al., 2 Aug 2025).

4. Latent structure and downstream generation

Can3Tok’s latent is intended to preserve spatial structure rather than act as a purely semantic code. The paper reports that t-SNE embeddings under continuous $c \in \mathbb{R}^3$9 rotations form closed loops, indicating rotation-consistent latent organization. Latents for different croppings of the same scene are close, while different scenes are well separated (Gao et al., 2 Aug 2025).

This latent enables two downstream generation pathways. In text-to-3DGS, each scene’s middle frame is captioned with BLIP, and a UNet diffusion model is trained to denoise

$c_h \in \mathbb{R}^h$0

to $c_h \in \mathbb{R}^h$1 conditioned on text. The decoded output is a canonical 3DGS that is rendered after de-canonicalization. In image-to-3DGS, an image encoder is trained to regress directly to $c_h \in \mathbb{R}^h$2 using an $c_h \in \mathbb{R}^h$3 loss between the predicted latent and the latent produced by the Can3Tok encoder (Gao et al., 2 Aug 2025).

After generation, geometry is returned to world coordinates by the inverse normalization:

$c_h \in \mathbb{R}^h$4

Rotations and appearance remain unchanged. Rendering then uses standard 3DGS differentiable splatting with projected Gaussian means and covariances, followed by ordered alpha blending; if spherical harmonics are used, the color term can be replaced by a view-dependent evaluation $c_h \in \mathbb{R}^h$5 (Gao et al., 2 Aug 2025).

A notable quantitative result is that text-to-3DGS generation achieves FID $c_h \in \mathbb{R}^h$6 on rendered views, compared with $c_h \in \mathbb{R}^h$7 for PointTransformer. This result is presented as evidence that the latent is usable for feedforward generation rather than only reconstruction (Gao et al., 2 Aug 2025).

5. Empirical validation, ablations, and efficiency

Can3Tok is evaluated on DL3DV-10K with a $c_h \in \mathbb{R}^h$8 train/test split. The pipeline runs 3DGS per video and filters each scene to $c_h \in \mathbb{R}^h$9K tokens. Training uses $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$0 A100 GPUs and takes approximately $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$1 days. A single forward pass, including encode and decode, takes approximately $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$2 s per scene, and training iteration speed is approximately $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$3 s/iter versus approximately $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$4 s/iter for L3DG (Gao et al., 2 Aug 2025).

The main quantitative comparison on the DL3DV-10K test set with filtering is summarized below.

Method	L2	Failure rate
L3DG	1200.4	100%
PointNet VAE	1823.0	100%
PointTransformer	230.7	70%
Can3Tok	30.1	2.5%

These metrics use parameter-space $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$5 over Gaussian features, with failure defined by $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$6 on a scene. Qualitatively, the reported behavior is that baselines fail to reconstruct global layout and local details, while Can3Tok preserves both and generalizes to novel scenes (Gao et al., 2 Aug 2025).

Ablation results isolate the role of each component. Without the learnable query, replacing cross-attention with self-attention leads to divergence and out-of-memory behavior even with $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$7GB GPUs, with $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$8 and failure $$\Sigma_{3D} = R(r)\,\mathrm{diag}(s_1^2, s_2^2, s_3^2)\,R(r)^T.$$9. Without normalization, $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$0 and failure is $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$1. Without voxel coordinate appending, $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$2 and failure is $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$3. Without semantic filtering, $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$4 and failure is $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$5. Without augmentation, $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$6 and failure is $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$7. These ablations show that scale canonicalization, learnable-query compression, dual positional encoding, semantic filtering, and random $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$8 augmentation all contribute materially to stability and accuracy (Gao et al., 2 Aug 2025).

This evidence supports a narrow but important conclusion: the model’s performance is inseparable from its preprocessing pipeline. The VAE architecture alone is not presented as sufficient.

6. Scope, limitations, and nomenclature

Can3Tok is specifically tied to scene-level 3DGS. The reported limitations are that success is not $$\mu = \pi(x), \qquad \Sigma_{2D} = J \Sigma_{3D} J^T,$$9 because low-quality 3DGS reconstructions, including motion blur and skewed view distributions, degrade latent discriminability; the method is sensitive to canonicalization errors if cameras or centers are misestimated; and domain biases in DL3DV-10K may persist in the latent. Proposed future directions include improved text and image conditioning, multi-view consistency constraints in the latent, hierarchical tokenization for larger unbounded scenes, and better open-world scale recovery via depth priors (Gao et al., 2 Aug 2025).

A recurrent source of confusion is nomenclature. The label “Can3Tok” is used consistently in the scene-level 3DGS paper for canonical 3D tokenization and latent modeling of 3D Gaussian scenes (Gao et al., 2 Aug 2025), but related documents use similar wording for unrelated tokenization paradigms. In one case, ACTT is described as realizing a three-stage CAN payload understanding scheme—tokenization, translation, and packing/selection—under a “Can3Tok” interpretation for automotive Controller Area Network data (Verma et al., 2018). In another, ChannelTok is described as “also referred to as Can3Tok” in the context of channel-wise flexible-length vision tokenization, where each latent channel is treated as a visual token and stochastic tail-dropping induces a coarse-to-fine ordering (Paul et al., 3 Jun 2026).

For technical usage, the most precise meaning of Can3Tok is therefore the scene-level 3DGS VAE introduced for DL3DV-10K. Unlike CaTok’s one-dimensional causal image tokenizer for autoregressive vision modeling (Chen et al., 6 Mar 2026) or ChannelTok’s channel-wise image tokenizer (Paul et al., 3 Jun 2026), Can3Tok addresses unbounded scene-level 3D Gaussian representations through canonical normalization, semantic filtering, cross-attention compression, and VAE latent modeling.