GaussianGPT: Autoregressive 3D Scene Generation

• GaussianGPT is an autoregressive 3D scene generator that converts explicit Gaussian primitives into a serialized discrete voxel grid for next-token prediction.
• It leverages a sparse 3D convolutional autoencoder with lookup-free quantization to compress and preserve detailed spatial structure.
• The model employs a GPT-style transformer with 3D rotary positional embeddings and distinct token heads, enabling flexible completion, outpainting, and controllable sampling.

Searching arXiv for the primary and closely related papers. GaussianGPT denotes, in its most specific usage, a transformer-based model for fully autoregressive generation of 3D Gaussian scenes. Rather than treating 3D generation as diffusion or flow-matching over a global latent, it converts explicit Gaussian primitives into a discrete latent voxel grid, serializes that grid into tokens, and models the resulting sequence with next-token prediction. The resulting formulation makes completion, outpainting, controllable sampling via temperature, and flexible generation horizons native consequences of causal generation over an explicit representation compatible with modern Gaussian splatting pipelines (Lützow et al., 27 Mar 2026).

1. Terminology and scope

The name GaussianGPT is not entirely uniform across adjacent literatures. In the narrow sense, it refers to the 2026 autoregressive 3D scene generator described above. In broader usage, closely related work has used the term or invoked it as shorthand for Gaussian-centered transformer designs that are technically distinct, including Gaussian-process-based attention and explicit Gaussian scene or feature representations (Lützow et al., 27 Mar 2026, Bui et al., 27 Feb 2025).

Usage	Representative paper	Technical meaning
GaussianGPT	(Lützow et al., 27 Mar 2026)	Autoregressive 3D Gaussian scene generation via tokenized latent grids
“GaussianGPT”-style GP attention	(Bui et al., 27 Feb 2025)	GP-based transformer attention with uncertainty calibration
Gaussian-centered explicit representations	(Zhang et al., 2024, James et al., 6 Nov 2025, Li et al., 22 Dec 2025)	3D Gaussian reconstruction/generation, 3D point encoding, or 2D image representation/compression

This ambiguity matters because the 2026 model is neither a Gaussian-process transformer nor a LLM operating over text. Its “Gaussian” component refers to the target scene representation: explicit 3D Gaussian primitives with attributes such as position, opacity, scale, rotation, color, and optionally spherical harmonics. Its “GPT” component refers to decoder-only causal next-token modeling over a serialized discrete scene representation (Lützow et al., 27 Mar 2026).

2. Discrete scene representation and compression

GaussianGPT begins from a scene represented as 3D Gaussian primitives and overlays it with a world-coordinate voxel grid. Each Gaussian is assigned to the voxel containing its center; absolute positions are replaced by relative offsets from voxel centers; if multiple Gaussians fall in the same voxel, the method randomly subsamples; and per-voxel Gaussian features are encoded using lightweight heads and concatenated into a voxel feature vector. This converts an unordered Gaussian set into a sparse voxel tensor suitable for spatial compression (Lützow et al., 27 Mar 2026).

Compression is handled by a sparse 3D convolutional autoencoder $(\mathcal{E}, \mathcal{D})$, following L3DG. The encoder downsamples the sparse grid into a latent 3D grid, and the decoder reconstructs voxel-level features from the latent codes. The paper’s key representational move is then to quantize the latent grid with lookup-free quantization (LFQ): the encoder output $\mathbf{z}$ is discretized by sign, yielding binary-valued discrete latent codes without a learned nearest-neighbor codebook lookup (Lützow et al., 27 Mar 2026).

The autoencoder objective combines rendering, perceptual, occupancy, and LFQ terms:

$$\mathcal{L} = \lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc} + \lambda_{occ}\mathcal{L}_{occ} + \lambda_{LFQ}\,\text{softplus}\!\left(\mathcal{L}_{LFQ} + 5\right).$$

Here, $\mathcal{L}_{RGB}$ is an $L_1$ color loss from re-rendered images, $\mathcal{L}_{perc}$ is a VGG19 perceptual loss, $\mathcal{L}_{occ}$ is an occupancy BCE loss in the decoder upsampling layers, and $\mathcal{L}_{LFQ}$ is a codebook entropy or usage regularizer encouraging high entropy and better code utilization. The re-rendering supervision is aggregated over sampled images and poses (Lützow et al., 27 Mar 2026).

This design means that GaussianGPT does not autoregress directly over raw continuous Gaussians. Instead, it autoregresses over a compressed discrete latent proxy that preserves spatial structure while making transformer modeling tractable.

3. Tokenization, serialization, and transformer architecture

After compression, the latent grid is serialized into a one-dimensional token stream. GaussianGPT uses a fixed column-wise $xyz$ traversal in which $z$ is the least significant dimension; the model iterates through a vertical column at each $\mathbf{z}$0 location and then moves to the next spatial position. The paper notes that this ordering does not preserve 3D locality perfectly in 1D, but ablation indicates that it works well and is simpler than space-filling alternatives such as Z-order and Hilbert traversal (Lützow et al., 27 Mar 2026).

The serialized sequence alternates between position tokens $\mathbf{z}$1 and feature tokens $\mathbf{z}$2, producing a stream of the form

$\mathbf{z}$3

The model uses separate vocabularies for the two token classes. This separates geometry or occupancy from appearance or latent content, avoids competition between spatial and semantic prediction, and allows chunk size to scale independently of codebook size. Because full-scene sequences can be too long, modeling is performed on spatial chunks, with each voxel position represented relative to the current chunk rather than globally (Lützow et al., 27 Mar 2026).

The causal model factorizes the token distribution as

$\mathbf{z}$4

and is trained with standard teacher-forced cross-entropy,

$\mathbf{z}$5

Because the sequence alternates token types, the transformer uses a position head only when the next token should be a position token and a feature head only when the next token should be a feature token; invalid vocabulary entries are masked (Lützow et al., 27 Mar 2026).

The backbone is a decoder-only GPT-style transformer based on GPT-2 and the nanochat implementation. Its critical positional mechanism is 3D rotary positional embedding: rather than encoding only sequence index, attention is conditioned on actual 3D coordinates so that relative spatial offsets influence attention even when nearby voxels are distant in the serialized stream. The model further extends rotary encoding with a fourth dimension for token type, helping attention distinguish “where is the voxel?” from “what is the voxel content?” Additional architectural choices include query-key normalization, per-layer residual scaling, Muon optimizer for some modules, no value embeddings, and no sliding-window attention in the core model (Lützow et al., 27 Mar 2026).

4. Autoregressive generation modes

At inference time, unconditional generation starts from a BOS token and proceeds causally by predicting a position token, then a feature token, and repeating until EOS. The reported sampling configuration uses temperature $\mathbf{z}$6 and nucleus sampling with $\mathbf{z}$7. This makes diversity directly controllable through standard autoregressive decoding parameters rather than through diffusion schedules or guidance heuristics (Lützow et al., 27 Mar 2026).

Completion is implemented by serializing an observed partial scene as a prefix prompt and continuing the sequence autoregressively. No separate completion-specific architecture is introduced. The same prefix-conditioning logic also enables outpainting: the model can generate one chunk, slide a window, append new columns, and repeat indefinitely. The appendix further describes a bootstrapped sliding-window procedure for growing scenes efficiently beyond the original training horizon (Lützow et al., 27 Mar 2026).

The paper also exploits a property specific to sequential decoding: explicit resampling and search. It mentions tree search or retry strategies for resampling empty columns so as to encourage more connected scenes. This is a notable difference from holistic denoisers. In GaussianGPT, local failures or undesirable partial decisions can be handled by branch-and-retry procedures over the causal sequence rather than by restarting an entire global refinement trajectory (Lützow et al., 27 Mar 2026).

Conceptually, the model therefore casts 3D scene synthesis as discrete incremental construction. Geometry, appearance, and layout emerge through a sequence of conditional decisions over chunk-relative spatial tokens, rather than through iterative denoising of a full latent scene.

5. Empirical behavior, ablations, and practical constraints

The reported evaluation covers ASE / SceneSplat++ synthetic indoor scenes, 3D-FRONT converted into Gaussian scenes, PhotoShape chairs for object-level generation, and ScanNet++ for real-world finetuning in the appendix. On PhotoShape chairs, GaussianGPT outperforms prior baselines on FID, KID, and Coverage, while remaining competitive on MMD. For scene generation, the qualitative results emphasize coherent indoor layouts, plausible object placement, and stylistically consistent chunks (Lützow et al., 27 Mar 2026).

A central qualitative result is large-scale autoregressive outpainting: the model can produce a $\mathbf{z}$8 scene by repeated continuation. Completion examples reportedly yield consistent floors and walls, plausible continuation of structure, and diversity across samples. These findings support the paper’s claim that causal scene construction is particularly suited to prefix-conditioned editing and scalable generation horizons (Lützow et al., 27 Mar 2026).

The main serialization ablation compares $\mathbf{z}$9, Z-order, Hilbert, and transposed variants. The simple $$\mathcal{L} = \lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc} + \lambda_{occ}\mathcal{L}_{occ} + \lambda_{LFQ}\,\text{softplus}\!\left(\mathcal{L}_{LFQ} + 5\right).$$0 traversal performs best or on par with the alternatives. The paper’s interpretation is that once 3D RoPE injects spatial locality directly into attention, the exact one-dimensional traversal becomes less critical. This directly counters the assumption that high-quality autoregressive 3D generation necessarily requires a sophisticated space-filling serialization (Lützow et al., 27 Mar 2026).

Several practical constraints are also explicit. First, full-scene token streams can be too long for direct transformer modeling, hence chunk-based generation. Second, the system operates through a quantized latent grid rather than direct continuous Gaussian decoding. Third, the fixed serialization order is a convenience rather than a geometrically perfect ordering. These are not framed as defects so much as structural compromises that make autoregressive scene modeling computationally workable.

6. Position within adjacent research

GaussianGPT sits at the intersection of two broader developments: autoregressive modeling beyond conventional text orderings, and the increasing use of explicit Gaussian representations across vision and graphics. In the autoregressive literature, “$$\mathcal{L} = \lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc} + \lambda_{occ}\mathcal{L}_{occ} + \lambda_{LFQ}\,\text{softplus}\!\left(\mathcal{L}_{LFQ} + 5\right).$$1-GPTs: A New Approach to Autoregressive Models” introduces shuffled-order causal transformers with double positional encoding, allowing arbitrary subset conditioning, infilling, and burst-style multi-token generation under a rejection strategy with a sub-linear number of model evaluations (Pannatier et al., 2024). GaussianGPT does not adopt arbitrary per-sample orderings; it keeps a fixed $$\mathcal{L} = \lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc} + \lambda_{occ}\mathcal{L}_{occ} + \lambda_{LFQ}\,\text{softplus}\!\left(\mathcal{L}_{LFQ} + 5\right).$$2 traversal and instead relies on 3D RoPE to recover spatial awareness. This suggests a contrast between two different ways of loosening the standard sequential bias: order modulation in $$\mathcal{L} = \lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc} + \lambda_{occ}\mathcal{L}_{occ} + \lambda_{LFQ}\,\text{softplus}\!\left(\mathcal{L}_{LFQ} + 5\right).$$3-GPT and spatially informed fixed-order serialization in GaussianGPT.

A separate line of work uses “GaussianGPT”-style language to describe Gaussian-process reinterpretations of attention. “Revisiting Kernel Attention with Correlated Gaussian Process Representation” argues that earlier GP-based transformer models constrain attention to be symmetric because ordinary GP kernels must be symmetric positive semidefinite, forcing $$\mathcal{L} = \lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc} + \lambda_{occ}\mathcal{L}_{occ} + \lambda_{LFQ}\,\text{softplus}\!\left(\mathcal{L}_{LFQ} + 5\right).$$4; CGPT relaxes this by modeling attention as cross-covariance between correlated Gaussian processes, thereby restoring asymmetric attention while preserving predictive variance and uncertainty calibration (Bui et al., 27 Feb 2025). This is a different technical program from GaussianGPT’s explicit 3D scene autoregression. The shared label reflects Gaussian mathematics in both cases, but one concerns probabilistic attention operators and the other concerns discrete generation over Gaussian scene primitives.

In 3D and image modeling more broadly, several Gaussian-based systems provide useful contrasts. GeoLRM is a geometry-aware large reconstruction model that predicts up to 512k Gaussians from 21 posed images using a two-stage proposal-and-reconstruction pipeline with deformable cross-attention; it is feed-forward, image-conditioned, and reconstruction-oriented rather than autoregressive (Zhang et al., 2024). “3D Gaussian Point Encoders” replace implicit PointNet-style point encoders with an explicit Gaussian basis optimized via natural gradients and distillation, making Gaussians the basis of recognition features rather than the target of generation (James et al., 6 Nov 2025). “GaussianImage++” applies 2D Gaussian Splatting to image representation and compression via distortion-driven densification, content-aware filtering, and learnable quantization, again using explicit Gaussian primitives but in a 2D rasterization setting rather than causal 3D synthesis (Li et al., 22 Dec 2025).

Taken together, these works indicate that GaussianGPT names a specific autoregressive 3D scene generator while also sitting inside a wider Gaussian-centered modeling trend. In that trend, Gaussians function variously as attention kernels, geometric basis functions, reconstruction primitives, image coding elements, and—uniquely in the 2026 model—as the explicit scene representation underlying token-by-token 3D generation (Lützow et al., 27 Mar 2026).