ARGS: Auto-Regressive Gaussian Splatting via Parallel Progressive Next-Scale Prediction

Abstract: Auto-regressive frameworks for next-scale prediction of 2D images have demonstrated strong potential for producing diverse and sophisticated content by progressively refining a coarse input. However, extending this paradigm to 3D object generation remains largely unexplored. In this paper, we introduce auto-regressive Gaussian splatting (ARGS), a framework for making next-scale predictions in parallel for generation according to levels of detail. We propose a Gaussian simplification strategy and reverse the simplification to guide next-scale generation. Benefiting from the use of hierarchical trees, the generation process requires only (\mathcal{O}(\log n)) steps, where (n) is the number of points. Furthermore, we propose a tree-based transformer to predict the tree structure auto-regressively, allowing leaf nodes to attend to their internal ancestors to enhance structural consistency. Extensive experiments demonstrate that our approach effectively generates multi-scale Gaussian representations with controllable levels of detail, visual fidelity, and a manageable time consumption budget.

• The paper introduces a reversible Gaussian simplification and tree-structured autoregressive generation method that reduces complexity to logarithmic steps.
• It leverages a transformer with tree-based attention and discretized tokenization to achieve high-fidelity and efficient 3D scene synthesis.
• Experimental results validate performance with metrics like SSIM and LPIPS, demonstrating ARGS's potential for scalable, interactive 3D asset generation.

ARGS: Parallel Hierarchical Autoregressive Generation for Gaussian Splatting

Introduction

"ARGS: Auto-Regressive Gaussian Splatting via Parallel Progressive Next-Scale Prediction" (2604.00494) introduces a unified framework for efficient generation, simplification, and multi-scale modeling of 3D scenes through hierarchical Gaussian splats. Unlike prior approaches dependent either on direct per-primitive optimization or sequential diffusion models, ARGS constructs a tree-structured hierarchy of Gaussian primitives enabling logarithmic ($\mathcal{O}(\log n)$) generative complexity. This is achieved via a reversible simplification pipeline and a tree-based autoregressive transformer, both of which facilitate controllable fidelity, efficient scene manipulation, and scalable rendering.

Progressive Simplification and Hierarchical Representation

The foundation of ARGS is a novel, reversible Gaussian simplification strategy designed to maintain fidelity at every reduction step. Leveraging moment-based Gaussian merging operators, each elimination merges the least visually significant Gaussian—computed using the determinant of the covariance matrix as a visibility proxy—with its nearest neighbor. This process yields a succession of coarser representations atop a binary merging tree, culminating in a single Gaussian representing the coarsest scene summary.

Figure 1: The simplification process visualized as a sequence, coarsening a full Gaussian Splatting object down to a singular primitive.

This ordered, recursive merging structure is further exploited to enable efficient multilevel encoding of scene geometry and appearance.

Hierarchical Autoregressive Generation

Conventional autoregressive generation proceeds point-by-point in a strictly sequential fashion, incurring linear complexity. ARGS inverts the tree construction resulting from simplification: synthesis starts from the root, and at each generative step, all splittable nodes are expanded in parallel to yield the next finer level. Owing to the binary hierarchical structure, a scene comprising $n$ splats can be generated in only $\log n$ autoregressive steps, a significant reduction compared to naïve sequential protocols.

Figure 2: Top: Vanilla AR protocols require $n-1$ sequential steps. Bottom: Hierarchical AR protocol expands multiple nodes per step, yielding logarithmic generation complexity.

The tree representation is formally maintained via parent-child mappings, and the generative graph supports path-wise and level-wise traversals. Only the leaf Gaussian parameters at each scale are stored at any time, providing a compact and lossless encoding.

Figure 3: Left: Merge pairs build the hierarchy during simplification. Middle: Tree traversal recovers all split nodes. Right: Only leaves from each level are retained for efficient access.

Transformer Architecture and Hierarchical Attention

For generative modeling, ARGS employs a transformer backbone operating on quantized representations of each Gaussian's pose, geometry, and appearance. The tree structure allows innovative attention masking schemes far more efficient than conventional causal schemes.

Figure 4: Left: Causal attention requires $n$ steps. Middle: Level-wise masking enables parallel processing at each hierarchy. Right: Tree-based masking adds context from earlier internal nodes for improved consistency.

With tree-based attention, leaf nodes at a given LoD attend not only to co-leaves but also to their ancestors, ensuring both local consistency and long-range structure propagation during autoregressive synthesis.

The forward process starts from the root Gaussian and, at each step, the transformer classifier determines which nodes to split (i.e., refine), then predicts parameters for the children, recursively descending the hierarchy to the desired LoD.

Figure 5: The transformer inference pipeline: at each step, nodes are classified as leaf/non-leaf and potentially split into child nodes, with all parameters predicted in a unified embedding space.

Experimental Validation

Simplification Quality

Quantitative assessment of the simplification pipeline used PSNR, SSIM, and LPIPS metrics on ModelSplat objects, comparing simplified renderings at various LoDs to ground truth.

Figure 6: Simplification error measured on PSNR, SSIM, and LPIPS at various percentile intervals, reflecting robust fidelity preservation across most of the dataset.

Aggressive simplification using the proposed method preserves high perceptual quality and maintains low error profiles even at significant reduction rates, demonstrating the efficacy of the merging strategy and tree hierarchy.

Hierarchical and Generative Performance

ARG's generative capabilities were measured both at the single-object and class level. Fitting experiments on categories such as airplanes showed near-parity between reconstruction from generated hierarchies and ground truth, with <0.03 difference on SSIM and marginal LPIPS increases at all but the finest scales.

Figure 7: SSIM performance at different hierarchy levels, indicating minority degradation due to hierarchical AR generation.

The hierarchical progressive generation process is visually illustrated as an object transitions from coarse to fine LoD, reflecting structural refinement and appearance detailing.

Figure 8: Coarse-to-fine generation evolution for a sample object at several intermediate steps.

In unconditional generation, ARGS produces diverse in-domain samples after training on a single semantic category, maintaining consistent geometry and plausible novel instances.

Figure 9: Representative unconditional samples generated for the airplane category, reflecting diversity and structural validity.

Multi-LoD representations support not just compression but also progressive decoding and interactive visualization in practical pipelines.

Figure 10: Scene rendering at increasing LoDs visualizing the fidelity/complexity trade-off inherent in hierarchical generation.

Transformer Training and Robustness

ARGS utilizes discretized, codebook-driven tokenization for continuous Gaussian parameters, three-dimensional rotary positional encodings for spatial equivariance, and noise-augmented training to mitigate autoregressive error propagation. This yields >99.6% split detection accuracy and >91% property accuracy in single-class training.

Implications, Limitations, and Future Directions

ARGS's hierarchical paradigm offers theoretical and practical advantages:

• Efficiency: Logarithmic generation/decoding complexity; supports large-scale or interactive applications with adaptive fidelity.
• Compression: Each sequence prefix is a valid scene; partial sequences yield lower-fidelity previews, supporting scalable transmission.
• Controllability: AR decoding is structurally editable at all hierarchy levels, supporting semantic or local manipulation.
• Unified pipeline: The reversible framework fuses fitting, compression, and novel generation with a shared architecture.

Limitations include:

• Absence of conditional generation (e.g., image- or text-driven synthesis).
• Inefficiency of single-Gaussian tokens; future work could use set-based or patch tokens.
• Current complexity of $O(n^2)$ for the simplification phase may hamper scalability to full scenes; a tree-based $O(n\log n)$ approach is suggested.
• Object-centric bias; extension to scene-centric modeling will require adaptation to handle variable camera distributions and heterogeneous scale.

The ARGS formalism points to a direction where 3D generative pipelines can enjoy the scalable complexity, interpretability, and editing capabilities that have driven the success of LLMs in natural language generation.

Conclusion

ARGS advances the landscape of 3D scene modeling through a principled, reversible hierarchical simplification and autoregressive generation pipeline. The tree-driven attention and transformer-based protocol enables efficient, multiscale, and editable Gaussian-based scene synthesis, with robust fidelity and scalable complexity. The introduced framework holds promise for efficient content creation, interactive editing, and compact 3D asset distribution as well as generalization to large-scale scene-level generation.