ZipSplat: Fewer Gaussians, Better Splats

Abstract: Feed-forward 3D Gaussian Splatting methods reconstruct a scene from posed or pose-free images in a single forward pass, yet current approaches predict one Gaussian per input pixel, tying the representation budget to camera resolution rather than scene complexity. A flat wall and a richly textured object thus produce equally many Gaussians despite very different geometric needs. We propose ZipSplat, a token-based feed-forward model that decouples Gaussian placement from the pixel grid. A multi-view backbone extracts dense visual tokens, and k-means clustering compresses them into a compact set of scene tokens. Cross- and self-attention refine these tokens, and a lightweight MLP decodes each into a group of Gaussians with unconstrained 3D positions. Because clustering is applied at inference, a single trained model spans the quality-efficiency curve without retraining. ZipSplat operates without ground-truth poses or intrinsics, yet sets a new state of the art on DL3DV and RealEstate10K with ${\sim}6{\times}$ fewer Gaussians than pixel-aligned methods, surpassing the best pose-free baseline by 2.1dB and 1.2dB PSNR, respectively. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++, outperforming all comparable baselines. Our project page is at ${\href{https://veichta.com/zipsplat}{https://veichta.com/zipsplat}}$.

• The paper introduces token-based 3D Gaussian Splatting that decouples Gaussian primitives from image pixels to dramatically reduce parameters while preserving high fidelity.
• It employs k-means clustering and attention mechanisms to aggregate multi-view features into adaptive scene tokens that allocate complexity based on geometric detail.
• Empirical results demonstrate superior pose-free reconstruction quality and scalable efficiency, achieving state-of-the-art performance with fewer Gaussians.

ZipSplat: Token-Based 3D Gaussian Splatting with Adaptive Capacity

Introduction

ZipSplat introduces a new architecture for feed-forward 3D Gaussian Splatting (3DGS), reframing the generation of explicit 3D scene representations from multi-view images. Contrary to dominant methods that tightly couple Gaussian primitives to the image pixel grid—thereby constraining scene capacity to camera resolution and introducing cross-view redundancy—ZipSplat predicts unconstrained Gaussians from a compact set of aggregated scene tokens. This design introduces both a dramatic reduction in the number of required Gaussians and a mechanism for adaptively allocating complexity according to geometric content.

Figure 1: Overview of ZipSplat: Multi-view features are clustered into adaptive scene tokens, processed by attention, and decoded into Gaussian primitives not tied to image rays.

Methodology

Scene Tokenization and Compression

Given $N$ input images, ZipSplat extracts dense multi-scale visual tokens via a multi-view foundation model backbone. To address redundancy from overlapping viewpoints, it applies k-means feature-space clustering to compress these tokens into $K$ scene tokens, where $K = r \cdot T$ and $r \in (0,1]$ is a tunable compression ratio applied only at inference time. This mechanism realizes a continuous quality–efficiency adjustment without extra training cycles.

Token aggregation proceeds with cross-attention and self-attention modules, ensuring global context and reintroducing fidelity lost during clustering. Scene tokens thus encode compact, spatially holistic summaries of scene structure, unanchored to individual image pixels.

Decoupled 3D Gaussian Generation

Each refined scene token is decoded by a lightweight MLP into a small group ($G$) of Gaussians, parameterized by center, scale, orientation, opacity, and low-order color harmonics. Crucially, the positions are predicted freely in $\mathbb{R}^3$ via inverse-log activation, unconstrained by image rays. This allows adaptive placement: tokens corresponding to geometrically intricate regions (e.g., object boundaries, high-frequency textures) are associated with denser, localized Gaussian groups; repetitive or flat surfaces receive fewer primitives.

This shift contrasts starkly with per-pixel formulations, where scene complexity is ignored in favor of uniform pixel/ray sampling. The result is superior parameter efficiency and improved 3D scene coverage, especially in multi-view or occluded regions.

Quality–Efficiency Control

A defining property is the ability to control the Gaussian budget by varying $r$ post-training. This allows a single ZipSplat model to traverse the quality–efficiency Pareto curve.

Figure 3: Left: Single-model quality–efficiency trade-off using compression ratio $r$; Right: Visual impact of compressing tokens, with graceful degradation as the budget is reduced.

Token Selection and Clustering Analysis

Varying the token selection mechanism reveals that at aggressive compression, feature-space k-means provides superior PSNR/LPIPS retention versus simple sub-sampling, as it merges cross-view redundancy while maintaining coverage of textured surfaces. As $r$ decreases, clusters naturally aggregate tokens from more input views, enhancing scene integration and cross-view consistency.

Figure 5: Left: PSNR/LPIPS under different token selection strategies; Right: Coverage of k-means clusters with respect to number of input views as $r$ decreases.

Training and Supervision

Free-form Gaussian placement introduces the risk of placing primitives outside camera frusta. To mitigate non-intersecting primitives receiving zero photometric gradients (and thus floating unanchored in training), ZipSplat supplements main losses (L1, LPIPS) with a one-directional Chamfer loss from Gaussian centers to pseudo-ground-truth 3D points (from back-projected depth maps). Gradients from this geometric regularization are detached for Gaussians already covered by rendered target views, ensuring the supervision does not degenerate into uniform grid coverage.

Further, the learning process is stabilized by coupled initialization of Gaussians, low initial opacity (to prevent occlusions), and normalized scene geometry.

Experimental Evaluation

Multi-View and Pose-Free View Synthesis

On DL3DV and RealEstate10K, ZipSplat establishes a new state-of-the-art among pose-free 3DGS approaches. For example, on DL3DV with 6 views, ZipSplat achieves 25.24 PSNR versus 24.10 for YoNoSplat, while utilizing $K$0 fewer Gaussians (62K vs. 301K). Even among methods leveraging ground-truth poses, ZipSplat is competitive and in cases outperforms them. Notably, with test-time token optimization, PSNR improvements of ~5dB are realized with minimal overhead.

Cross-Dataset and View Scalability

Evaluated zero-shot on Mip-NeRF360 and ScanNet++ (including scenarios with up to 128 views, much higher than training conditions), ZipSplat exhibits consistent quality improvements as the view count grows, in contrast to per-pixel baselines—which plateau or degrade due to accumulation of redundant Gaussians and memory constraints. With pose conditioning, ZipSplat further expands its lead, indicating the inherent generalization advantage from decoupling representations from the rigid pixel grid.

Resource Efficiency and Scalability

Inference profiling demonstrates that the dominant computational and memory cost is attributable to the backbone feature extractor, and that ZipSplat’s compression machinery and Gaussian decoding incur negligible additional overhead even at large view counts ($K$1). The output representation’s memory and storage footprint are sub-linear in $K$2 when employing adaptive compression, maintaining real-time rendering capability under resource constraints.

Figure 7: Inference time and peak GPU memory remain controlled across increasing view counts with adaptive token compression.

Figure 9: Overall rendering speed and Gaussian storage efficiency scale favorably under view-dependent compression, enabling deployment even on memory-limited devices or platforms.

Ablation and Analysis

Key ablations confirm:

• The superiority of the token-based formulation is robust to choice of backbone.
• Increasing Gaussians per token ($K$3) rapidly saturates returns, with $K$4 balancing quality and efficiency.
• Initialization and geometric (Chamfer) regularization are essential for convergent and stable training, particularly in the absence of pixel/pose priors.

Practical and Theoretical Implications

ZipSplat fundamentally advances feed-forward 3DGS by allocating representational resources based on scene content, not camera configuration, yielding compact and scalable models. The direct 3D prediction paradigm decouples error propagation from pose/depth estimation typical in ray-uplifting methods, enhancing cross-view and temporal (4D) consistency and making the system more robust for downstream scene understanding, AR, and robotics pipelines.

The model’s capacity for continuous quality–efficiency control at inference provides a powerful practical tool for balancing resource constraints against fidelity in real-time or embedded applications.

Future Directions

Key extensions include integration of semantic labeling tasks tightly into the ZipSplat framework, leveraging the compact token-to-Gaussian path for per-object or per-region scene understanding, and extending the method to jointly compress and reconstruct dynamic scenes (4D), exploiting multi-frame redundancy in both temporal and spatial domains.

Conclusion

ZipSplat proposes an efficient and flexible approach to multi-view 3D Gaussian Splatting, dispensing with pixel-grid anchoring in favor of adaptive, token-based scene abstraction. The result is a system that delivers high-fidelity, pose-free scene reconstructions with dramatically reduced parameter and storage budgets, and inference-time flexibility. This token-centric approach suggests a new direction for learned scene representations that adapt model complexity to the true underlying geometric and visual content.