ZipSplat: Adaptive 3D Gaussian Splatting

• ZipSplat is a feed-forward 3D Gaussian splatting framework that reconstructs multi-view scenes using adaptive token-based Gaussian primitives.
• It decouples representation density from camera resolution, achieving up to 33× fewer Gaussians while improving reconstruction quality.
• Its four-stage pipeline—token extraction, k-means clustering, transformer refinement, and MLP decoding—enables flexible, pose-free scene reconstruction.

ZipSplat is a feed-forward 3D Gaussian Splatting framework that reconstructs scenes from multi-view images by directly predicting compact, content-adaptive sets of 3D Gaussian primitives, decoupling representation density from camera resolution. Unlike prior methods that assign one Gaussian per input pixel—resulting in inefficiencies tied to image resolution rather than geometric complexity—ZipSplat introduces a token-based architecture that determines Gaussian placement based on learned scene features. The system functions without requiring ground-truth camera poses or intrinsics and establishes new state-of-the-art results on multi-view benchmarks with significantly fewer Gaussians, outperforming previous pose-free baselines in both efficiency and reconstruction quality (Veicht et al., 3 Jun 2026).

1. Architectural Overview

ZipSplat employs a four-stage pipeline: token extraction, token compression by k-means clustering, transformer-based refinement, and MLP decoding to scene Gaussians.

1. Token Extraction: A pretrained multi-view backbone, such as DA3-Giant, processes $N$ images (posed or unposed) to produce $T$ dense visual tokens $\{x_i\}_{i=1}^T \in \mathbb{R}^d$ that aggregate local image features across viewpoints.
2. Token Compression: K-means clustering in the token feature space reduces the $T$ visual tokens to $K = r \cdot T$ scene tokens $\{z_j\}_{j=1}^K$, where $r \in (0,1]$ is an inference-time compression ratio controlling the trade-off between number of Gaussians and reconstruction fidelity.
3. Attention-Based Refinement: Each scene token $z_j$ recovers detail and global context via two transformer blocks: a cross-attention layer that allows scene tokens to attend to all visual tokens, and a self-attention layer for global contextualization among scene tokens.
4. MLP Decoding: A compact two-layer MLP decodes each refined scene token into $G$ anisotropic Gaussians, predicting fully unconstrained 3D positions, scales, rotations, opacities, and spherical harmonic color coefficients.

2. Tokenization and Scene Representation

Visual token extraction is performed as

$$x_i = f_\phi(I_1, \ldots, I_N)[i], \quad i = 1, \ldots, T,$$

where $T$0 is the pretrained multi-view backbone and $T$1.

The token compression step solves the k-means objective:

$T$2

yielding cluster centers $T$3 as initial scene tokens ($T$4). This mechanism enables a continuous efficiency-quality trade-off, with the clustering performed at inference without retraining. The number of scene tokens, and thus the total number of predicted Gaussians, scales sublinearly with the number of input views $T$5 since k-means removes redundancy among cross-view tokens.

3. Transformer-Based Token Refinement

Refinement leverages attention to infuse tokens with fine detail and holistic context. For cross-attention:

$T$6

with $T$7 (scene tokens), $T$8 (visual tokens), and learned projections $T$9.

Subsequently, self-attention operates on the updated scene tokens:

$\{x_i\}_{i=1}^T \in \mathbb{R}^d$0

After this process, each refined token $\{x_i\}_{i=1}^T \in \mathbb{R}^d$1 is augmented with both reintegrated local evidence and global scene awareness.

4. Gaussian Parameterization and Decoding

Each refined token generates $\{x_i\}_{i=1}^T \in \mathbb{R}^d$2 Gaussians via a two-layer MLP $\{x_i\}_{i=1}^T \in \mathbb{R}^d$3, where $\{x_i\}_{i=1}^T \in \mathbb{R}^d$4 covers all predicted parameters. Notably, the 3D center positions are predicted as

$\{x_i\}_{i=1}^T \in \mathbb{R}^d$5

ensuring unconstrained spatial placement in $\{x_i\}_{i=1}^T \in \mathbb{R}^d$6. Scale and opacity are controlled with softplus and sigmoid activations, rotation with normalized quaternions, and color with spherical harmonics.

5. Training Objectives and Pose-Free Operation

ZipSplat optimizes a composite loss:

$\{x_i\}_{i=1}^T \in \mathbb{R}^d$7

where:

• $\{x_i\}_{i=1}^T \in \mathbb{R}^d$8 is photometric,
• $\{x_i\}_{i=1}^T \in \mathbb{R}^d$9 for perceptual similarity,
• $T$0 when pseudo-depth is available,
• $T$1 is a geometric Chamfer loss to keep Gaussian centers near valid surfaces.

For unposed images, ZipSplat omits camera tokens and instead learns a pose embedding. At inference, camera extrinsics refinement via photometric+LPIPS reprojection loss offers substantial quality gains (up to $T$2 dB PSNR in under $T$3 seconds), with Gaussians held fixed.

6. Comparative Results and Efficiency Analysis

Extensive experiments demonstrate ZipSplat's ability to achieve high reconstruction fidelity at far reduced Gaussian counts compared to pixel-aligned and prior token-based approaches. The following summarizes reported performance:

Dataset	# Input Views	Baseline Method	# Gaussians	PSNR (dB)	SSIM	LPIPS
DL3DV	24	DA3 per-pixel	6.1M	21.70	—	—
		YoNoSplat	1.2M	22.01	—	—
		ZipSplat	249K	24.23	0.773	0.194
RealEstate10K	6	YoNoSplat	301K	24.99	—	—
		ZipSplat	62K	26.20	—	—

ZipSplat achieves up to $T$4 reduction in Gaussian count at equal or superior quality, consistently outperforming pose-free baselines by $T$5 dB and $T$6 dB PSNR on DL3DV and RealEstate10K, respectively. Quality gracefully degrades as $T$7 (the clustering ratio) is reduced at inference, giving a continuous quality-efficiency knob without retraining. The scene token—and thus Gaussian—budget grows sublinearly with number of input views due to k-means redundancy removal.

7. Applications and Broader Significance

ZipSplat's content-adaptive Gaussian prediction pipeline enables efficient 3D scene reconstruction in both pose-free and posed multi-view settings. Its ability to operate on unknown camera parameters, coupled with inference-time control over quality and efficiency, positions it as an effective solution for large-scale, cost-sensitive, or zero-shot scene representation tasks. Moreover, by decoupling the primitive budget from input image resolution, ZipSplat avoids inefficiencies of per-pixel approaches and adapts computation to true scene complexity (Veicht et al., 3 Jun 2026). A plausible implication is improved scalability for downstream tasks such as rendering, relighting, and AR/VR scene synthesis, particularly in environments where acquisition resources or annotation quality vary dramatically.