---
title: 'ZipMap: Linear-Time 3D Reconstruction'
url: https://www.emergentmind.com/topics/zipmap
type: topic
---

# ZipMap: Linear-Time 3D Reconstruction

ZipMap is a stateful feed-forward model for 3D reconstruction from image collections that achieves linear-time, bidirectional scene processing via the use of test-time training (TTT) fast-weight layers. Designed to overcome the quadratic scaling bottleneck of transformer-based approaches, ZipMap efficiently zips hundreds of input frames into a compact hidden scene state in a single forward pass, enabling high-fidelity geometry reconstruction and real-time novel-view querying with competitive or superior accuracy compared to state-of-the-art quadratic-time models such as VGGT and $\pi^3$ [2603.04385].

## 1. Motivation and Background

Transformer-based models have driven advances in multi-view 3D vision but suffer from the computational cost of global self-attention, which scales quadratically with the number of frames $N$. Methods like VGGT and $\pi^3$ exhibit $O(N^2)$ complexity, rendering them impractical for large-scale image collections. Sequential and streaming approaches exist but often degrade reconstruction quality due to their reduced capacity for bidirectional information aggregation. ZipMap addresses these limitations by introducing a feed-forward stateful architecture capable of both linear runtime and bidirectional scene reasoning through rapid scene aggregation [2603.04385].

## 2. Model Architecture and Scene State Aggregation

ZipMap's architecture consists of $L=24$ identical Transformer-style blocks, each interleaving two operations:
- **Local Window Attention:** Within each input image (or ray-map), a self-attention mechanism with rotary positional embeddings aggregates spatially local patch tokens ($p \times d$ array per view). This attention operates per-frame, leading to complexity $O(p^2)$ per image, independent of $N$.
- **Global Large-Chunk TTT Layer:** Inspired by LaCT, ZipMap utilizes a SwiGLU-MLP $f_W(x) = W_2(\mathrm{SiLU}(W_1 x) \odot (W_3 x))$ as a "fast-weight" memory function. Parameters $W = \{W_1, W_2, W_3\}$ are updated using a global in-context associative memory built from all image tokens:
  - Tokens $x_i$ are projected to keys $k_i$, values $v_i$, and queries $q_i$.
  - A virtual TTT reconstruction loss, $\mathcal{L}_{\text{ttt}}(W) = -\sum_{i=1}^{N \cdot p} f_W(k_i)^T v_i$, is minimized via a single fast-weight update.
  - The gradient is normalized using a Newton–Schulz orthonormalization, and $W$ is renormalized (including $L_2$ regularization) before its application.
  - Updated weights $\hat{W}$ are applied to produce global context outputs $o'_i = f_{\hat{W}}(q_i)$, with outputs gated and RMS-normalized.

Because the TTT update is done once per layer (rather than per token pair), ZipMap achieves hidden-state aggregation cost of $O(N p d^2)$, compared to $O((N p)^2 d)$ in global self-attention layers.

## 3. Hidden Scene State Representation

ZipMap "zips" $N$ images into a compact scene representation stored in the fast weights of each layer. Letting $x_1,\dots,x_N$ be per-frame token embeddings after local attention, hidden-state aggregation is defined as:

$$
s = \text{Zip}(\{x_1,\dots,x_N\}) \triangleq \{W^{(1)}, \dots, W^{(L)}\}
$$

where $W^{(\ell)}$ is the MLP fast-weight state after the TTT update at layer $\ell$. Each block executes:

- $W^{(\ell)} \leftarrow \text{TTTUpdate}(W^{(\ell-1)}; \{k_{i,j}, v_{i,j}\})$, where $k_{i,j}$ and $v_{i,j}$ are projected from $x_i[j]$,
- The gradient $\nabla_W \mathcal{L}$ is aggregated over all tokens, orthonormalized, and applied,
- At the final block, the sequence $\{W^{(\ell)}\}$ forms a scene state of size $O(L d^2)$.

Novel-view query rays are processed efficiently by applying $\hat{W}^{(\ell)}$ to tokenized queries without further gradient updates.

## 4. Computational Complexity and Training Workflow

ZipMap's design delivers strict linear runtime in $N$:

- **Prior approaches:** $T_{\text{quad}}(N) = O((N p)^2 d)$,
- **ZipMap:** $T_{\text{zip}}(N) = O(N p^2 d + L N p d^2 + L p d^2)$, assuming constant $p$ and $L$ with respect to $N$.

**Training loss** combines several geometric, photometric, and camera alignment objectives:

$$
\mathcal{L} = \mathcal{L}_{\text{point}} + \mathcal{L}_{\text{depth}} + w_c \mathcal{L}_{\text{cam}} + \mathcal{L}_{\text{color}}^t + \mathcal{L}_{\text{depth}}^t + \mathcal{L}_{\text{point-normal}} + \mathcal{L}_{\text{depth-grad}}
$$

Components include mean point error, depth error, camera pose error, and RGB/depth supervision for novel views (MSE and LPIPS), along with normal and depth gradient matching penalties.

Single-pass pseudocode executes tokenization, local window attention, TTT gradient updates, and prediction heads (pose, depth, confidence, points), outputting the final fast-weights state and scene predictions.

## 5. Empirical Performance

ZipMap attains state-of-the-art accuracy while significantly reducing runtime on large image sets. Performance comparisons (NVIDIA H100, $N=750$ frames):

| Model         | Complexity | Runtime (s) |
|---------------|------------|-------------|
| VGGT          | $O(N^2)$   | 200.36      |
| $\pi^3$       | $O(N^2)$   | 151.16      |
| CUT3R         | $O(N)$     | 31.25       |
| TTT3R         | $O(N)$     | 31.20       |
| ZipMap        | $O(N)$     | 9.999       |

Additional benchmark results demonstrate:
- ScanNetV2 absolute trajectory error (ATE): ZipMap 0.034 m vs. $\pi^3$ 0.030 m vs. VGGT 0.035 m,
- KITTI video depth: AbsRel = 0.057 (ZipMap) vs. 0.038 ($\pi^3$) vs. 0.073 (VGGT),
- RealEstate10K camera AUC@5°: 53.34% (ZipMap) vs. 63.10% ($\pi^3$) vs. 38.71% (VGGT),
- DTU point-map: accuracy 1.228 mm, completeness 1.649 mm, normal consistency 0.675, matching or surpassing prior methods [2603.04385].

## 6. Real-Time Querying, Streaming, and Extensions

Because the aggregated scene state is stored in fast weights, novel-view querying after initial aggregation consists solely of apply steps, enabling constant-time ($\sim$100 FPS) querying independently of the size of the input collection. Streaming reconstruction is supported by processing each new frame $I_t$ via $W^{(t)} \leftarrow \text{TTTUpdate}(W^{(t-1)}; \{k_{t,j}, v_{t,j}\})$, assigning pose and local geometry without recurrent error accumulation and in $O(1)$ per-frame time. *This suggests* improved scalability and robustness for online scene reconstruction scenarios.

Empirically, streaming ZipMap outperforms previous linear-time sequential models (CUT3R, TTT3R) on camera, depth, and point-map benchmarks, even with shorter context lengths [2603.04385].

## 7. Significance and Future Implications

By leveraging TTT-based fast-weight state aggregation, ZipMap establishes a scalable paradigm for large-scale, high-fidelity 3D reconstruction and scene understanding. Its bidirectional, linear-time aggregation, efficient scene state representation, and applicability to both batch and streaming settings open new directions for real-time and large-scale 3D perception pipelines using feed-forward architectures [2603.04385]. A plausible implication is the potential for extending ZipMap-like architectures to even larger temporal or spatial scales, or to other modalities that benefit from rapid global context aggregation.

Source: https://www.emergentmind.com/topics/zipmap