---
title: 'tttLRM: Test-Time Training 3D Reconstruction'
url: https://www.emergentmind.com/topics/tttlrm
type: topic
---

# tttLRM: Test-Time Training 3D Reconstruction

Searching arXiv for the specified paper and closely related work names mentioned in the provided data.
I’ll look up the paper on arXiv to verify metadata before writing the article.
Searching arXiv for `2602.20160` and `tttLRM`.
tttLRM, short for **Test-Time Training Large Reconstruction Model**, is a large 3D reconstruction model that uses **test-time training (TTT)** as its core sequence-modeling and memory mechanism to enable **long-context, autoregressive 3D reconstruction** with **explicit** outputs, primarily **3D Gaussian Splatting**, while keeping computation **linear in the number of views** [2602.20160]. It is designed for settings in which many posed RGB images of a scene or object must be compressed into a fixed-size latent state and then decoded into explicit 3D structure suitable for high-resolution novel view synthesis, feedforward reconstruction, or streaming reconstruction and refinement from incoming observations. In this formulation, the model’s fast weights function as an **implicit 3D representation in latent space**, and virtual queries decode that latent into explicit formats such as Gaussian splats or a triplane NeRF [2602.20160].

## 1. Definition, scope, and stated contributions

tttLRM addresses a specific bottleneck in large reconstruction models for 3D: existing systems such as LRM, GS-LRM, and Long-LRM are described as being bottlenecked by **quadratic attention** and/or fixed small numbers of input views. The proposed replacement is a **TTT/LaCT fast-weight layer** that stores all multi-view information in a **fixed-size, learned fast-weight memory**, updates that memory **at test time** via gradient descent, and can be queried by virtual tokens to decode different explicit 3D formats [2602.20160].

The model is framed around three operating requirements that are all explicit in the source description: support for **high-resolution** novel view synthesis up to \(1024^2\), ingestion of **long sequences** of views up to **64–128 views** and **more than 1M tokens**, and operation in both **feedforward** mode and a **streaming / autoregressive** mode where images arrive over time [2602.20160]. The explicit outputs are primarily **3D Gaussian splats**, though the same latent can also be decoded into a **triplane grid** for NeRF-like rendering.

The stated contributions are organized around seven points. First, tttLRM is presented as a **TTT-based long-context LRM** in which fast weights serve as the scene memory, enabling long-context 3D reconstruction with **linear complexity** in the number of tokens or views. Second, it applies **LaCT**—**Large Chunk Test-Time Training**—to 3D as the core sequence module, using large token chunks for good GPU utilization and fixed-size memory compression. Third, it introduces an **implicit-to-explicit decoding** view in which fast weights represent an implicit latent 3D state that can be decoded into **3D Gaussian splats** or **triplane features**. Fourth, it provides an **autoregressive / streaming** causal variant that updates fast weights as views arrive and progressively refines Gaussian splats. Fifth, it includes a **distributed / sequence-parallel training** and inference scheme that shards the sequence dimension across multiple GPUs. Sixth, it uses **pretraining on novel view synthesis** through **TTT-LVSM**, then fine-tunes for 3D reconstruction. Seventh, it reports **state-of-the-art feedforward 3DGS reconstruction** on DL3DV-10K scenes and Objaverse/GSO objects, outperforming Long-LRM and GS-LRM on PSNR, SSIM, and LPIPS while remaining much faster than optimization-based baselines [2602.20160].

A common misconception is that “test-time training” implies updating the entire network during inference. In tttLRM, that is not the case: **slow weights** are learned during training and are **frozen at test time**, while only the internal **fast weights** are updated per scene [2602.20160].

## 2. Architecture, tokenization, and explicit 3D decoding

The inputs are a set of posed RGB images and corresponding ray embeddings,
$$
\{I_i \in \mathbb{R}^{H\times W \times 3}\}_{i=1}^N,\qquad
\{R_i \in \mathbb{R}^{H\times W \times 9}\}_{i=1}^N,
$$
where the ray embeddings encode position and direction per pixel. The main output is a set of **3D Gaussian primitives** representing the scene, with an optional alternative output of a **triplane grid** for NeRF-style rendering [2602.20160].

Each image is patchified and tokenized according to
$$
\{T_{i,j}\}_{i=1}^{N}{}_{j=1}^{HW/p^2} =
\text{Tokenize}\big(\text{Patchify}([\{I_i\}_{i=1}^{N}, \{R_i\}_{i=1}^{N}])\big).
$$
The model concatenates \(I_i\) and \(R_i\) along channels, patchifies using size \(p \times p\), and linearly projects patches to tokens \(T_{i,j} \in \mathbb{R}^d\). The typical setting is \(p=8\) or \(16\), with hidden dimension \(d=768\) [2602.20160].

The backbone is a stack of **24 LaCT blocks**. Each block combines a **window attention** module, used locally within each image to capture within-view spatial relationships, with a **TTT / LaCT fast-weight layer**. In simplified form, omitting feed-forward MLPs, each layer performs
$$
T_i \leftarrow T_i + \text{WinAttn}(T_i),
$$
followed by
$$
W = \text{Update}(\{T_i\}_{i=1}^{N}),\qquad
T_i \leftarrow \text{Apply}(W, T_i).
$$
Here, **Update** uses the LaCT rule, treating token projections as key–value pairs, computing gradients of an MSE loss, and updating \(W\) with a **Muon** optimizer; **Apply** uses the current fast weights to map tokens in a manner analogous to attention with a compressed KV cache. Both steps are implemented with **linear scaling in the number of tokens** [2602.20160].

A distinctive architectural feature is the use of **virtual tokens**. After processing input tokens and updating \(W\), tttLRM does not simply attach 3D heads to the input token stream. Instead, it introduces query tokens that are **only applied**, not used to update \(W\):
$$
T_i^{\text{v}} \leftarrow \text{Apply}(W, T_i^{\text{v}}).
$$
For **3DGS**, these virtual tokens are **virtual views** with known cameras, often aligned with or subsampled from actual views. For **triplane NeRF**, they are learnable triplane feature tokens not necessarily tied to images. The queried virtual tokens are then passed to a token decoder that produces explicit 3D representation parameters [2602.20160].

For Gaussian decoding, each virtual-view patch token predicts Gaussian attributes including RGB color \(\mathbf{c}_k\), scale \(s_k\) or anisotropic covariance parameters, rotation parameters, opacity \(\alpha_k\), and a **per-pixel depth** \(d_k\). Given ray origin \(\mathbf{o}_k\) and direction \(\mathbf{r}_k\), the Gaussian center is
$$
\boldsymbol{\mu}_k = \mathbf{o}_k + d_k \mathbf{r}_k.
$$
Each Gaussian can therefore be written as
$$
G_k = (\boldsymbol{\mu}_k, \Sigma_k, \mathbf{c}_k, \alpha_k, \dots),
$$
where \(\Sigma_k\) is obtained from scale and rotation. The union of Gaussians from all virtual views forms the scene representation, which is rendered by a standard 3DGS rasterizer [2602.20160].

This architecture makes a sharp distinction between latent memory and explicit geometry. The fast weights are the latent scene memory; the Gaussian or triplane outputs are decoded products of that memory rather than the memory itself. This suggests that the framework is intended as a generic latent-3D-to-explicit-3D decoder family rather than a single-format predictor.

## 3. Test-time training as fast-weight scene memory

In tttLRM, **test-time training** refers to optimization of the **fast weights \(W\)** inside each LaCT layer while keeping the rest of the network fixed. The fast weights are initialized to a **learned default** from training and then updated at test time using gradients from a TTT objective defined over token-derived key–value pairs [2602.20160]. Conceptually, for a chunk of tokens \(\{x_i\}\) projected to keys \(k_i\) and values \(v_i\), LaCT defines a parametric function \(f_W\) and minimizes
$$
\mathcal{L}_{\text{TTT}}(W) = \sum_i \|f_W(k_i) - v_i\|_2^2,
$$
with the update
$$
W \leftarrow W - \eta \nabla_W \mathcal{L}_{\text{TTT}}(W).
$$
The updates are performed in large chunks, stated as **up to \(\sim 1\)M tokens**, for efficiency [2602.20160].

The scene-level training objective does **not** require explicit 3D supervision. The primary reconstruction loss is a rendering loss,
$$
\mathcal{L}_{\text{RGB}} = \text{MSE}(I_{\text{pred}}, I_{\text{gt}}) + \lambda\, \text{Perceptual}(I_{\text{pred}}, I_{\text{gt}}),
$$
where the perceptual term uses **VGG-19** features. For scenes, the training loss also includes a **scale-invariant depth loss** between Gaussian positions along the depth axis and pseudo-ground-truth depth from a monocular depth estimator, as well as **opacity regularization** to encourage sparsity in Gaussians:
$$
\mathcal{L}_{\text{depth}} = \text{SI-Depth}(\hat{d}, d_{\text{pseudo}}),\qquad
\mathcal{L}_{\text{opacity}} = \sum_k \phi(\alpha_k).
$$
The total loss is
$$
\mathcal{L} = \mathcal{L}_{\text{RGB}} + \lambda_{\text{depth}} \mathcal{L}_{\text{depth}} + \lambda_{\text{opacity}} \mathcal{L}_{\text{opacity}}.
$$
At training time these losses update the slow weights; at test time Gaussian generation is feedforward given \(W\), unless extra post-optimization is deliberately added [2602.20160].

The linear-complexity claim follows from the LaCT formulation. For sequence length \(L\) and token dimension \(d\), each LaCT block has complexity
$$
O(L d^2),
$$
which is linear in \(L\). This contrasts with conventional attention at
$$
O(L^2 d),
$$
where each token attends to all others. In the fast-weight formulation, the entire key–value set is compressed into a fixed-size parameter matrix \(W\), each token contributes a constant amount of gradient information, and no pairwise token–token interactions are formed [2602.20160].

This yields an interpretation of sequence modeling as **online regression of values from keys, learned at test time**. In the 3D setting, the consequence is that a fixed-size learned memory can summarize many posed observations without quadratic growth in cost.

## 4. Long-context and autoregressive reconstruction

In this framework, **autoregressive** means **causal in the sequence of input views**. The fast-weight state \(W_t\) at time step \(t\) depends only on views \(1,\dots,t\), and as new views arrive the model updates \(W\) and immediately produces updated Gaussians for query or virtual views [2602.20160]. The recurrence is
$$
W_t = \text{Update}_{\text{LaCT}}(W_{t-1}, \text{tokens}(\mathcal{I}_{(t)})),
$$
and the decoded 3D state is a function of \(W_t\).

The paper’s simplified algorithm initializes \(W \leftarrow W_0\). For each mini-batch of input views \(\mathcal{I}_{(b)}\), it first updates the fast weights,
$$
\_, W \leftarrow \mathcal{F}(W, \mathcal{I}_{(b)}),
$$
and then predicts Gaussians for query views \(\mathcal{I}^{v}_{(b)}\),
$$
G_{(b)}, \_ \leftarrow \mathcal{F}(W, \mathcal{I}^{v}_{(b)}).
$$
The Gaussian set at the final batch \(G_{(B)}\) is the current scene reconstruction [2602.20160].

The model supports both **feedforward** and **streaming** operation. In feedforward mode, all \(N\) views are processed together, either as one sequence or in large chunks, then the latent memory is queried with virtual tokens to decode Gaussians. In streaming mode, the sequence is processed in batches, for example **4 views at a time**, with fast weights carrying cumulative information from earlier observations [2602.20160].

A crucial design point is that the output Gaussian set is **regenerated from scratch** at each autoregressive stage rather than simply appended to. This allows the model to **correct earlier errors**. The source description contrasts this with a cheaper **Predict&Merge** strategy, where only new Gaussians are predicted and then merged with previous ones. That strategy cannot correct old Gaussian errors and yields worse metrics:
$$
\text{Predict\&Merge: PSNR } 21.50 \quad \text{vs Ours: } 23.63.
$$
The stated qualitative progression is: with only **4 views**, reconstruction is coarse but consistent; at **8 views**, coverage and fidelity improve; at **32 views**, quality approaches full long-context feedforward reconstruction [2602.20160].

The paper also notes a capacity limit. Because \(W\) is fixed size, very long sequences or highly complex scenes eventually degrade performance, especially in outdoor or high-frequency scenes. To mitigate drift, the appendix explores **selective fast-weight updates** using an EMA of squared gradients as an approximate Fisher information estimate and an elastic regularization toward an EMA anchor \(W_{\text{anchor}}\). This produces a slight autoregressive improvement, reported as **PSNR 24.81 \(\rightarrow\) 24.95** [2602.20160].

A common misunderstanding is to equate “autoregressive” with token-by-token generation of images or geometry. Here it specifically denotes causal state updates over incoming **views**, with \(W\) acting as the recurrent state.

## 5. Pretraining, transfer, and empirical performance

The backbone architecture is stated to be identical to **TTT-LVSM**, a LaCT-based large view synthesis model. Pretraining uses a **novel view synthesis** task in which multi-view images are used to predict unseen views via the same fast-weight latent representation. For tttLRM, the backbone parameters are initialized from TTT-LVSM, while the Gaussian or triplane token decoder and, if needed, virtual token definitions are added or modified for reconstruction [2602.20160].

The reported effect of this initialization is both **faster convergence** and **better final quality**. For the **Gaussian 3DGS** decoder, training without pretraining gives **PSNR = 32.77, SSIM = 0.969, LPIPS = 0.026**, while with pretraining it gives **PSNR = 33.14, SSIM = 0.972, LPIPS = 0.024**. For the **triplane NeRF** decoder, the corresponding improvement is from **PSNR = 26.40, SSIM = 0.903, LPIPS = 0.093** to **PSNR = 27.87, SSIM = 0.925, LPIPS = 0.075** [2602.20160]. The interpretation given in the source is that the TTT backbone learns a strong **implicit view-consistent 3D prior** during NVS pretraining.

The reported benchmarks cover both objects and scenes. On **GSO**, trained on **Objaverse**, at **512×512** with **8 input views**, **GS-LRM** gives **PSNR 32.83, SSIM 0.969, LPIPS 0.029, Time 0.7s**, while **tttLRM** gives **PSNR 34.02, SSIM 0.974, LPIPS 0.025, Time 0.3s** [2602.20160]. At **512×512** with **16 views**, **GS-LRM** gives **PSNR 33.55, SSIM 0.976, LPIPS 0.023, Time 2.5s**, while **tttLRM (10 virtual)** gives **PSNR 34.67, SSIM 0.978, LPIPS 0.022, Time 0.8s**. At **24 views**, the description states that tttLRM similarly outperforms GS-LRM in both quality and speed. It is also reported to scale to **1024×1024** resolution, where GS-LRM runs out of memory, and to produce high-fidelity single-view reconstructions when combined with a multi-view diffusion generator [2602.20160].

On **DL3DV-140** with **32 views**, the reported numbers are: **3DGS (30k iters)** at **PSNR 23.60, SSIM 0.779, LPIPS 0.213, 13 min**; **Scaffold-GS (30k)** at **24.77, 0.805, 0.205, 16 min**; **Long-LRM (feedforward)** at **24.10, 0.783, 0.254, 1.0 s**; **Long-LRM + 10-step optim** at **25.60, 0.826, 0.233, 37 s**; **tttLRM (feedforward)** at **25.07, 0.822, 0.215, 7.2 s**; and **tttLRM + 10-step optim** at **26.37, 0.854, 0.201, 42 s** [2602.20160]. On **DL3DV-140** with **64 views**, **Long-LRM (64v)** gives **24.63, 0.799, 0.243**; **3DGS (30k)** gives **26.55, 0.852, 0.164**; **tttLRM (feedforward)** gives **25.95, 0.844, 0.195**; and **tttLRM + 10-step optim** gives **27.65, 0.880, 0.177**. The source states that **Tanks&Temples** shows similar trends, with tttLRM better than Long-LRM and competitive with, and sometimes better than, optimization-based methods while being **100× faster** [2602.20160].

These results situate tttLRM between feedforward explicit predictors and per-scene optimization pipelines. It preserves explicit 3DGS outputs and real-time rendering while narrowing the quality gap to optimization-heavy systems.

## 6. Scalability, relation to prior work, and limitations

The scalability argument is tied to both algorithmic complexity and distributed implementation. Letting \(N\) denote the number of images, \(H \times W\) the resolution, \(p\) the patch size, and \(L \approx N \cdot HW/p^2\) the total number of tokens, each LaCT block has complexity \(O(Ld^2)\), so cost is linear in both sequence length and image resolution. The appendix further reports that even **3 layers of standard attention** become slower than **24 LaCT layers** once the token count exceeds about **2M**, corresponding to approximately **256 views at 540×960** [2602.20160].

The distributed scheme shards tokens along the **sequence dimension** across GPUs. Each GPU receives a subset of image tokens; fast-weight updates \(W\) are synchronized across GPUs via **DDP**; each GPU predicts Gaussians for its own virtual views; Gaussians are gathered into the global scene; and each GPU renders some target views and computes losses. This is stated to enable training on **more than 1M tokens** and scaling to **128 views** [2602.20160].

In relation to prior work, the paper positions tttLRM at the intersection of **test-time training**, **NeRF**, **3DGS**, and **long-context 3D reconstruction**. Prior TTT has been used for classification, language models, and pointcloud 3D through systems such as **TTT3R** and **Test3R**, but not for **explicit photorealistic NVS-grade 3D representations**. NeRF provides implicit fields but typically requires slow per-scene optimization; 3DGS is explicit and faster to render but still commonly optimized per scene; GS-LRM and related models provide feedforward GS prediction from few views but remain limited by short sequences and quadratic attention. Long-LRM, **Gamba/MvGamba**, and **Stream3R** address long sequences with attention, SSMs, or causal transformers, but tttLRM instead uses **fast-weight models** that turn long-sequence modeling into online regression with linear complexity [2602.20160].

The stated limitations are equally specific. First, **fixed memory capacity** means that extremely complex scenes or very long sequences can degrade performance. Second, there is a **slight quality loss** relative to the best purely implicit NVS models such as LVSM, reflecting a trade-off between pure NVS quality and explicit, real-time-renderable 3D output. Third, the system has substantial **implementation complexity**, requiring integration of TTT/LaCT and sequence-parallel multi-GPU training. Fourth, **autoregressive drift** and forgetting remain present for very long streams, even though selective update helps [2602.20160].

Taken together, these points place tttLRM as a unified framework in which the same TTT-based backbone and fast-weight memory can support different explicit decoders, including **GS** and **triplane**, and plausibly others by changing the virtual tokens and decoder head. The central conceptual move is to treat fast weights not merely as an adaptation mechanism but as the **latent 3D scene state** from which explicit geometry can be generated.

Source: https://www.emergentmind.com/topics/tttlrm