Spatial Test-Time Training (S-TTT)

• Spatial Test-Time Training (S-TTT) is a framework that enables per-instance online adaptation of a compact inner module to enhance spatial reasoning under distribution shifts.
• It specializes in updating only the inner component at test time while keeping the main backbone fixed, ensuring efficient and targeted adaptations.
• S-TTT has been effectively applied to streaming visual intelligence, 3D PET image denoising, and spatially constrained radar echo extrapolation to improve performance.

Spatial Test-Time Training (S-TTT) denotes a family of test-time training mechanisms in which inference includes an online or per-instance adaptation step directed at spatial structure. Recent arXiv literature uses the term in multiple related settings: as a framework for streaming visual-based spatial intelligence in long-horizon egocentric video, where a subset of parameters is updated online to capture and organize spatial evidence over time (Liu et al., 12 Mar 2026); as the spatial-domain adaptation layer in a U-shaped model for 3D PET image denoising, where an inner Spatial Reconstruction Model is updated through self-supervision to correct spatial structural degradations under distribution shift (Yang et al., 9 Jun 2026); and as the spatial-only specialization of a broader Spatio-temporal Test-Time Training block for radar echo extrapolation, obtained by disabling temporal attention so that adaptation targets within-frame structure (Di et al., 4 Jan 2026). Across these usages, S-TTT is defined less by a single architecture than by a common operational principle: only a restricted inner component is adapted at test time, while the surrounding backbone remains fixed or largely fixed.

1. Scope, terminology, and problem setting

The term S-TTT is attached to three distinct but structurally related problem settings. In "Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training" (Liu et al., 12 Mar 2026), the objective is streaming visual-based spatial intelligence on potentially unbounded egocentric video streams. In "U-TTT: Towards Generalizable PET Image Denoising via Test-Time Training" (Yang et al., 9 Jun 2026), S-TTT is the spatial-domain layer that learns-and-adapts per instance during inference for 3D PET denoising. In "REE-TTT: Highly Adaptive Radar Echo Extrapolation Based on Test-Time Training" (Di et al., 4 Jan 2026), S-TTT is not the main named block, but a spatial-only case recovered from the broader ST-TTT design by disabling temporal attention.

Work	What S-TTT denotes	Test-time adapted part
"Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training" (Liu et al., 12 Mar 2026)	a framework for streaming visual-based spatial intelligence built on test-time training	fast weights $W$ and small scale/shift for TTT queries/keys
"U-TTT: Towards Generalizable PET Image Denoising via Test-Time Training" (Yang et al., 9 Jun 2026)	the spatial-domain test-time adaptation layer in a 4-level encoder–decoder	the inner Spatial Reconstruction Model weights $W$
"REE-TTT: Highly Adaptive Radar Echo Extrapolation Based on Test-Time Training" (Di et al., 4 Jan 2026)	a spatial-only TTT recovered from ST-TTT when temporal attention is disabled or identity	the inner-loop parameters $W$ of the learner $f_{in}$

Despite the differing application domains, the motivating difficulty is similar. The spatial intelligence setting emphasizes that the core challenge is not simply longer context windows but how spatial information is selected, organized, and retained over time. The PET denoising setting emphasizes structural blurring, local contrast loss, partial volume effects, motion, and distribution shifts across dose reduction factors and scanners. The radar setting emphasizes non-stationary meteorological distributions across regions and extreme events. In each case, a fixed-parameter model is described as insufficiently adaptive.

A recurrent misconception is that S-TTT is equivalent to generic test-time adaptation or to a larger attention window. The visual-streaming formulation explicitly rejects the idea that the problem is merely one of longer context windows, and the radar formulation explicitly distinguishes its learned reconstruction objective from entropy minimization or batch-normalization-based adaptation. The PET formulation likewise makes clear that the inner objective is a specific spatial reconstruction proxy rather than a general image prior such as TV or SSIM.

2. Core update mechanism and mathematical formulation

In the streaming visual formulation, S-TTT stores spatial evidence in fast weights inside the TTT layers. The fast-weight network is a bias-free SwiGLU MLP,

$f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$

where $W=\{W_1,W_2,W_3\}$ are the fast weights adapted online, and $x$ is $k_t$ for update or $q_t$ for apply. The illustrative point update is

$$W_t = W_{t-1} - \eta \nabla_W \mathcal{L}(f_{W_{t-1}}(k_t), v_t),$$

while the chunked Muon-based update used in the model is

$W$0

A standard choice for the self-supervised memory loss is

$W$1

The layer output combines local attention and the adapted fast-weight memory:

$W$2

These equations define S-TTT as an online associative memory whose parameters are updated inside the streaming loop rather than only during offline training (Liu et al., 12 Mar 2026).

In the PET denoising formulation, the update is per feature tensor rather than over a long stream. Given $W$3, a $W$4 convolution expands channels and the result is evenly split:

$W$5

The inner Spatial Reconstruction Model predicts

$W$6

and the self-supervised loss is

$W$7

The adapted parameters are then obtained by one online gradient step with a learnable inner learning rate $W$8:

$W$9

after which the adapted SRM refines the test feature:

$W$0

The outer denoising objective is

$W$1

Only the SRM weights are updated online; the surrounding denoising backbone remains fixed at test time (Yang et al., 9 Jun 2026).

In the radar formulation, the inner-loop model reconstructs a label view from a training view:

$W$2

and the test-time update is

$W$3

The defining point for S-TTT in that paper is that a spatial-only TTT is recovered by making the temporal module identity and using only spatial attention or masking per frame. This places S-TTT as a strict subset of the broader ST-TTT construction rather than as an independently trained full architecture (Di et al., 4 Jan 2026).

3. Architectural realizations and spatial inductive bias

The streaming visual model adopts a hybrid architecture that interleaves TTT layers with standard self-attention anchor layers at a $W$4 ratio, so that approximately $W$5 of layers are TTT and $W$6 are anchor layers. Anchor layers maintain full-context attention to preserve pretrained visual-semantic alignment and reasoning, while TTT layers compress long context into adaptive fast weights. Inside each TTT layer, Large-chunk updates (LaCT) and Sliding-window attention (SWA) operate in parallel and share Q/K/V projections. LaCT processes tokens in large chunks aligned with multiple frames; SWA applies local attention over a window of size $W$7. To promote spatial awareness, the model applies lightweight depthwise 3D convolution to the Q/K/V of the TTT branch, with learnable scale/shift on Q and K, Dirac-initialized kernels, and spatiotemporal-grid reshaping so that fast weights learn predictive mappings between spatiotemporal contexts rather than isolated tokens (Liu et al., 12 Mar 2026).

The PET denoising realization is markedly different in macro-architecture but similar in adaptation philosophy. U-TTT is a 4-level encoder–decoder denoising network in which each level contains consecutive S-TTT and F-TTT blocks. S-TTT contains an inner SRM that explicitly mixes local spatial connectivity and channelwise nonlinear transformation:

$W$8

The first $W$9 channels pass through a $f_{in}$0 depthwise convolution, while the remaining $f_{in}$1 channels pass through a modified gated linear unit. The paper sets $f_{in}$2 and $f_{in}$3, uses $f_{in}$4 convolutions for pre/post channel mixing, and processes 3D volumes with $f_{in}$5 patches during training and testing (Yang et al., 9 Jun 2026).

The radar work uses a SimVP-style encoder–translator–decoder pipeline with residual TTT blocks in the translator. In full ST-TTT, the projections that define the TTT views are specialized: a temporal attention module for $f_{in}$6, a linear projection for $f_{in}$7, and a motion-enhanced projection for $f_{in}$8. For completeness, the paper gives the generic scaled dot-product attention form

$f_{in}$9

and describes a locality mask and motion-aligned bias,

$f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$0

$f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$1

A spatial-only S-TTT is recovered when temporal attention is disabled or frozen to an identity weighting. This suggests that, in the radar setting, S-TTT is best understood as the spatial restriction of a spatio-temporal TTT design rather than as a separate named family (Di et al., 4 Jan 2026).

4. Data regimes, execution modes, and computational profile

The data regime in the streaming visual setting is explicitly designed to teach fast-weight dynamics. The first stage uses a dense scene-description dataset constructed from object-centric 3D scene graphs (SceneVerse), where each sample pairs a spatial video stream with a comprehensive 3D scene description comprising global context, objects and counts, and object relations. The dataset has approximately 16k samples: 3.6k from ScanNet and 12.5k from ARKitScenes. The second stage uses a 3M spatial VQA corpus, approximately 2.5M open-sourced plus approximately 0.5M self-constructed, covering relative direction/distance, counting, recall/order, route planning, and room size estimation. At inference, the model runs a constant-memory streaming loop with fast weights, an SWA KV cache of capacity $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$2, a TTT pending KV cache, chunk size $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$3, and test-time learning rate $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$4. The reported implementation initializes from Qwen3-VL-2B-Instruct, uses chunk size $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$5, and anneals the window size from $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$6 to $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$7 in the first stage (Liu et al., 12 Mar 2026).

The PET denoising setting is patch-wise and per-instance rather than streaming. Four whole-body PET datasets are used: $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$8 (Base), $f_W(x) = W_2 [\mathrm{SiLU}(W_1 x) \odot (W_3 x)],$9 (OOD-DRF), $W=\{W_1,W_2,W_3\}$0 (OOD-Scanner), and $W=\{W_1,W_2,W_3\}$1 (OOD-Scanner). All datasets are reconstructed using OSEM from list-mode data downsampled to simulate DRFs, and $W=\{W_1,W_2,W_3\}$2 patches are used. The inner update is a single unrolled gradient step embedded in the forward pass for each 3D feature tensor. The outer denoising model is trained with AdamW at learning rate $W=\{W_1,W_2,W_3\}$3 for $W=\{W_1,W_2,W_3\}$4 iterations, batch size $W=\{W_1,W_2,W_3\}$5, patch size $W=\{W_1,W_2,W_3\}$6, and model width $W=\{W_1,W_2,W_3\}$7. The full model has 10.20M parameters and 43.52G FLOPs per forward pass, and the inner update is described as adding minimal overhead because it is a single gradient step in a small inner SRM (Yang et al., 9 Jun 2026).

The radar setting is sequence-based and can continue in streaming mode as frames arrive. The Beijing dataset contains 639 sequences and the Hangzhou heavy-precipitation dataset contains 985 sequences extracted, with 2018 held out for zero-shot testing. Inputs are downsampled to $W=\{W_1,W_2,W_3\}$8, and the forecasting protocol uses $W=\{W_1,W_2,W_3\}$9 past frames to predict $x$0 future frames at 6-minute cadence. The inner-loop adaptation executes per test sequence, the backbone and projection parameters remain frozen, and only the inner-loop parameters of the ST-TTT blocks are updated. The paper states that, with $x$1 inputs and lightweight TA/ME modules, the extra cost is modest and compatible with near real-time operation on a modern GPU such as an RTX 4090 (Di et al., 4 Jan 2026).

The computational rationale differs across domains but converges on the same design choice: adaptation is restricted to a compact inner module. In the visual-streaming model, SWA is $x$2 time and $x$3 memory, the TTT branch is $x$4 with memory $x$5, and the hybrid total is approximately

$x$6

At 1024 frames and 352×480 resolution, the reported requirement is approximately 11.9 GB and approximately 799 TFLOPs, versus approximately 21.2 GB and approximately 1403 TFLOPs for Qwen3-VL-2B; Spatial-MLLM-4B runs OOM at 512/1024 frames. The PET and radar papers do not report the same streaming complexity analysis, but both justify S-TTT by keeping the adapted inner model small and the test-time step count limited (Liu et al., 12 Mar 2026).

5. Empirical performance and ablation evidence

In streaming visual-based spatial intelligence, Spatial-TTT-2B achieves the best overall score on VSI-Bench with Avg 64.4 despite being only 2B. The numerical-task scores are Object Count 70.8, Absolute Distance 47.8 (best), Object Size 71.7, and Room Size 65.9. The multiple-choice scores are Relative Distance 61.8 (second-best), Relative Direction 73.0 (best), Route Plan 47.4 (best), and Appearance Order 77.0 (second-best). On MindCube-Tiny, Spatial-TTT reaches 76.2 ACC, surpassing Gemini-3-pro at 63.9 and MindCube-3B at 51.7. On VSI-SUPER Recall/Count over 10–120 minutes at 1 fps, Recall ACC across durations is 38.3, 35.0, 28.3, 30.0, and Count MRA is 31.8, 45.6, 36.2, 38.4. The ablations are especially diagnostic: removing the spatial-predictive 3D convolution drops Avg from 64.4 to 62.1, removing dense supervision drops Avg to 61.3, and removing hybrid anchoring drops Avg to 53.9, the largest degradation (Liu et al., 12 Mar 2026).

In PET denoising, the component analysis reports PSNR averaged over DRFs or scanners. Baseline (MLP instead of TTT layers) yields 47.63 (Base), 45.68 (OOD-DRF), and 41.89 (OOD-Scanner). S-TTT only improves these to 48.45, 46.07, and 42.43. F-TTT only yields 48.72, 46.55, and 42.78. The combined S-TTT + F-TTT model, U-TTT, reaches 48.91, 46.86, and 43.10. The inner-model ablation shows the same pattern: Linear gives 48.31, 45.97, 42.21; MLP gives 48.57, 46.15, 42.54; $x$7 gives 48.80, 46.77, 42.96; and the proposed DWConv-plus-gated-FC design gives 48.91, 46.86, 43.10. These results support the paper’s claim that the added DWConv branch confirms the importance of spatial connectivity modeling (Yang et al., 9 Jun 2026).

In radar nowcasting, REE-TTT reports top SSIM on Beijing at 0.763 and the best composite event scores FAR 0.453, POD 0.678, CSI_10 0.657, CSI_25 0.451, CSI_35 0.176, and ETS 0.427. On cross-region zero-shot testing in Hangzhou, the model reports MSE 59.155, SSIM 0.540, FAR 0.528, POD 0.596, CSI_10 0.516, CSI_25 0.368, CSI_35 0.200, and ETS 0.315. The explicit ablation relevant to S-TTT is the comparison between attention-based projections and standard linear projections: on Hangzhou zero-shot, ETS improves from 0.306 to 0.315 and CSI_35 from 0.182 to 0.200. The paper states that it did not isolate an explicit S-TTT variant, but expects a pure S-TTT to lie between Linear proj and full ST-TTT (Di et al., 4 Jan 2026).

Taken together, these results indicate that S-TTT is empirically strongest when the spatial inductive bias is closely matched to the domain. In long-horizon video, the crucial additions are hybrid anchoring, large-chunk updates, and the spatial-predictive 3D convolution. In PET, the crucial additions are the DWConv branch and the complementary F-TTT block. In radar, the best results come from spatial adaptation combined with temporal and motion conditioning rather than from spatial adaptation alone.

6. Relation to prior work, limitations, and application domains

All three formulations position S-TTT against conventional test-time adaptation. The visual-streaming paper argues that simply scaling attention windows faces quadratic costs and poor scaling, and that token-wise linear Q/K/V projections ignore local spatiotemporal structure. The PET paper contrasts its differentiable inner-loop layer design with conventional fixed-parameter models and with earlier TTT methods whose auxiliary objectives are not directly aligned with the primary task at test time. The radar paper contrasts its reconstruction-based inner objective and LayerNorm-based residual TTT blocks with TENT-like entropy minimization and BN-only updates. A common conclusion is that S-TTT is most useful when the test-time objective is tied to the structure the downstream task actually needs (Liu et al., 12 Mar 2026).

The limitations are similarly domain-specific. In the visual-streaming setting, anchor layers still incur quadratic attention cost, fast-weight drift remains possible despite Muon orthogonalization and L2 normalization, the model does not explicitly reconstruct 3D geometry or use metric depth, and depthwise 3D kernels assume local continuity. In the PET setting, inner-loop hyperparameters and SRM capacity must be chosen to avoid overfitting to noise patterns in extreme low-dose cases, and excessively aggressive spatial updates could oversmooth high-frequency detail. In the radar setting, some averaging in upsampling can blur the strongest convective cores, richer spatio-temporal attention designs remain to be evaluated, and aggressive inner-loop settings can overfit transient noise in sparse echoes (Yang et al., 9 Jun 2026).

The application profile follows directly from these design choices. Spatial-TTT is targeted at robotics navigation and manipulation, AR/VR scene understanding, and long-horizon video spatial reasoning tasks such as distance or direction queries, route planning, room-size estimation, appearance order, recall, and counting. U-TTT targets robust clinical PET denoising under unseen dose levels and unseen scanners, and the paper proposes low-dose CT denoising and MRI reconstruction as plausible extensions of the same S-TTT design. REE-TTT targets precipitation nowcasting, especially cross-regional and extreme-event scenarios where non-stationary meteorological distributions make static models brittle (Di et al., 4 Jan 2026).

A general interpretation suggested by these works is that S-TTT is not a single canonical module but a recurring design pattern for spatially informed inner-loop adaptation. What remains constant is the restriction of test-time updates to a compact inner learner, the use of self-supervision during inference, and the attempt to encode spatial continuity, locality, or geometry directly into the adaptation mechanism. What changes from domain to domain is the representation on which the update acts: fast weights over long video streams, a 3D spatial reconstruction model over PET features, or a spatially restricted attention-based learner inside a radar nowcasting model.