Fast Spatial Memory with Elastic Test-Time Training

Abstract: Large Chunk Test-Time Training (LaCT) has shown strong performance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfitting. As a result, LaCT is typically instantiated with a single large chunk spanning the full input sequence, falling short of the broader goal of handling arbitrarily long sequences in a single pass. We propose Elastic Test-Time Training inspired by elastic weight consolidation, that stabilizes LaCT fast-weight updates with a Fisher-weighted elastic prior around a maintained anchor state. The anchor evolves as an exponential moving average of past fast weights to balance stability and plasticity. Based on this updated architecture, we introduce Fast Spatial Memory (FSM), an efficient and scalable model for 4D reconstruction that learns spatiotemporal representations from long observation sequences and renders novel view-time combinations. We pre-trained FSM on large-scale curated 3D/4D data to capture the dynamics and semantics of complex spatial environments. Extensive experiments show that FSM supports fast adaptation over long sequences and delivers high-quality 3D/4D reconstruction with smaller chunks and mitigating the camera-interpolation shortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck.

• The paper introduces Fast Spatial Memory (FSM) with LaCET, a method that integrates elastic weight consolidation into fast-weight adaptation for stable, long-context 4D scene reconstruction.
• It employs chunked test-time training and EMA-based anchor updates to balance plasticity and stability during neural rendering across dynamic, multiview scenes.
• Results show improved PSNR, SSIM, and reduced LPIPS, highlighting FSM's potential in scalable 4D novel view synthesis over extended sequences.

Fast Spatial Memory with Elastic Test-Time Training: Technical Summary

Introduction

This work presents Fast Spatial Memory (FSM), an efficient, scalable approach for long-context 4D scene reconstruction that advances the capacity for neural rendering from posed, multiview and temporally-extended observations. Central to FSM is the introduction of Large Chunk Elastic Test-Time Training (LaCET), an architectural and algorithmic generalization of Large Chunk Test-Time Training (LaCT), augmented by elastic consolidation to stabilize fast-weight dynamics during inference.

The motivation behind LaCET and FSM stems from persistent challenges in spatiotemporal modeling—specifically, the high activation-memory requirements of transformer-based sequence models for rendering, and the instability/catastrophic forgetting induced by fully plastic fast-weight adaptation over extended sequences. FSM leverages an elastic stability mechanism inspired by elastic weight consolidation (EWC) to regularize adaptation, supporting robust, high-fidelity inference from arbitrarily long input streams.

Figure 1: Model overview (left) and LaCET block structure (right); FSM assimilates visual tokens (with temporal and camera metadata) and infers arbitrary view-time queries via online-adapted fast weights.

Methodology: Elastic Chunked Test-Time Adaptation

Fast Weights and Test-Time Training

At its core, FSM builds on the fast-weight paradigm, where neural parameters for attention layers are updated online—both at train and test time—allowing immediate adaptation to context variation. The vanilla TTT mechanism, however, struggles with memory efficiency and instability. LaCT improves throughput by chunk-wise updates, but remains sensitive to inference-time drift and overfitting during long-horizon adaptation.

Elastic Weight Consolidation for Inference Stability

LaCET addresses these issues by introducing an elastic penalty, analogously to EWC in continual learning. Fast-weight parameters maintain an anchor (EMA of past fast weights) and an online-estimated Fisher importance matrix. After each chunk update, LaCET softly restores critical weights towards the anchor, suppressing drift in parameters that are essential for stable behavior, while allowing less critical components to adapt freely. This mechanism achieves a tunable balance between plasticity and stability in fast adaptation over extended streams.

Sequence Modeling and Feature Design

The FSM architecture processes patchified, temporally-annotated image sequences. Visual tokens integrate RGB, Plücker ray maps (providing explicit geometric augmentation), and timestamp embeddings. FSM is compatible with both LVSM-style (direct patch regression) and LRM-style (explicit Gaussian splat-based rendering) decoders.

Figure 2: FSM-LVSM system architecture for flexible 4D view-time synthesis.

Anchor update policies (global, streaming, streaming-EMA) and Fisher estimation strategies are systematically evaluated, demonstrating the superiority of streaming-EMA for robust temporal memory. The photometric objective combines MSE and LPIPS losses, and FSM is pretrained on diverse static/dynamic scenes, both real and synthetic, with consistent normalization of spatial and temporal domains.

Ablations: Elasticity and Generalization in Long Context

Extensive ablations elucidate the impact of LaCET's elasticity on generalization and stability:

• Without elasticity (fully plastic LaCT), test-time updates quickly diverge or overfit, limiting chunk granularity and sequence scalability.
• With LaCET, the generalization gap between train and test performance is significantly reduced, with higher PSNR and SSIM, and lower LPIPS under variable chunking and temporal coverage.
• Elasticity enables multi-chunk adaptation (as opposed to single-chunk LaCT), mitigating the activation-memory bottleneck and supporting substantially longer input streams.

  Figure 3: Ablation analysis—qualitative performance under matched conditions, validating the regularization and stability gains of elastic consolidation.

  

Figure 4: Test-time scaling curves; LaCET (with elasticity) markedly improves robustness with sparser or more temporally distant contexts.

Main Results: Novel View Synthesis in 4D and 3D

Quantitative Evaluation

• FSM-LVSM achieves PSNR 32.16 / SSIM 0.931 / LPIPS 0.043 on the Stereo4D 4D NVS benchmark (256×256), surpassing both explicit-geometry and alternative rendering-based baselines.
• On the NVIDIA dynamic scene benchmark, FSM-LVSM (256×256) outperforms all fast feedforward models and approaches per-scene optimization-based methods.
• In the static 3D regime (DL3DV-140), FSM-LVSM attains performance on par or better than prior LVSM and LRM models at comparable resolution, and dominates in the dynamic setting among models tractable at scale.

  Figure 5: Qualitative comparison on Stereo4D; FSM achieves sharper structure, temporal consistency and fewer ghosting artifacts compared to baseline models.

  

  Figure 6: Qualitative comparison on DL3DV, illustrating generalization to static and dynamic geometry.

Critically, FSM achieves these results with reduced overfitting to camera-pose interpolation shortcuts, a recognized failure mode in models lacking explicit elasticity. The multi-chunk, elastic fast-weight dynamics better capture long-range, non-local associations between appearance, geometry, and temporal change, critical for accurate 4D scene representation.

Implications, Theoretical and Practical

Theoretically, FSM establishes that in-forward, elastic regularization over fast weights enables effective generalization and memory retention in chunked sequence adaptation, overcoming longstanding issues in test-time adaptation and continual sequence modeling. Practically, this framework enables scalable, efficient 4D reconstruction with bounded activation-memory, broadening applicability to embodied AI, robotics, AR/VR, asset generation, and video understanding over long-duration, dynamic input streams.

Importantly, FSM's architectural neutrality—being compatible with both geometry-free and explicit-geometry decoders—suggests that the elasticity principle can further be extended or hybridized to alternative rendering targets and memory architectures. However, the results also indicate remaining limitations in geometric faithfulness and motion consistency when supervision is limited to photometric (appearance) reconstruction; integrating geometric or correspondence signals remains an open path to further improvements.

Figure 7: Additional comparison—FSM-LVSM maintains accuracy and visual fidelity across a broader range of challenging scenes.

Figure 8: Diverse qualitative results on complex, dynamic scenes from Stereo4D.

Figure 9: Failure case—FSM exhibits ghosting under extreme view extrapolation and complex dynamic motion, underscoring continuing challenges in 4D geometric consistency.

Conclusion

FSM with LaCET represents a significant technical advancement in scalable, robust 4D spatiotemporal memory construction via elastic test-time adaptation. The algorithmic innovations demonstrably close the generalization gap in long-context novel-view-time synthesis and pave the way for practical, high-resolution, efficient long-sequence inference. While challenges remain in achieving fully geometry-faithful 4D representations, especially under weak supervision, the elastic regularization principles established here form a foundation for next-generation continual, adaptive, and memory-efficient scene models.

For further technical details and extended experiments, see "Fast Spatial Memory with Elastic Test-Time Training" (2604.07350).