Titans: Learning at Test Time

• The paper introduces Titans, a novel architecture that integrates long-term neural memory into sequence models for continual test-time adaptation and improved long-context processing.
• It leverages a multi-component memory system—including short-term attention, neural long-term memory, and persistent tokens—to overcome limitations of conventional Transformers and RNNs.
• Empirical results across language, vision, and time-series domains show reduced perplexity, enhanced retrieval, and up to 17× inference acceleration compared to standard models.

Titans: Learning at Test Time is a paradigm and architectural framework for integrating long-term, explicit neural memory modules into sequence models—especially Transformers—enabling continual, online adaptation and memorization at inference via test-time learning. The approach targets fundamental limitations of conventional attention and recurrence (context size constraints, limited long-range recall) by introducing a deep memory component that is updated online in response to novel or surprising input, thereby bridging the gap between meta-learning and scalable memory architectures. Building on key ideas from meta-learning, memory-augmented neural networks, and reinforcement learning, Titans and its derivatives support flexible, sample-efficient test-time specialization, continual adaptation, and scaling to multi-million-token contexts across language, reasoning, vision, and time series domains (Behrouz et al., 2024, Nepi et al., 10 Oct 2025, Behrouz et al., 29 May 2025).

1. Motivation and Memory Landscape

Conventional RNNs, SSMs, and even state-of-the-art Transformers display distinct limitations in long-context reasoning and scalable memory management. RNN-based models compress history into a fixed-size hidden state, resulting in rapid loss of distant dependencies as context extends. Transformer attention, in contrast, provides precise recency-weighted retrieval but incurs quadratic computation and memory with context length $N$ (O$(N^2)$), making attention infeasible for million-token contexts and precluding accumulation of knowledge across arbitrary time scales (Behrouz et al., 2024). From a memory-theoretic viewpoint, attention serves as accurate short-term memory, while recurrences offer persistent yet heavily compressed long-term memory.

Titans address these limitations by architecturally separating short-term (attention window) from long-term (explicit neural memory) mechanisms, and crucially, by equipping the memory with the ability to adapt its parameters—via a deep associative module—during test time, using gradient-based updates controlled by surprise, momentum, and gating. This design draws inspiration from neuropsychological concepts of multi-component memory and from the observation that effective learning at inference is critical for generalization and continual adaptation (Behrouz et al., 2024, Nepi et al., 10 Oct 2025).

2. Titans Architecture and Memory Update Mechanisms

The canonical Titans architecture integrates three memory "heads" in each sequence-processing layer:

• Core (short-term memory): A local-window attention processor (can be a Transformer block restricted to chunked windows).
• Neural long-term memory (LTM): A deep multi-layer perceptron (MLP)-based associative memory $M_t$ whose parameters can be updated online. The module handles explicit key-value memorization and surprise-driven updates.
• Persistent memory tokens: A small set of parameters $P$ prepended to every chunk; these tokens remain fixed and anchor cross-chunk information flow.

The core update procedure at time $t$ for input $x_t$ is as follows (Behrouz et al., 2024, Nepi et al., 10 Oct 2025):

• Key/value projections: $k_t = W_K x_t$, $v_t = W_V x_t$
• Associative loss: $\ell(M_{t-1}; x_t) = \| M_{t-1}(k_t) - v_t \|_2^2$
• Surprise/momentum accumulation: $$S_t = \eta_t S_{t-1} - \theta_t \nabla_M \ell(M_{t-1}; x_t)$$
• Forgetting gate: $(N^2)$0 decides blend between old memory and the surprise update.
• Online update: $(N^2)$1
• Query/retrieval: Given a query $(N^2)$2, $(N^2)$3 (no weight update during read).

The chunked design enables parallelization via scans and batched updates, with O$(N^2)$4 scaling in the sequence length $(N^2)$5 and the hidden dimension $(N^2)$6. This memory module is meta-learned during training, with all outer-loop parameters (e.g., projections, memory depth, gating schedules) optimized via backpropagation through the test-time (inner loop) memory updates.

3. Variants and Integration Strategies

Several integration schemes (also termed "Editor's term": memory integration archetypes) mediate how the Titans memory module interacts with attention and the core model (Behrouz et al., 2024, Nepi et al., 10 Oct 2025, Chen et al., 7 Jun 2026):

• Memory as Context (MAC): The memory readout is injected as context tokens into each chunk, allowing attention to process $(N^2)$7 with the rest of the windowed sequence.
• Memory as Gate (MAG): Attention and memory outputs are combined via a learned gating mechanism, modulating per-token contributions from short- and long-term streams.
• Memory as Layer (MAL): The memory is stacked as an explicit residual or functional layer (notably used for test-time speech emotion recognition (Chen et al., 7 Jun 2026)), with memory read/write and injection as residuals in Transformer blocks.
• Pure LMM: The deep memory module operates standalone, without attention, serving as a control for long-range recall capacity.

Dense placements of MAL branches and careful gating initialization are found crucial in practical systems.

4. Test-Time Learning and Chunkwise Parallelization

At inference, Titans perform online memory adaptation for each input token or chunk, efficiently updating neural memory using batch-parallel associative gradient updates. The TNT (Test-time and chunkwise Training) paradigm extends the scalability of Titans by decoupling training chunk sizes from inference chunk sizes via a hierarchical memory structure (Li et al., 10 Nov 2025). A global memory processes large context blocks, enabling hardware-friendly batch parallelism, while multiple local memories handle fine-grained adaptation and are periodically reset for parallelization. A brief stage of fine-tuning adapts only local memories to small chunks, ensuring fine-grained test-time updates and maximizing accuracy.

This two-stage approach eliminates the traditional throughput/accuracy trade-off, achieving up to 17× acceleration over naïve training and, after fine-tuning, delivering perplexity and reasoning improvements beyond baseline linear-RNN and Transformer models.

5. Empirical Results and Benchmark Performance

Comprehensive empirical validation across multiple domains demonstrates the strength and versatility of the Titans framework (Behrouz et al., 2024, Behrouz et al., 29 May 2025, Li et al., 10 Nov 2025):

• Language modeling & reasoning: On Wikitext, Lambada, PIQA, HellaSwag, WinoGrande, ARC-Easy/Challenge, SIQA, and BoolQ, Titans reduce perplexity (e.g., Wikitext ppx ≈24–27 vs. Transformers’ 28–32) and yield higher accuracy (+1–3% absolute gains).
• Needle-in-a-haystack retrieval: Pure LMM and MAC variants achieve near-perfect (≈98–100%) long-range retrieval, outperforming baselines such as DeltaNet, Mamba, and linear RNNs.
• BABILong (million-token context): Titans MAC outperforms much larger recurrent and Transformer models in few-shot multi-fact retrieval, with ATLAS further improving long-context accuracy to ~80% at 10M token context (Behrouz et al., 29 May 2025).
• Recommendation and time series: LMM outperforms LSTM and iTransformer baselines in next-item prediction and time-series regression.
• Speech emotion recognition: MAL residual adapters add 0.5–2 F1 points over strong LALM+LoRA models across diverse SER datasets (Chen et al., 7 Jun 2026).
• Training efficiency: With TNT, Titans scale linearly in context and outpace Transformer-FlashAttention in throughput at very long context lengths (Li et al., 10 Nov 2025).

Ablation studies show long-term memory (neural memory) is the key source of improvement, while persistent tokens and static adapters add little.

6. Specializations and Extensions

The design of Titans supports rich extensions:

• ATLAS: Replaces online, greedy memory update with a locally optimal, high-capacity, deep memory module using a parallelized second-order optimizer (Muon), polynomial/exponential feature expansion, and a sliding window of recent tokens. ATLAS generalizes memory kernels to approximate softmax attention and achieves top accuracy on extreme context-length tasks (Behrouz et al., 29 May 2025).
• Test-time curricula (TTC-RL): Leverages Titans as a substrate for targeted RL adaptation via test-time task-specific curricula, selected from massive task banks using posterior-variance-minimizing search (SIFT) (Hübotter et al., 6 Oct 2025). RL updates employing GRPO with advantage normalization provide strong improvements on math/coding evaluation (e.g., Qwen3-8B pass@1 on AIME25: 23.3%→41.7%; CodeElo: 13.7%→29.3%).
• Domain adaptation: MAL is used as a plug-in test-time memory for emotion recognition without backbone adaptation, serving conversational context at inference via differentiable memory that accumulates dialogue state (Chen et al., 7 Jun 2026).
• Test-time meta-learning frameworks: Connections to meta-test time training (MTTT) clarify how inner-loop adaptation with neural memory subsumes linear and softmax attention and enables efficient inner-loop adaptation for structured inputs (Sun et al., 2023).

7. Critical Analysis and Limitations

Despite substantial progress, limitations remain. Chunking introduces a trade-off between computational efficiency and global context integration—small chunks degrade local accuracy, while large chunks negate O$(N^2)$8 cost savings (Nepi et al., 10 Oct 2025). Direct test-time memory updates, in the absence of co-adaptation with the backbone, may encounter adaptation mismatches or plateau. Persistent memory tokens provide limited benefits and may be outweighed by more expressive, dynamically managed memory (Nepi et al., 10 Oct 2025). Parameter and gate initialization are critical for stable learning, and hyperparameter tuning (e.g., memory depth, chunk size, update rates) is required for robust results across domains. Furthermore, the absolute performance of Titans in recommendation and language modeling can fall short of highly optimized Transformer baselines, highlighting ongoing gaps (Nepi et al., 10 Oct 2025).

Key directions for future research include joint adaptation of backbone and memory at test time, dynamic or learned chunking strategies, modular or sparse memory management, integration with policy distillation for rapid meta-adaptation, and further exploration of test-time RL or self-supervision (Hübotter et al., 6 Oct 2025, Behrouz et al., 29 May 2025). Empirical and theoretical analyses consistently indicate that deep, feature-augmented neural memory with gradient-based test-time adaptation forms a critical ingredient for scalable, continual, and reasoned sequence processing.