In-Place Test-Time Training

Abstract: The static train then deploy" paradigm fundamentally limits LLMs from dynamically adapting their weights in response to continuous streams of new information inherent in real-world tasks. Test-Time Training (TTT) offers a compelling alternative by updating a subset of model parameters (fast weights) at inference time, yet its potential in the current LLM ecosystem is hindered by critical barriers including architectural incompatibility, computational inefficiency and misaligned fast weight objectives for language modeling. In this work, we introduce In-Place Test-Time Training (In-Place TTT), a framework that seamlessly endows LLMs with Test-Time Training ability. In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, enabling adrop-in" enhancement for LLMs without costly retraining from scratch. Furthermore, we replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective explicitly aligned with the Next-Token-Prediction task governing autoregressive language modeling. This principled objective, combined with an efficient chunk-wise update mechanism, results in a highly scalable algorithm compatible with context parallelism. Extensive experiments validate our framework's effectiveness: as an in-place enhancement, it enables a 4B-parameter model to achieve superior performance on tasks with contexts up to 128k, and when pretrained from scratch, it consistently outperforms competitive TTT-related approaches. Ablation study results further provide deeper insights on our design choices. Collectively, our results establish In-Place TTT as a promising step towards a paradigm of continual learning in LLMs.

• The paper presents a novel in-place test-time training framework that repurposes MLP output projections as fast weights updated using LM-aligned objectives.
• It achieves computational efficiency through chunk-wise updates, maintaining compatibility with pre-trained Transformer models for seamless integration.
• Empirical results demonstrate improved long-context generalization, reduced perplexity, and robust adaptation in LLMs across diverse benchmarks.

In-Place Test-Time Training: Enabling Continual Adaptation in LLMs

Motivation and Problem Setting

The static “train-then-deploy” paradigm, where weights are fixed after pre-training, imposes major limitations on the adaptivity of LLMs. This constraint impedes dynamic adaptation to streaming data and evolving contexts, especially when reasoning over long-horizon tasks where in-situ weight updates would be beneficial. Prominent solutions, such as in-context learning, are bottlenecked by finite context window sizes and quadratic attention complexity. Architectural modifications like efficient attention or state-space models address window size but not the inability to update model parameters post-deployment.

Test-Time Training (TTT) addresses these shortcomings by introducing “fast weights”—trainable parameters updated at inference—encoded via self-supervised objectives. However, TTT faces three key barriers in LLMs: incompatibility with conventional Transformer architectures (especially pre-trained LLMs), computational inefficiency due to sequential updates that do not utilize hardware parallelism, and objectives misaligned with next-token prediction. Existing TTT approaches often demand model changes, retraining from scratch, or operate as inefficient, sequential token mixers.

In-Place Test-Time Training: Core Framework

The authors propose In-Place Test-Time Training (In-Place TTT), a framework purposely crafted to address the three primary barriers:

Architectural compatibility is achieved by repurposing the final projection matrix of each MLP block in standard Transformers as the fast weights for TTT. The input projections ($\mathbf{W}_{\text{up}}, \mathbf{W}_{\text{gate}}$) remain frozen, preserving general pre-trained knowledge, while the output projection ($\mathbf{W}_{\text{down}}$) is updated in-place during inference. This strategy retains architectural integrity and allows seamless "drop-in" integration with pre-trained LLMs—significantly lowering the adoption barrier relative to TTT layers that require scratch training or disruptive model redesign.

Computational efficiency is obtained via a chunk-wise update mechanism. Instead of expensive per-token updates which impede parallel execution, activations are partitioned into manageable chunks (e.g., 512–1024 tokens). The weights are updated in a causally consistent manner post-processing of each chunk, supporting high parallelism. Furthermore, the update rule is formulated to be associative, enabling context parallelism (CP) for even greater hardware utilization.

Task alignment is realized by introducing an LM-aligned update objective tailored to next-token prediction (NTP), in place of generic self-reconstruction used in prior TTT. Specifically, value targets for TTT are generated by a convolutional projection of the token embeddings, explicitly encoding predictive information relevant for the language modeling task as opposed to merely reconstructing input features.

The overall loop is: for each chunk, activations are computed, current fast weights are applied, and then updated with the LM-aligned objective, as illustrated in the method overview.

Figure 1: The In-Place TTT module operates on input chunks, first applying current fast weights to activations and then updating using an NTP-aligned value derived from embeddings. This cycle enables efficient, causal adaptation at test time.

Theoretical Analysis of LM-Aligned Objective

The authors’ formal analysis leverages the induction head framework to elucidate the superiority of LM-aligned targets: in the canonical setting where a key $k^*$ appears (with value $v^*$) and later reoccurs in the query, it is shown that a reconstruction-based target at best leaves the logit for $v^*$ unchanged (up to orthogonality assumptions), while the LM-aligned objective provably increases the logit for the correct next token with high expectation. This property ensures that TTT updates are maximally informative for NTP tasks, contrasting prior approaches that do not accumulate predictive power through test-time adaptation.

Empirical Results

Drop-In Enhancement for Pre-Trained LLMs

A central claim is that In-Place TTT can be added to pre-trained models such as Qwen3-4B-Base or LLaMA-3.1-8B with minimal training. Empirical evaluation on the RULER benchmark reveals that the In-Place TTT-augmented Qwen3-4B-Base exhibits superior accuracy as context length increases, with gains especially marked at 64k, 128k, and even in context extrapolation to 256k tokens—a critical test for long-horizon inference. Similar improvements are realized for other pre-trained model families and sizes, demonstrating broad applicability.

Superiority over Prior Methods and Efficient Attention

When training from scratch, In-Place TTT outperforms other chunk-wise TTT variants, delta rule models, and efficient attention backbones on sliding window perplexity (SWP) for both 500M and 1.5B model scales. The decrease in perplexity with larger context length indicates effective utilization of context by the TTT mechanism.

Figure 2: In-Place TTT models (500M and 1.5B) consistently realize lower perplexity on the Pile benchmark across large context windows compared to SWA, GLA, DeltaNet, and LaCT.

At 4B-scale, clear gains are observed in both common sense reasoning tasks and long-context evaluation (e.g., RULER-16k accuracy rises from 6.58 to 19.99 with Full Attention backbone). The TTT framework complements, rather than obviates, attention.

Ablation Analyses

Ablations reveal that performance scales with the size of TTT-enabled fast weights (more MLPs), and chunk sizes of 512–1024 yield optimal trade-offs between accuracy and parallelism. Both the convolutional and projection components of the LM-aligned value target are necessary for best generalization on long sequences.

Figure 3: Performance improves with state size, chunk size tuning, and requires both convolutional and projection modules in the LM-aligned objective.

Computational Overhead

Throughput and memory profiling on 4B models with both Sliding-Window and Full Attention show negligible overhead is introduced by In-Place TTT, validating the claim of hardware efficiency.

Figure 4: Prefill throughput and peak memory for In-Place TTT indicate practical deployment feasibility even at high context lengths, across both SWA and full attention backbones.

Implications and Future Directions

This work brings TTT into practical relevance for the current LLM landscape by enabling weight adaptation “in-place,” thus supporting continuous learning without retraining or breaking compatibility with valuable pre-trained weights. The approach is orthogonal and complementary to research on efficient attention, retrieval-augmented architectures, and memory augmentation. Notably, the design admits drop-in integration into any Transformer variant with MLP blocks. The theoretical justification for optimality of the LM-aligned objective points towards a broader principle—test-time trainable parameters gain maximal usefulness when their supervision matches the deployed task.

Practically, this method can unlock improved context generalization, rapid adaptation to domain or task drift, and extend the utility of existing LLMs to longer document windows and evolving streams. The preservation of throughput and memory footprint makes it suitable for scaling to larger models and production settings. Theoretically, the interaction of in-place adaptation with other forms of architectural memory and hierarchy in Transformers is a worthy area for further study. Continued research should explore extensions to other modalities, synergistic integration with state-space models, non-autoregressive objectives, and more sophisticated test-time optimizers.

Conclusion

In-Place TTT enables practical, hardware-efficient continual adaptation in LLMs via minimal architectural intervention: fast weights are seamlessly integrated into existing MLPs and updated using a next-token-prediction-aligned objective. The approach yields consistent empirical improvements in long-context reasoning, context extrapolation, and robust adaptation—both when added to pre-trained models as a drop-in module and when used in pretraining. It bridges the gap between efficient deployment and dynamic continual learning, and sets a new state-of-the-art paradigm for inference-time model evolution in LLMs (2604.06169).