RW-TTT: Batched Serving for Request-Owned Test-Time Training State

Abstract: Test-time training (TTT) adapts an LLM during generation by reading and updating request-owned state, such as fast weights, low-rank deltas, or streaming learner state. This breaks batched LLM serving, which assumes shared static weights: serial execution is correct but slow, while naive batching can corrupt request state. We formulate this problem as read-write TTT serving and present RW-TTT , which tags each decode step with its owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates only to the owner. On one GPU with eight fast-weight InPlace-TTT streams, RW-TTT reaches 274.61 aggregate tok/s, 9.31x over sequential serving and 3.44x over per-stream replicas under the same memory budget. It preserves behavior on RULER, a long-context benchmark, and passes owner/version checks.

• The paper introduces RW-TTT, which formulates LLM serving as a read-write test-time training problem, clearly distinguishing between state READ and WRITE transitions.
• It details a batching planner that groups requests by backend, operator shape, state effect, and version to ensure atomicity and isolation of mutable states.
• Robust evaluations reveal up to 9.31× throughput improvements and validate behavioral correctness through rigorous benchmarks and operator-level optimizations.

RW-TTT: Batched Serving for Request-Owned Test-Time Training State

Problem Motivation and Formulation

The deployment of test-time training (TTT) techniques in LLM serving introduces mutable, request-owned model-side state, which fundamentally breaks the shared, static-weight invariant that underpins modern high-throughput serving solutions. Conventional approaches such as vLLM, S-LoRA, and Punica either assume strictly-read-only model parameters or restrict per-request state (e.g., adapters) to immutable during inference. In contrast, TTT implies that streams can both read and update private fast weights, low-rank deltas, or streaming learner state dynamically throughout generation, leading to the need for explicit mechanisms to manage state ownership, versioning, and update atomicity.

To address this, RW-TTT formalizes LLM serving as a read-write TTT serving problem, distinguishing between READ transitions (where request-owned state is applied but not mutated) and WRITE transitions (which commit updates to request state). The system ensures versioned, rollback-safe state management that isolates each request’s mutable context, preventing state leakages and write conflicts in a batched environment. This formulation makes batching non-trivial since requests occupy distinct model state and advance at different rates.

Figure 1: Per-request READ/WRITE semantics illustrating sequential evaluation of the shared model with request-owned, versioned TTT state, supporting version-preserving READ and version-bumping WRITE events.

System Design: RW-TTT Serving Contract and Planner

RW-TTT introduces a serving system that tags every decode step by the request’s identity, the expected version, and the READ/WRITE effect. Legal batches are formed by grouping requests with identical backend type, operator shape, state effect, and version, thereby ensuring semantic correctness and atomicity of owner-specific state transitions.

The key primitives of the serving stack are as follows:

• Ownership: Each mutable TTT state is strictly owned by a single request; only that request can trigger state transitions.
• Legal batching: Only requests at the same effect (READ or WRITE), backend, operator shape, and expected version are batched together, with injective owner mapping to preserve isolation.
• State Versioning: Each WRITE event creates a new version; failed updates or speculative execution are handled with owner-local rollback or checkpoint mechanisms.
• Bounded Waiting: The batching planner may delay transitions within a bounded decode-step window to improve batching density, trading off latency.
• Behavioral Contract: The system must preserve sequential TTT reference semantics at the individual request level, validated through numerical and end-to-end task correctness checks.

  Figure 2: Serving architecture—requests maintain owner-versioned mutable state alongside the KV cache; a batching planner orchestrates grouping and assignment for READ/WRITE phases, returning outputs via the explicit owner map.

Evaluation: Throughput, Scaling, and Robustness

RW-TTT is evaluated with a Qwen3-4B In-Place-TTT checkpoint in a scenario comprising eight concurrent streams, 4096 prompt tokens, and 512 decode tokens. The scheduler-inclusive aggregate metric provides a hardware-realistic throughput baseline. Sequential execution yields an aggregate of 29.51 tok/s due to the lack of safe batching, and three naive independent batch-1 TTT replicas under a 33 GiB GPU memory budget achieve 79.78 tok/s. RW-TTT demonstrates significant throughput improvements:

• Aggregate throughput of 274.61 tok/s (9.31× the sequential path, 3.44× the replica baseline).
• Consistent throughput recovery even under bursty or write-heavy stress traces, with speedups above 5.9× over sequential.

  Figure 3: Serving capacity scaling under a 16K prompt; left—runs ordered by peak memory; right—runs ordered by concurrent request units, illustrating memory and parallelism scaling of RW-TTT.

Behavioral and Numerical Correctness

The serving contract is validated under the RULER benchmark suite (32K/64K context tasks). RW-TTT exactly preserves adapted model behavior relative to the sequential In-Place TTT path:

• RULER-32K: 76.41 vs. 76.30
• RULER-64K: 67.72 vs. 67.72

Additional validation layers include per-token NLL, PPL agreement across nine configs, and explicit stress injection of state-version or owner mapping faults (all pass). The contract thus ensures both semantic and calibrated numerical preservation, decoupling changes in server-side parallelism from adaptation semantics.

Operator-Level Optimizations

WRITE-side operations in TTT serving, typically the primary bottleneck, were targeted by custom Triton kernels. Microbenchmarking confirms substantial latency reductions: selective commit, dense update, and checkpoint WRITE show up to 3.1×, 2.5×, and 1.2× speedup over PyTorch eager execution, respectively.

Figure 4: WRITE-side operator speedups, measured as kernelized (Triton) vs. PyTorch eager latency, highlighting key state-commit bottlenecks.

Implications and Future Directions

RW-TTT clarifies the distinction between activation-side stateless batching and the new requirements for tracking, versioning, and isolating mutable model-side state across overlapping requests. The serving architecture decouples backend TTT update rules from server-side correctness, making it agnostic to future fast-weight, LoRA-style, or streaming adaptation methods. Immediate practical implications include enabling scalable deployment of online-adaptive LLMs with in-situ adaptation, without requiring full request isolation or incurring unacceptable throughput penalties.

Theoretically, the explicit read-write versioned state management sets the foundation for correctness-preserving speculative execution, fine-grained rollback, and dynamic co-scheduling policies targeting multi-tenant or heterogeneous adaptation backends. Open questions remain regarding phase skew mitigation, state packing for large-scale streaming, cross-request communication, and integration with ultra-long context or cross-modal adaptation strategies.

Conclusion

RW-TTT presents a robust, versioned batching contract and runtime system for request-owned mutable model state, recovering the serving throughput lost by naïve serialization while retaining strict per-request adaptation semantics. This architectural separation allows the serving stack to efficiently host adaptive LLMs using any underlying online update rule, ensuring scalable, correct deployment for next-generation context- or user-adaptive workloads. The design supports immediate application to both fast-weight and LoRA-style TTT, with extension potential for more general learnable stateful inference workflows.