Miles v0.1: Production-Level Post-Training

Abstract: We present Miles v0.1, a full-stack, production-ready system for frontier post-training. Building upon the clean design of slime, Miles designs each stage of the reinforcement-learning (RL) training loop around a single principle: components should be verified, clean, and customizable. With accuracy, efficiency, reliability, and scalability as first-class goals, Miles aims to make frontier-scale RL accessible to researchers and enterprises alike. This report walks through the system end to end: rollout engines built on SGLang, a trainer with a choice of two backends (NVIDIA Megatron-LM and PyTorch FSDP), and three weight-synchronization transports for different deployment topologies. Beyond full-parameter RL, Miles also supports LoRA RL, on-policy distillation, supervised fine-tuning, and true-on-policy rollout-training alignment, and extends the same architecture to diffusion models. We close with an end-to-end case study: fully asynchronous agentic RL on a GLM-5.2 744B-A40B model over terminal-use coding tasks, running on 64 NVIDIA GB300 GPUs with a median step time of 263 seconds over the first 30 measured steps. Miles is open-sourced at https://github.com/radixark/miles, with the project website at https://miles.radixark.com.

• The paper outlines Miles v0.1, a full-stack post-training system that enhances numerical fidelity in rollout, training, and evaluation of front-tier scale RL.
• This innovation introduces asynchrony between separate GPU pools for rollout and training, achieving a 56% reduction in rollout time and up to a 86.4% reduction in weight-update time..
• The system also emphasizes preserving exact token IDs across rollout and training to maintain consistency, situating the system at the forefront of large model optimization in RL.

System objective and design

Miles v0.1 presents a full-stack post-training system designed for frontier-scale language-model RL, with emphasis on numerical fidelity, throughput, deployment flexibility, and explicit verification boundaries. Its central architectural decision is to treat rollout, training, and weight synchronization as separable but coordinated subsystems. This permits the same infrastructure to support synchronous and asynchronous RL, full-parameter and LoRA optimization, on-policy distillation, supervised fine-tuning, and diffusion-model post-training. The system is implemented around SGLang rollout engines, Megatron-LM or PyTorch FSDP training backends, and multiple weight-transfer mechanisms (2609.08368).

The paper’s motivating problem is that contemporary agentic RL differs materially from a short-completion generate–update loop. Trajectories can contain many turns, tool calls, long contexts, external environments, and MoE routing decisions. Rollout and training have opposing systems requirements: rollout prioritizes latency and KV-cache reuse, whereas training prioritizes throughput and collective communication. More importantly, a trajectory can be numerically or token-wise different when reconstructed by the trainer from the messages supplied by an agent harness. Miles therefore treats fidelity as a first-class systems invariant rather than as an incidental property of the data pipeline.

The basic loop contains three stages: SGLang generates trajectory groups, the trainer computes an objective and updates the policy, and the updated weights are synchronized back to the rollout fleet.

Figure 1: The Miles RL loop connecting rollout generation, policy training, and weight synchronization.

A trajectory group contains multiple attempts at the same prompt, which is necessary for group-relative objectives such as GRPO (Shao et al., 2024). The system deliberately operates at group granularity when grouping, buffering, filtering, and discarding data. This avoids breaking the statistical unit required by group-relative advantage estimation.

Asynchronous rollout and execution scheduling

Miles supports fully asynchronous RL in which rollout generation and training occupy disjoint GPU pools and progress concurrently. The rollout engines continuously produce trajectories while the trainer consumes completed groups from a bounded buffer. This avoids the synchronization barrier imposed by synchronous batches, where the trainer waits for the slowest trajectory and the rollout fleet subsequently waits for the optimizer.

The bounded buffer is not merely a queue. It is the control point for admission, retry, filtering, and staleness management. A group may be rejected because generation aborted, because a user-defined filter found no usable advantage signal, or because the group was generated under weights older than the configured staleness limit. Staleness is conservatively defined using the oldest weight version represented anywhere in the group. This is especially important for multi-turn episodes, whose individual turns may be generated under different policy versions.

Miles exposes queue occupancy, mean and maximum staleness, and discarded-group counts on every training step. These metrics distinguish rollout starvation from trainer saturation. A permanently empty queue indicates inadequate generation capacity; a full queue accompanied by increasing staleness indicates that training is the bottleneck. This observability is a substantive contribution because asynchronous RL can degrade silently through wasted rollouts and stale data without producing an execution failure.

Generation capacity is replenished either at group granularity or sample granularity. The default sample-granularity policy immediately replaces completed trajectories, maintaining approximately constant concurrency even when trajectory lengths vary by an order of magnitude. The tradeoff is that a trajectory waiting on a tool call continues to occupy a slot, so the configured in-flight limit is not identical to the number of actively decoding requests.

Evaluation is also treated as a scheduling problem. Miles provides shared-engine, dedicated-fleet, and external evaluation modes. Shared-engine evaluation pauses new generation, whereas snapshot-based evaluation can run concurrently with training after snapshot export. Crucially, evaluation results are associated with the policy version actually evaluated, not simply the step at which the result arrives.

Figure 2: Shared-engine and snapshot-based evaluation modes on the asynchronous training timeline.

Token fidelity and MoE routing consistency

The paper’s strongest technical claim is that multi-turn agentic RL requires preserving the exact token IDs sampled by the rollout engine. Reconstructing a trajectory from messages can alter chat-template rendering, tool-call serialization, reasoning fields, or historical context. Such changes produce a training sequence that the policy did not generate, invalidating the correspondence between rollout and trainer log-probabilities.

Miles addresses this with token-in-token-out (TITO) sessions. The session server owns tokenization, stores prompt and completion token IDs, records rollout log-probabilities, and checkpoints the resulting history after every successful turn. Subsequent turns reuse the deepest valid token checkpoint and tokenize only the appended suffix.

Figure 3: Token-in-token-out sessions preserve the exact token IDs emitted by the rollout engine for later training.

The approach supports both linear and branching sessions. Linear sessions permit only tail extension or a limited retry, producing one training sequence. Branching sessions retain an append-only history tree and allow multiple trajectories to emerge from a common prefix. This supports agent harnesses that fork, compact, or otherwise reshape their context.

The paper is explicit about the risks of relaxed replay matching. A permissive matcher that ignores tool calls can merge histories whose visible text is identical but whose actions differ. Because Miles treats the stored token snapshot as authoritative after a match, this can silently train on a tool history that never occurred. Strict comparison of template-consumed fields is therefore the safe default.

TITO is guarded by model-family registrations and append-only tokenization tests. The CPU test is insufficient by itself, since parser behavior during live inference can violate an invariant that holds under isolated template rendering. A GPU test with the real model, stop-token handling, and tool-call parser is also required. Vision-language inputs remain unsupported by the session server, so multimodal models must use a lower-level token interface.

Exact tokens do not fully solve the MoE consistency problem. Rollout and training can still select different experts because of numerical differences, kernel differences, or precision differences. Rollout Routing Replay (R3) records the expert assignments for each token and replays them during training, preventing the trainer from routing a sampled token through a different expert path (Ma et al., 13 Oct 2025). This is particularly relevant because routing discrepancies can assign gradient updates to experts that did not contribute to the sampled action.

R3 carries a nontrivial memory cost. For a 32K-token sequence with 60 layers and top-8 routing, the recorded routing tensor occupies approximately 60 MB per trajectory. The paper consequently treats R3 as a recipe-level choice rather than a universal default. The GLM-5.2 case study does not enable it, and the authors note that asynchronous weight staleness introduces additional mismatch sources that R3 cannot remove.

Training precision, memory, and backend abstraction

Miles frames numerical precision as a contract shared by rollout and training. Quantizing only one side, or applying distinct quantization procedures, can generate substantial train–rollout mismatch. The system therefore implements common quantization logic across checkpoint conversion, trainer forward passes, rollout inference, and live weight export.

The supported end-to-end recipes include BF16, blockwise FP8, MXFP8, and NVFP4. FP8 blockwise execution is available on NVIDIA Hopper and Blackwell and selected AMD hardware; MXFP8 and NVFP4 require Blackwell. MXFP8 and NVFP4 remain beta technologies and have only been tested on specified model families. The paper does not claim architecture-independent validity.

Miles also provides dequantized backward execution for NVFP4 and the Four Over Six adaptive block-scaling method (Cook et al., 1 Dec 2025). These mechanisms have different scopes: dequantized backward changes only the training backward pass, while Four Over Six changes quantized values and therefore must be identically enabled in both trainer and rollout kernels.

Memory management combines actor offloading with optimizer-state streaming. Actor offloading moves the paused training process out of GPU memory, potentially to host RAM or node-local disk. Optimizer streaming keeps optimizer state on disk and loads only the buckets needed for each update. On Qwen3-30B-A3B, the latter reduces actor offloading from 24 seconds to 5.2 seconds and reloading from 8.9 seconds to 1.3 seconds. The performance benefit is accompanied by operational restrictions: streamed checkpoints require the same parallel layout on resume, cannot resume from checkpoints written without streaming, and may block during checkpoint saves.

Megatron-LM and FSDP expose a common trainer interface but target different operating regimes. Megatron supports tensor, pipeline, context, expert, and expert-tensor parallelism, making it the principal backend for large MoE models. FSDP loads Hugging Face checkpoints directly and is more convenient for new architectures and data-parallel-scale experiments. Current LoRA support is limited to Megatron, and disk-based offloading beyond host memory is unavailable in the FSDP path.

Objectives and mismatch correction

Miles separates advantage estimation from loss implementation. The available estimators include GRPO, GSPO, REINFORCE++, and PPO with a learned value function. A typed loss interface also allows supervised objectives and user-defined losses to reuse the training stack.

Because rollout and training generally disagree on token probabilities, Miles computes an importance ratio from their log-probability difference. Two correction schemes are provided. Truncated importance sampling clamps the ratio and retains the token with a bounded weight. Clip-or-pop discards tokens whose ratio lies outside the permitted interval. The default interval is $[0,2]$. The system reports the unclipped ratio, clipped-token fraction, and mean absolute deviation from one, enabling users to diagnose whether correction is compensating for a benign numerical discrepancy or substantial policy mismatch.

True-on-policy alignment targets the stricter case in which rollout and training assign exactly the same probability to every sampled token. It uses common attention and matrix-multiplication kernels, deterministic execution, batch-invariant operations, matching rotary and activation implementations, and a rollout-side prefill rescore. For supported Qwen3 dense configurations, the reported absolute log-probability difference is exactly zero.

This guarantee is deliberately narrow. It covers sampled-token log-probabilities, not equality of the entire output distribution, and it does not address trajectories generated under older weights. It is currently registered only for selected Qwen3 0.6B and 4B configurations. The throughput cost of deterministic and batch-invariant execution is also acknowledged.

Weight synchronization across deployment topologies

Weight transfer can dominate frontier-scale RL. The paper reports that a full NCCL update for Kimi K2 1T-A32B takes nearly a minute, motivating three transports: NCCL broadcast, RDMA-based peer-to-peer transfer, and disk-delta updates.

P2P transfer assigns training ranks to rollout ranks and writes serving-ready shards directly into rollout memory. The approach uses a CPU-resident SGLang replica to reuse serving-side reshaping and sharding logic, avoiding a second implementation of model partitioning. On H100 systems with a 1 GB bucket, P2P reduces update time from 58.30 to 8.48 seconds for GLM-5 744B-A40B and from 53.28 to 7.23 seconds for Kimi K2 1T-A32B.

Model	Broadcast	P2P	Reduction
Qwen3-30B-A3B	2.67 s	2.16 s	19.1%
GLM-5 744B-A40B	58.30 s	8.48 s	85.5%
Kimi K2 1T-A32B	53.28 s	7.23 s	86.4%

The result is strongly topology-dependent. P2P can be up to approximately 70% slower than broadcast on a single node because it lacks additional aggregate bandwidth and incurs host-side reshaping and pinned-memory staging. The paper therefore retains broadcast as the default and restricts P2P’s practical recommendation to multi-node deployments with suitable model mappings.

Disk-delta updates avoid direct trainer-to-engine connectivity. Rollout hosts share a base checkpoint, receive compressed byte-level deltas through a shared filesystem, validate checksums, and reload the patched checkpoint. XOR deltas are compact but non-idempotent; overwrite deltas are larger but safe to apply repeatedly. The method is restricted to Megatron and is incompatible with colocation, LoRA, and prefill–decode disaggregation in the current implementation.

Miles also supports initial weight-verification checks that intentionally fill rollout tensors with random values before synchronization. A missing tensor update therefore cannot pass unnoticed. This is an effective validation design, although the check is intended for debugging and continuous integration rather than routine production execution.

Additional post-training recipes

LoRA RL makes the adapter rather than the full model the synchronization and optimization unit. This reduces optimizer-state memory, training arithmetic, and weight-transfer volume. In colocated deployments, adapters can be transferred through interprocess communication; in disaggregated deployments, serving-ready adapter tensors are broadcast over NCCL. Support depends on coordinated implementation in Megatron, Miles’s export path, and SGLang, so model-name similarity alone does not establish compatibility.

On-policy distillation uses student-generated trajectories while a teacher scores the same sampled tokens. The difference between student and teacher log-probabilities becomes a per-token reverse-KL estimate and is incorporated into the advantage.

Figure 4: On-policy distillation uses teacher scores on student-generated tokens, optionally combined with task reward.

In the reported Qwen3.5-35B-A3B experiment, response length fell from 14,070 to 6,132 tokens over five steps, a 56% reduction, while accuracy changed from 84.0% to 85.2%. The paper correctly qualifies the latter result: the 1.2-point change lies within an evaluation standard error of approximately 1.6 points. The defensible conclusion is reduced response length without a reliable accuracy change, not benchmark improvement.

Miles-Diffusion generalizes the architecture to image and video diffusion RL. Trajectories contain denoising states and per-step log-probabilities, while an FSDP2 trainer re-scores selected steps. Streaming raw tensor bytes and overlapping deserialization, reward computation, and generation reduces LTX-2.3 rollout time from 157.4 to 87.6 seconds per step and total step time from 321.9 to 252.1 seconds. The system provides deterministic execution and per-parameter dtype controls because diffusion importance ratios are especially sensitive to small train–inference discrepancies.

GLM-5.2 case study

The end-to-end reference run trains GLM-5.2 744B-A40B on terminal-use coding tasks using 64 NVIDIA GB300 GPUs. Thirty-two GPUs perform rollout and 32 perform training. The trainer uses TP 2, PP 4, CP 4, and EP 8; rollout uses eight engines with DP attention. Training remains in BF16, whereas rollout uses FP8 weights and KV cache. Optimizer state is streamed to local disk because each rank holds approximately 279 GB after parallel partitioning.

The agent operates for up to 30 turns or one hour, with a maximum 65,536-token session and eight attempts per task. The training batch contains 64 trajectories, while as many as 128 trajectories are in flight. Fully asynchronous scheduling maintains approximately 90–100 concurrent requests, and affinity routing yields a 96% prefix-cache hit rate.

Figure 5: The GLM-5.2 reference run reports step time, rollout–trainer log-probability divergence, and raw task reward.

The run’s median step time over the first 30 measured steps is 263 seconds, excluding a 1,042-second warm-up step. The mean sampled-token log-probability divergence is 0.0369 over 100 steps and remains near its initial value. The nine-step moving average of raw task reward rises from 0.438 to 0.556. These measurements demonstrate that the specified configuration can execute a 744B-parameter agentic RL workload on a 64-GPU cluster, but they do not establish a general performance or learning advantage. The reward trend comes from one 100-step run on one task distribution, so the paper appropriately reports it as an observation rather than a statistically isolated improvement.

Limitations and open questions

The paper’s evidence is primarily systems evidence rather than a controlled algorithmic comparison. The central case study uses one model, one task distribution, one cluster configuration, and one short 100-step run. Consequently, the reported reward increase cannot be separated from run-to-run variation, and the interaction between asynchronous staleness, truncation, and agentic reward remains unquantified.

Coverage is also uneven. MXFP8 and NVFP4 are beta recipes tested on limited model families. P2P weight transfer requires supported Megatron–SGLang mappings. FSDP lacks several capabilities available in Megatron, including LoRA and disk-based offloading beyond host memory. The session server does not yet support image or video inputs, and relaxed replay matching can silently merge distinct tool histories. R3 introduces substantial trajectory-memory overhead and is not enabled in the principal case study.

Several questions remain open within the paper’s scope. It is not established how much each fidelity mechanism—TITO, R3, true-on-policy alignment, low-precision contracts, and importance-ratio correction—contributes independently to learning stability. Nor is the throughput cost of exact alignment characterized across larger model families. Finally, the paper does not provide broad ablations comparing synchronous and asynchronous schedules at equal hardware budgets or equal numbers of policy updates.

Conclusion

Miles v0.1 is a systems report centered on the operational and numerical requirements of frontier post-training. Its principal contributions are the integration of asynchronous agentic rollout, token-exact trajectory recording, MoE routing replay, shared precision contracts, memory-aware training, topology-specific weight synchronization, and explicit verification mechanisms. The system demonstrates substantial engineering results, including 85.5–86.4% reductions in multi-node weight-update time with P2P transfer, a 56% response-length reduction in on-policy distillation, and a 263-second median step time for a GLM-5.2 744B agentic RL run on 64 GB300 GPUs. These results establish a coherent production-oriented architecture, while the paper’s stated coverage limits and single-configuration measurements appropriately constrain the claims.