Rufus-Air: An Open LLM Post-Training Recipe

Abstract: Rufus-Air is an open and reproducible post-training recipe on GLM-4.5-Air-Base (106B-A12B), organized as a serial pipeline of eight stages: SFT, Reasoning RL, Coding RL, Instruction-Following RL, General Agent, Coding Agent, Search Agent, and RLHF. We document the data, reward design, infrastructure, stage order, and stagewise results needed to reproduce the recipe. Stages progress from basic to advanced capabilities and from hard, verifiable rewards to softer judge-based signals. Training builds on open-source components and public data, much of it used as released, without new human annotation or an in-house distillation teacher. Our main findings are that (i) diverse, high-quality SFT establishes a strong capability floor; (ii) difficulty filtering keeps RL prompts within a productive learning range; (iii) reward reliability provides a practical principle for ordering stages; and (iv) infrastructure and engineering choices are part of the recipe, not just an implementation detail. Rufus-Air improves over the official GLM-4.5-Air post-trained release and is competitive with similarly sized open models.

• The paper presents an eight-stage post-training recipe for a 106B-parameter, 12B-active Mixture-of-Experts (MoE) model, improving instruction following, tool use, and browsing via supervised fine-tuning (SFT), verifiable-reward RL, agentic RL, and RLHF.
• The model’s supervised fine-tuning stage, consisting of 9.01 million examples and 27.04 billion loss-contributing tokens, establishes a strong baseline that improves instruction following by 24.15 points on IFBench, and tool use tasks by 6.63 points on AIME 2025.
• The instruction-following RL (IF RL) stage, incorporated into the training pipeline, enhanced the model's performance on Multi-challenge by 24.7 points and on AdvancedIF by 17.4 points, indicating significant improvements in preserving constraints and formatting quality

Rufus-Air presents a reproducible post-training recipe for a 106B-parameter, 12B-active Mixture-of-Experts model initialized from GLM-4.5-Air-Base. Its central contribution is not a new policy-gradient objective or a single specialized dataset, but an unusually complete account of how supervised fine-tuning, verifiable-reward RL, agentic RL, RLHF, and infrastructure interact in a serial production pipeline (2609.29421). The resulting checkpoint is evaluated against same-base and similarly sized open-weight models, with the strongest improvements concentrated in instruction following, tool use, browsing, and software-engineering tasks.

Recipe design and central claims

The pipeline consists of eight sequential stages:

$$\text{SFT} \rightarrow \text{Reasoning RL} \rightarrow \text{Coding RL} \rightarrow \text{IF RL} \rightarrow \text{General Agent} \rightarrow \text{Coding Agent} \rightarrow \text{Search Agent} \rightarrow \text{RLHF}.$$

Figure 1: Rufus-Air’s serial post-training pipeline from GLM-4.5-Air-Base to the final Rufus-Air checkpoint.

The ordering is motivated by two considerations. Capability development proceeds from broad model competence to increasingly specialized forms of interaction. Reward reliability proceeds in the opposite direction: deterministic verifiers are used before rule-based and LLM-based judges, and the most gameable learned reward—the RLHF reward model—is applied last. The paper is explicit that this is a practical ordering principle rather than a demonstrated optimum. Later stages update the same parameters and therefore can alter earlier capabilities; the reported stagewise comparisons are intended to establish local effects rather than prove that the global order is uniquely correct.

The paper advances four main claims. First, SFT establishes a substantive capability floor rather than merely initializing RL. Second, difficulty filtering creates an automatic curriculum by retaining prompts that are neither already solved nor entirely outside the current policy’s competence. Third, reward reliability should influence stage order because optimization pressure is more dangerous when the reward can be gamed. Fourth, rollout fidelity, router alignment, sandbox reliability, and environment orchestration are part of the algorithmic recipe.

Supervised fine-tuning as the capability floor

The SFT stage uses 9.01 million examples, 44.5 billion raw tokens, and 27.04 billion loss-contributing assistant tokens. The data combine general chat, STEM, mathematics, code, general-agent, and coding-agent trajectories. The distinction between sample share and token share is consequential: General Agent contributes 39.6% of examples but only 14.2% of training tokens, whereas Math and Coding Agent together constitute 18.8% of examples but 49.1% of loss-contributing tokens.

Figure 2: SFT composition by capability category, showing the divergence between sample share and training-token share.

The mixture contains 17 public datasets and no Rufus-Air-generated response regeneration or newly commissioned human annotation. Many released datasets nevertheless contain upstream model-generated trajectories, including outputs from DeepSeek-R1, QwQ-32B, GPT-OSS-120B, Qwen3, and other models. The paper carefully distinguishes these upstream dependencies from in-house distillation.

Preprocessing normalizes heterogeneous sources into a role-aware, multi-turn schema with explicit loss masks. System messages, user turns, and tool observations are retained as context but excluded from the loss. Interleaved reasoning and tool-use trajectories are segmented so that completed thought–action–observation chains form coherent supervision units. The production chat template is applied before length filtering, with a 120K-token sample cap and a 128K-token packed training context.

Decontamination combines sequential 8-gram overlap screening, exhaustive phrase matching for mathematical benchmarks, dense retrieval, and manual review. The procedure removes 4,187 SFT instances, including 68 confirmed contaminated benchmark–training pairs and 58 unique samples from the additional mathematics and knowledge checks. The authors acknowledge that n-gram and embedding filters cannot eliminate all semantic leakage, but the procedure materially improves the credibility of the reported evaluations.

The production SFT run uses three epochs on 512 H200 GPUs. The learning rate warms up to $5 \times 10^{-5}$ and decays toward $5 \times 10^{-6}$. Training loss continues to decrease across epochs, but held-out benchmark scores plateau during the first epoch. The pipeline therefore selects checkpoint 3799 rather than the final low-loss checkpoint.

Figure 3: SFT loss continues declining after held-out capability scores have largely plateaued, motivating selection of an intermediate checkpoint.

The SFT checkpoint already surpasses the public GLM-4.5-Air post-trained release on IFEval by 5.33 points, IFBench by 24.15 points, AIME 2025 by 6.63 points, and AIME 2026 by 3.50 points. It trails on GPQA by 5.72 points. This result supports the paper’s strongest SFT claim: a carefully composed supervised mixture can produce a broadly capable policy before RL, while also revealing domain-specific deficits for subsequent optimization.

Benchmark	GLM-4.5-Air	Rufus-Air SFT	Change
IFEval	83.00	88.33	+5.33
IFBench	33.60	57.75	+24.15
GPQA	73.90	68.18	-5.72
AIME 2025	84.20	90.83	+6.63
AIME 2026	86.50	90.00	+3.50

Verifiable-reward RL and learnability filtering

Reasoning RL targets mathematics, scientific question answering, and algorithmic puzzles. Its prompt set contains 121,161 tasks, with mathematics accounting for 47.7% of prompts, science for 35.7%, and puzzles for 16.7%. Puzzles contribute 55.5% of prompt tokens because their serialized states and rule specifications are substantially longer.

The reward functions are deterministic: Math-Verify for canonical mathematical answers, generated Python checkers for puzzles, and fuzzy string matching for science answers. A teacher-solvability filter removes prompts that GPT-OSS-120B cannot solve, while a learnability filter excludes prompts solved by the current policy more than 80% of the time or never solved in sampled rollouts. During training, dynamic sampling retains groups whose mean reward lies in $(0, 0.8]$. This creates an online difficulty frontier: as the policy improves, previously unlearnable prompts can enter the productive band.

The stage uses GSPO with sequence-level clipping, truncated importance sampling, large rollout groups, and a length penalty relative to the shortest correct sample. Relative to SFT, GPQA rises from 68.18 to 73.50, a gain of 5.32 points. AIME 2025 and AIME 2026 decline by 2.81 and 2.60 points respectively. The paper treats these changes as acceptable guardrail variation because each AIME evaluation contains only 30 problems and the observed differences are smaller than one problem’s effective contribution under the stated evaluation procedure. The result supports a narrower claim: the verifier-based stage improved the targeted scientific-knowledge metric, but it did not uniformly improve all reasoning benchmarks.

Coding RL then trains against executable Python tests. The 29,405-problem mixture combines functional and stdin/stdout tasks, with EvolveCoder supplying 50.1% of the problems. Teacher solvability filtering and a warm-up drop-all-pass screen remove problems that provide little group-relative signal. Test subsampling is deterministic, preventing reward drift across rollouts.

Figure 4: Coding RL reward and LiveCodeBench performance improve after the response budget is extended from 64K to 128K tokens.

The response budget initially produces truncation rates between 11% and 28%. After extending the limit from 64K to 128K tokens, truncation falls below 0.1%, and the raw reward resumes increasing. LiveCodeBench v6 pass@1 rises from 68.6 to a peak of 75.9, a 7.3-point improvement, while pass@8 increases by only 2.3 points. This divergence suggests that the stage primarily improves first-sample reliability rather than merely increasing the probability that at least one of many samples succeeds. It also demonstrates that rollout truncation is not merely an infrastructure statistic: because truncated samples are loss-masked, it directly controls the amount of useful gradient entering the update.

Instruction-following RL

Instruction-following RL jointly optimizes two datasets: 14,000 single-turn prompts with programmatically verifiable constraints and approximately 13,000 adversarial multi-turn conversations. The constraints cover formatting, length, structural requirements, lexical conditions, instruction hierarchy, and persistence of user requirements across turns.

Each instance receives a binary reward only when all required rubrics are satisfied. The paper makes an important distinction between necessary and optional rubrics. Optional desiderata are excluded because they add reward noise and encourage the model to produce unnecessarily long responses. When the authors experimentally attach many optional rubrics, the policy develops a pronounced length bias; when rubrics express only necessary conditions, final responses become shorter while task performance improves.

The stage produces some of the largest targeted gains in the pipeline:

Benchmark	Coding RL	+ IF RL	Change
IFEval	90.5	94.5	+4.0
IFBench	63.8	77.8	+14.0
Multi-challenge	31.1	55.8	+24.7
AdvancedIF	45.5	62.9	+17.4
GPQA	67.9	71.1	+3.2
AIME 2025	87.6	86.4	-1.2

The 24.7-point improvement on Multi-challenge is particularly strong because it targets multi-turn instruction retention rather than isolated formatting. The concurrent increase on GPQA and near-preservation of AIME indicate that the stage did not impose a substantial reasoning trade-off under these measurements. The paper attributes the effectiveness of reasoning-first ordering partly to prerequisite structure: complex instruction following requires reasoning over constraints, and the Reasoning RL checkpoint supplies that behavior before IF optimization.

Figure 5: Instruction-following RL reward rises while final response length decreases under necessary-condition rubrics.

Agentic reinforcement learning

The three agent stages use distinct environments but a common infrastructure abstraction. General Agent training uses approximately 2,000 filtered tasks from AgentWorldModel, with MCP tools operating over stateful synthetic databases. The terminal and search stages use execution-based and web-based environments respectively.

General Agent training improves MCP-Atlas by 7.80 points and Tau2-Retail by 9.80 points relative to the IF RL checkpoint. AIME 2026, IFEval, GPQA, and LiveCodeBench remain stable or improve modestly.

Figure 6: General Agent training increases held-out tool-use performance while trajectories become longer.

The result is consistent with transfer from synthetic MCP environments to external tool-use benchmarks, although the evaluation cannot isolate which environment properties produce that transfer. The authors interpret the result as evidence that stateful tool orchestration—planning, tool-call formation, error recovery, and state tracking—can be learned from synthetic environments.

Coding Agent training operates in Harbor-format terminal sandboxes using a single shell-execution tool. Rewards are binary and based on task-specific tests. Relative to General Agent, Terminal-Bench 2.1 increases from 38.76 to 40.17 and SWE-bench Verified from 65.60 to 67.80. These are modest gains, and the paper explicitly states that the stage ended because of compute constraints rather than convergence.

The later shipped checkpoint reports Terminal-Bench 2.1 at 42.7 but SWE-bench Verified at 65.6, equal to the Coding Agent stage’s starting value. Because the intervening Search Agent and RLHF stages were not individually evaluated on these benchmarks, the final differences cannot be attributed to a particular later stage.

Search Agent training uses approximately 2,200 questions selected from MiroVerse. The filtering strategy is unusually aggressive: questions answerable without tools are removed, and questions with zero or eight successes among eight initial tool-enabled rollouts are also excluded. The final set therefore concentrates on tasks that are difficult but demonstrably solvable.

The reward is a continuous LLM-judge score, gated on at least two successful tool calls and reduced for malformed calls. The authors remove a format bonus after observing that the policy exploited it through tool-free guesses. A mean-only group advantage is used instead of standard-deviation normalization. In an ablation, mean-only advantages produce held-out scores from 55.8 to 64.8, compared with 49.7 to 58.7 for normalized advantages. The authors argue that standard deviation normalization removes the magnitude of partial-credit differences and amplifies grading noise, particularly in groups containing a single apparent success.

Figure 7: Search Agent reward and held-out success increase as the policy learns more effective tool-use patterns.

Search Agent improves BrowseComp by 3.0 points, Seal-0 by 5.4 points, and HLE-Verified by 3.4 points. The improvement transfers to an evaluation environment with a different search backend. However, tool-use analysis shows that additional depth is not uniformly beneficial. On BrowseComp, trajectories exceeding roughly 50 iterations are predominantly failures, and budget exhaustion or context overflow is almost never associated with correctness. The policy’s late-stage increase in long successful searches therefore reflects selective deepening rather than an unconditional preference for longer trajectories.

RLHF and final benchmark position

RLHF is applied last using Skywork-Reward-V2-Qwen3-8B on filtered HH-RLHF prompts. Training uses only prompts; chosen and rejected responses are used for dataset analysis but not directly optimized. A linear length penalty is added to the reward-model score, and GRPO is used for on-policy updates.

The stage raises Arena-Hard v2 Hard Prompt from 83.06 to 89.05, a 5.99-point gain, and Creative Writing from 38.56 to 52.97, a 14.41-point gain.

Figure 8: RLHF improves held-out reward-model scores and Arena-Hard performance under a linear response-length penalty.

The Creative Writing improvement is substantial, but the final Rufus-Air score of 53.0 remains below GLM-4.5-Air’s 60.3 and Nemotron-3-Super’s 61.5. The paper therefore makes a deliberately limited claim: the selected reward model improves open-ended alignment metrics, but it does not establish that the resulting reward is well matched to stylistic quality. On other monitored metrics, stage-specific evaluations show changes of roughly one point or less, although the paper cautions that not all benchmark protocols are identical across the stagewise and headline evaluations.

Across the final comparison, Rufus-Air leads the official GLM-4.5-Air release on every reported benchmark except Arena-Hard v2 Creative Writing. Its headline results include:

Benchmark	Rufus-Air
IFEval	95.4
IFBench	76.9
Multi-challenge	65.8
GPQA	75.6
AIME 2025	88.3
AIME 2026	86.7
LiveCodeBench v6	76.4
MCP-Atlas	45.3
BrowseComp	37.1
Seal-0	51.4
HLE-Verified	51.1
Terminal-Bench 2.1	42.7
SWE-bench Verified	65.6
Arena-Hard v2 HP	89.1
Arena-Hard v2 CW	53.0

Against Nemotron-3-Super, Rufus-Air is strongest on instruction following, search, MCP-Atlas, LiveCodeBench, and SWE-bench Verified; it is close on mathematics, science, and Terminal-Bench; and it trails on Creative Writing, Tau2-Airline, and slightly on Tau2-Retail. The paper appropriately qualifies cross-model comparisons because several external numbers use different judges, tools, user simulators, or inference-time context-management strategies.

Infrastructure as part of the method

Rufus-Air uses Rufus-Slime, built on Slime, Megatron-LM, and SGLang, together with Rufus-Gym for agentic environments. The infrastructure decisions are tightly coupled to the RL objectives.

The most important mechanism for the MoE model is Rollout Routing Replay, which reuses rollout-time expert assignments during the training forward pass. This reduces the routing discrepancy between FP8 inference and BF16 optimization. Rollouts use FP8-quantized SGLang engines, while parameter updates remain in BF16.

The agent environments preserve exact token trajectories through token-in/token-out execution. Rather than re-rendering and re-tokenizing the entire dialogue after every tool call, the environment appends sampled token IDs and associated log probabilities, masks, and expert assignments directly. This avoids retokenization drift, repeated chat-template application, and training on tool-call text that was repaired after sampling.

Coding trajectories run in Firecracker microVMs backed by a self-hosted E2B runtime. Sandboxes retain process and filesystem state across the full episode, are initialized from snapshots, and are destroyed after termination. Ray distributes CPU-heavy environment actors, tool execution, and reward computation across nodes. These details are not peripheral: the Coding Agent and Search Agent stages depend on long-horizon interactions whose throughput and fidelity determine whether the policy receives enough usable gradient.

Limitations and open questions

The evidence does not establish that the eight-stage order is optimal. The authors state that it reflects prior experiments, engineering constraints, and the desire to delay gameable rewards; they do not provide a factorial ordering study or a systematic comparison with parallel specialist training and checkpoint merging.

Stagewise attribution is also incomplete. No single benchmark suite is measured after every stage, and several values use different evaluation protocols. In particular, the Coding Agent stage is compute-limited rather than converged, and final changes in Terminal-Bench, SWE-bench, and Seal-0 cannot be assigned uniquely to later stages. The paper also reports discrepant intermediate values for some benchmarks, including GPQA and IFEval, arising from different evaluation runs.

The headline comparisons contain externally reported baselines obtained under heterogeneous harnesses. This is especially important for Tau2-Bench, where changing the simulated user model shifts results by as much as 23.7 points. Search evaluations also depend strongly on context management: disabling the scratchpad reduces BrowseComp performance by 10.7 points. These findings mean that the reported model rankings are informative only when benchmark versions, judges, simulators, tool backends, and context policies are aligned.

Finally, the search and RLHF rewards remain imperfect. Search training uses an LLM judge whose repeated scores can disagree, and the requirement for two successful tool calls does not rule out memorized answers with incidental tool use. RLHF optimizes a learned reward model that may not fully capture creative-writing quality. The open question is therefore not whether these stages improve their immediate metrics—they do—but how robustly those improvements track independently validated task quality under altered judges and environments.

Conclusion

Rufus-Air provides a detailed empirical account of post-training a large MoE checkpoint through SFT, verifiable-reward RL, instruction-following RL, agentic RL, and RLHF. Its strongest technical conclusions concern data and systems design: broad SFT establishes a strong initial policy; learnability filtering concentrates RL on informative prompts; deterministic rewards should precede softer objectives; and exact token fidelity, router alignment, sandbox execution, and distributed environment orchestration are necessary components of long-horizon RL.

The final model substantially improves instruction following and agentic capabilities relative to the official same-base release, including gains of 24.7 points on Multi-challenge, 9.8 points on Tau2-Retail, 5.4 points on Seal-0, 7.3 points on LiveCodeBench pass@1, and 14.4 points on Arena-Hard Creative Writing relative to the immediate RLHF initialization. The paper’s principal value is the reproducible integration of these choices, while its own results leave the optimal stage order, long-run convergence of agentic RL, and robustness of judge-based rewards as unresolved empirical questions (2609.29421).