SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

Abstract: On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.

• The paper introduces SimpleOPD, which aligns teacher and student tokens through identical text spans, enabling on-policy distillation across heterogeneous tokenizers without learned vocabulary mappings.
• The method combines termination-token masking with student-reference KL regularization to prevent response-length growth, repetition, truncation, and unstable policy drift during long-context training.
• SimpleOPD improves reasoning performance across model families, including a 22.80-point ProofBench gain for Intern-S2-Preview and a 21.2-point gain under Gemini-2.5-Pro judging, while results vary by tokenizer, data mix, and benchmark.

On-policy distillation (OPD) is particularly well suited to reasoning transfer because supervision is evaluated on trajectories sampled from the student rather than on fixed teacher demonstrations. This paper studies a difficult variant of that setting: transferring long-context mathematical proof reasoning from SU-01, a 30B-A3B model capable of producing reasoning traces exceeding 100K tokens, into shorter-context students with heterogeneous architectures and tokenizers. The central claim is that tokenizer incompatibility need not prevent token-level OPD, provided that teacher supervision is constructed in a shared text space and combined with explicit controls on policy drift and termination behavior (2608.14277).

Problem setting and contributions

The paper focuses on the mismatch between a long-context teacher and short-context students. This mismatch is not merely a difference in maximum sequence length. The teacher and student may use distinct chat templates, token vocabularies, token boundaries, and model distributions. Directly comparing their token-level probabilities is therefore ill-defined when a single student token corresponds to multiple teacher tokens, or vice versa. Existing OPD formulations have predominantly assumed shared vocabularies or closely related model families, leaving heterogeneous cross-tokenizer distillation comparatively underexplored (2608.14277).

The authors identify a second failure mode that arises even when token-level supervision is available. Direct OPD tends to increase the student’s response length because the teacher assigns relatively high probability to extended reasoning trajectories. The student consequently becomes less likely to emit structural termination tokens, such as </think> and <|im_end|>, leading to repetition, truncation, and unstable optimization. The paper’s proposed method, SimpleOPD, addresses these problems with three design choices:

1. Text-span token alignment: teacher and student tokens are aligned only when they cover exactly the same textual span at the same position in the response.
2. Termination-token advantage masking: OPD supervision is removed from structural termination tokens.
3. Student-reference KL regularization: the distilled policy is constrained relative to the student’s initial policy to limit excessive distributional drift.

The method is intentionally conservative. It does not construct a learned mapping between incompatible token vocabularies, and it does not require supervised fine-tuning on teacher-generated trajectories. Its objective is to retain only reliable local correspondences while allowing the student to remain responsible for generating its own rollouts.

SimpleOPD and cross-tokenizer alignment

Let the student generate a response using its own tokenizer and chat template. The resulting token sequence is decoded into a surface string. The teacher then receives the original input formatted with its own chat template, followed by that same response string, and tokenizes the response using its native tokenizer. Thus, the teacher scores the exact text produced by the student, but under the teacher’s own conditional distribution.

The alignment procedure operates on tokenizer-native text spans. A student token and a teacher token are matched only if they have consumed identical response prefixes and contribute identical current text spans. Partial overlaps are discarded. For example, if one tokenizer represents a substring as a single token while the other splits it into several pieces, those positions do not receive teacher supervision. A linear two-pointer scan enumerates the matches, making the alignment computationally simple.

For unmatched student positions, the method falls back to the student’s own log-probability. This effectively prevents unmatched positions from introducing an artificial teacher target. The authors report the lexical-overlap ratio, defined as the proportion of student response tokens receiving aligned teacher supervision, as an empirical measure of how much signal survives cross-tokenizer conversion.

The resulting loss is a partial reverse-KL surrogate. In the shared-tokenizer case, every token is aligned and the objective reduces to ordinary reverse-KL OPD. In the heterogeneous case, it is not an exact sequence-level KL because different tokenizations induce different factorizations of the same string. The method instead optimizes teacher-student agreement on the subset of positions for which a one-to-one textual correspondence is unambiguous.

Figure 1: SimpleOPD evaluates the student-generated response with the teacher’s tokenizer and transfers supervision only across identical text spans, with KL regularization and termination-token masking for stability.

This construction has an important implication: SimpleOPD does not require complete vocabulary overlap, but its effective supervision density depends on the tokenizers and on the student’s generated text. The paper’s lexical-overlap experiments show that the aligned-token ratio is already high at the beginning of training and increases during training for Qwen3.5-35B-A3B, Intern-S2-Preview, and GLM-4.7-Flash. This suggests that the partial objective retains substantial supervision in practice, although the paper does not establish that the overlap ratio is sufficient for arbitrary tokenizer pairs.

Figure 2: Lexical-overlap ratios remain substantial and generally increase during cross-tokenizer OPD training.

The per-token visualization further illustrates the mechanism: aligned tokens receive teacher-derived reverse-KL supervision, whereas unmatched positions remain unsupervised by the teacher. This is preferable to forcing an arbitrary probability correspondence, but it also means that the method cannot directly transfer information expressed exclusively through tokenizer-specific segmentations.

Figure 3: Cross-tokenizer reverse-KL supervision is available for aligned spans, while unmatched teacher or student pieces are explicitly excluded.

Length explosion and stabilization

The instability analysis is one of the paper’s strongest contributions because it demonstrates that naive OPD can produce misleading apparent gains. During direct OPD from SU-01 to Intern-S2-Preview, response length, truncation, and repetition all increase as training progresses. Qwen3.5-35B-A3B exhibits an even more severe degradation: performance deteriorates while truncation remains high and response length grows rapidly.

Figure 4: Direct OPD on Intern-S2-Preview produces increasing response length, repetition, and truncation despite initial task improvements.

Figure 5: The same direct-OPD pathology is more pronounced for Qwen3.5-35B-A3B.

The associated case studies show that the failure is not limited to low-quality answers. In one AIME25 rollout, the student reaches the correct answer, $279$, and then repeats an identical ten-sentence self-check block 972 times until the 160K-token limit is reached. In another AnswerBench rollout, generation collapses into a one-token loop involving repeated emissions of $2^?$. These examples establish that correctness at an intermediate point does not imply usable generation under the learned policy: the termination mechanism itself can degenerate.

The authors first mask the OPD loss on </think> and <|im_end|>. The rationale is that these tokens control response structure and termination rather than mathematical content. Masking improves stability and evaluation results, but does not eliminate length expansion; truncation still rises during later training.

Figure 6: Termination-token masking reduces early length instability but is insufficient as a standalone correction.

The decisive intervention is a KL penalty between the evolving student and its initial reference policy. With coefficient $0.5$ in the Intern-S2-Preview experiment, truncation is reduced to nearly zero, and performance improves consistently. ProofBench@4 increases from 21.70 for the base model to 38.50 with reference KL, while AnswerBench@8 rises to 79.10 and AIME25@8 to 95.80. The implication is direct: constraining policy drift preserves the student’s ability to terminate and prevents the teacher’s long-context behavior from overwhelming the student’s shorter-context operating regime.

Figure 7: Student-reference KL regularization stabilizes training and reduces truncation to approximately zero.

The coefficient ablation on GLM-4.7-Flash supports a non-monotonic trade-off. A coefficient of $1.0$ gives the best combined result, improving ProofBench@4 from 30.75 to 39.71 and AnswerBench@8 from 69.59 to 71.97. A coefficient of $0.5$ provides insufficient stabilization, whereas $1.2$ constrains the student more strongly and may limit teacher-capability acquisition. The appropriate coefficient therefore depends on the teacher-student gap rather than being a universal constant.

Main mathematical reasoning results

SimpleOPD is evaluated on ProofBench, AnswerBench, AIME25, and AMOBench. The training data contain 4,528 proof-oriented problems: 63 from the Open Proof Corpus, 2,948 from AoPS, 900 from competition-training books, and 617 from Shuzhimi and related olympiad materials. The use of exclusively proof-oriented data allows the experiments to isolate the transfer of natural-language proof reasoning from SU-01.

The method yields consistent improvements across same-tokenizer and cross-tokenizer students:

Student	ProofBench@4	AnswerBench@8	AIME25@8	AMOBench@8
Qwen3-4B	11.42	47.50	71.25	23.00
Qwen3-4B-OPD	23.72	64.50	90.83	35.00
Qwen3-30B-A3B	13.80	59.13	88.33	36.50
Qwen3-30B-A3B-OPD	36.47	74.46	93.75	52.75
Qwen3.5-35B-A3B	26.78	73.16	94.60	57.25
Qwen3.5-35B-A3B-OPD	42.39	80.15	96.66	61.25
Intern-S2-Preview	21.70	76.03	88.33	58.00
Intern-S2-OPD	44.50	80.10	95.00	59.50

The same-tokenizer results establish that the stabilization recipe is effective even without the additional difficulty of vocabulary mismatch. Qwen3-4B gains 12.30 points on ProofBench, 17.00 on AnswerBench, and 19.58 on AIME25. Qwen3-30B-A3B gains 22.67 points on ProofBench, reaching 36.47. These improvements indicate that OPD transfers more than answer-level accuracy: its largest effects occur on proof-oriented evaluation, where the form and organization of reasoning matter.

The cross-tokenizer results are more consequential for the paper’s thesis. Qwen3.5-35B-A3B improves by 15.61 points on ProofBench, while Intern-S2-Preview improves by 22.80 points, from 21.70 to 44.50. The latter nearly matches SU-01’s reported 45.00 under the corresponding evaluation configuration. Intern-S2-OPD also exceeds SU-01 on AnswerBench and AIME25, reaching 80.10 and 95.00, respectively. These results support the claim that exact tokenizer identity is unnecessary for meaningful OPD transfer.

A second ProofBench evaluation uses Gemini-2.5-Pro as judge under the same setting used for SU-01. Here, Intern-S2-OPD improves from 34.0 to 55.2, a gain of 21.2 percentage points, and is reported to surpass Gemini-2.5-Pro and GPT-5 while remaining below SU-01 and DeepSeek-V3.2-Speciale. The result is numerically strong, but it depends on model-based judging; consequently, the comparison is sensitive to judge choice and evaluation protocol.

Figure 8: Under Gemini-2.5-Pro judging, Intern-S2-OPD reaches 55.2 on ProofBench, a 21.2-point gain over Intern-S2-Preview.

The cross-family experiments extend the result beyond the Qwen and Intern families. GLM-4.7-Flash improves from 30.8 to 39.7 on ProofBench and from 69.6 to 72.0 on AnswerBench. Gemma-4-26B-A4B improves from 25.5 to 34.2 on ProofBench, although AnswerBench declines from 68.8 to 67.5. The contrast is informative: ProofBench gains persist under substantial architectural and tokenizer differences, but transfer is not uniformly beneficial across benchmarks. GLM uses a different BPE vocabulary from SU-01, whereas Gemma uses SentencePiece, and the latter exhibits the less balanced outcome.

Baselines, domain transfer, and sequence length

Against EOPD and G-OPD, SimpleOPD obtains the strongest overall Intern-S2 results. It reaches 44.50 on ProofBench, 80.10 on AnswerBench, 95.00 on AIME25, and 59.50 on AMOBench. It outperforms both comparison methods on ProofBench, AnswerBench, and AMOBench, while EOPD is marginally better on AIME25 by 0.33 points. The advantage is concentrated on natural-language proof reasoning rather than uniformly across all mathematical benchmarks.

Figure 9: SimpleOPD compares favorably with EOPD and G-OPD, with its largest advantage on ProofBench.

The paper also reports out-of-domain scientific transfer despite using only mathematical proof data for distillation. Intern-S2-OPD improves over Intern-S2-Preview on FrontierScience-Olympiad from 60.6 to 60.9, FrontierScience Research from 1.7 to 5.0, HLE from 19.6 to 20.5, and HiPhO from 38.6 to 41.1. The 2.5-point HiPhO gain is especially notable, and the distilled student exceeds SU-01’s 35.0 on that benchmark. These results imply that the student’s pre-existing domain-specific capabilities can be retained while acquiring a more effective general reasoning procedure. They do not, however, demonstrate transfer to arbitrary domains, since the reported gains are limited and benchmark-specific.

Data composition matters. Proof-only distillation produces the best ProofBench result, 44.50, whereas adding verifiable mathematical data lowers ProofBench to 38.50 while increasing AnswerBench only from 80.10 to 81.10 and leaving AIME25 unchanged at 95.00. The result supports a relatively strong claim: data that is useful for verifiable mathematical correctness can weaken transfer of natural-language proof ability when mixed into the same OPD process.

Distillation length also affects performance. Increasing the maximum length from 6K to 32K raises Intern-S2-OPD’s ProofBench score from 40.07 to 44.50 and AnswerBench from 77.97 to 80.10. For Qwen3.5-35B-A3B, the corresponding increases are 38.80 to 42.39 on ProofBench and 77.25 to 80.16 on AnswerBench. The effect is largest on ProofBench, indicating that long traces contain supervision relevant to multi-step proof construction. This benefit is conditional on stable training: without termination masking and reference KL, longer rollouts amplify the failure modes documented earlier.

A stronger teacher, DeepSeek-V4-Flash, distilled into Intern-S2-Preview at only 6K, yields ProofBench@4 of 39.71, AnswerBench@8 of 77.94, and AIME25@8 of 97.50. The ProofBench score exceeds the 38.80 obtained from SU-01 at the same length, supporting the paper’s claim that teacher strength affects transfer quality even when the student and teacher belong to different model families.

Limitations and open questions

The proposed cross-tokenizer objective is a partial surrogate rather than an exact distributional comparison. Unaligned spans receive no teacher supervision, and the paper does not report a systematic relationship between lexical-overlap ratio and final task performance. It also does not compare SimpleOPD against stronger learned cross-tokenizer mappings under identical training budgets, so the relative value of simple exact-span matching remains incompletely characterized.

The experiments are concentrated on one primary teacher, SU-01, and one dominant training domain, mathematical proofs. The DeepSeek-V4-Flash experiment broadens the teacher set, but the evidence remains insufficient to determine how the method behaves with teachers whose reasoning format, tokenizer, chat template, or termination protocol differs substantially from those studied. The Gemma results also show that cross-family transfer can improve one benchmark while degrading another.

Evaluation introduces additional uncertainty. ProofBench is non-verifiable and relies on LLM judges, with results varying between DeepSeek-V4-Flash and Gemini-2.5-Pro configurations. The paper mitigates this by repeated rollouts and averaging, but it does not provide human-validation analyses of the strongest cross-model comparisons. Finally, the reference-KL coefficient is tuned separately for model groups—$0.5$ for several students and $1.0$ for GLM and Gemma—leaving open how to select it automatically from measurable teacher-student distribution mismatch.

Conclusion

SimpleOPD provides a compact OPD recipe for transferring long-context reasoning into shorter-context, heterogeneous students. Its technical contribution is the use of exact text-span alignment to obtain tokenizer-agnostic supervision without imposing an artificial vocabulary mapping. Its practical contribution is the identification of termination-token masking and student-reference KL regularization as necessary controls against length inflation, repetition, and truncation.

Across multiple model families, the method produces substantial gains, including a 22.80-point ProofBench improvement for Intern-S2-Preview under the main evaluation and a 21.2-point gain under Gemini-2.5-Pro judging. The results support cross-tokenizer OPD as a viable mechanism for capability transfer, while also showing that stability, evaluation protocol, tokenizer similarity, data composition, and distillation length materially determine the outcome.