Negative Self-Distillation: Learning to Reason by Avoiding Flaws

Abstract: On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for LLM self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation (NSD), a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition (eg, acting as a ``careless reasoner'') and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines.

• The paper introduces Negative Self-Distillation (NSD), a method that improves reasoning by teaching models to avoid flawed reasoning behaviors, surpassing traditional self-distillation methods on benchmark tests.
• NSD uses self-generated negative conditioning to identify and suppress tokens sensitive to reasoning flaws, achieved through gated unlikelihood and KL regularization for optimization stability.
• NSD achieves significant accuracy gains across different model sizes, with average improvements of 2.3%, 7.5%, and 6.0% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B models, respectively, and notably increases reflection and exploratory behavior in mathematical reasoning tasks.

Problem setting and central thesis

“Negative Self-Distillation: Learning to Reason by Avoiding Flaws” (2609.11699) addresses a specific failure mode in label-dependent self-distillation for mathematical reasoning. On-policy self-distillation (OPSD) uses privileged information, typically a ground-truth solution, to produce dense token-level supervision for trajectories sampled from the student. The paper argues that this supervision can be structurally misaligned with the behavior required for difficult reasoning: because the teacher is conditioned on the answer, it generates unusually confident and linear traces, whereas successful problem solving often requires uncertainty, hypothesis testing, backtracking, and explicit self-correction.

The paper’s central claim is therefore contradictory to the usual premise of distillation: reasoning improvement can be obtained more reliably by suppressing model behaviors associated with flawed reasoning than by imitating an answer-conditioned teacher. The proposed method, Negative Self-Distillation (NSD), constructs a negative teacher from the same model using a question-specific negative condition. The student is then trained to diverge from the negative teacher only on tokens whose probabilities are selectively amplified by that negative condition.

The method is designed to eliminate both external teacher dependence and gold-answer supervision. Its training signal is generated from unlabeled problem statements, while the negative condition is intended to induce plausible but incorrect reasoning tendencies such as premature conclusions, invalid shortcuts, or failure to verify an intermediate result.

attachment:figure1

Figure 1: NSD constructs a negative teacher from the base model through self-generated negative conditioning and trains the student to diverge from the resulting distribution.

Negative self-distillation framework

NSD operates on an unlabeled dataset of problem statements. For each problem, the current student first produces an initial solution trace. The model is then prompted, conditioned on the problem and this trace, to generate an adaptive negative instruction. This instruction functions as a “careless reasoner” condition: it does not specify the correct answer, but instead encourages a reasoning pattern likely to produce a plausible error.

The resulting negative teacher shares the initial model parameters with a benign reference teacher. The two teachers differ only in their conditioning context:

• The reference teacher receives the original problem.
• The negative teacher receives the original problem together with the generated negative condition.
• The student generates the training trajectory under the original problem alone.

The negative teacher is not treated as globally incorrect. Its token distribution is used comparatively. For each student-generated token, NSD computes the positive probability gap between the negative and reference teachers:

$$G_t = \max\left(0, p_{\mathrm{neg},t} - p_{\mathrm{ref},t}\right).$$

A token is therefore activated only when the negative condition increases its probability. This distinction is important. Ordinary linguistic tokens may occur in both benign and flawed traces and should not be unlearned merely because the negative teacher assigns them high probability. The gate instead targets tokens that are specifically sensitive to the negative intervention.

attachment:figure2

Figure 2: NSD compares benign and negatively conditioned token distributions, applies a gate to negative-condition-sensitive tokens, and combines selective unlikelihood with KL regularization.

This design gives NSD a contrastive interpretation. The method does not attempt to identify all incorrect tokens in a trajectory, nor does it require an externally verified decomposition of the solution. It identifies tokens whose likelihood changes under a deliberately flawed context and uses that change as a proxy for reasoning relevance.

Gated unlikelihood and optimization stability

A direct unlikelihood loss is problematic in this setting. For a sampled token with student probability $p_t$, standard unlikelihood uses $-\log(1-p_t)$. Its logit gradient is proportional to $p_t$ after accounting for the softmax Jacobian. Thus, if the gate is activated for a highly predictable token, the update is largest precisely when the token is near-certain. Such tokens are often punctuation, whitespace, function words, or fixed syntactic structures rather than reasoning decisions.

NSD replaces standard unlikelihood with a bounded gated penalty:

$$\mathcal{L}_{\mathrm{GU}}^{(t)} = G_t \frac{1}{2-p_t}.$$

The corresponding logit gradient is proportional to

$G_t \frac{p_t(1-p_t)}{(2-p_t)^2},$

which converges to zero as $p_t$ approaches one. The penalty consequently emphasizes low- and mid-confidence tokens and attenuates updates on high-confidence structural tokens.

attachment:figure3

Figure 3: The sigmoid-bounded gated unlikelihood suppresses penalties and gradients on high-probability structural tokens while retaining stronger supervision for low- and mid-confidence tokens.

The paper’s analysis supports this mechanism empirically and analytically. It shows that standard unlikelihood can assign substantial gradients to high-probability tokens, whereas the proposed objective redistributes the update toward tokens more likely to encode ambiguous reasoning choices. This is not merely a numerical stabilization device: it determines which part of the autoregressive distribution the optimization is allowed to alter.

NSD adds a pointwise reference-weighted KL term:

$$\mathcal{L}_{\mathrm{KL}}^{(t)} = p_{\mathrm{ref},t} \log\frac{p_{\mathrm{ref},t}}{p_{\theta,t}}.$$

The complete token-level objective is

$$\mathcal{L}_{\mathrm{NSD}}^{(t)} = \mathcal{L}_{\mathrm{GU}}^{(t)} + \alpha \mathcal{L}_{\mathrm{KL}}^{(t)}.$$

This term anchors the student to the benign reference distribution without requiring a full-vocabulary KL computation. The paper reports that removing the KL term causes mid-training distributional collapse: the student drifts from the reference, the gate becomes spuriously activated, and the resulting updates enter a learning–forgetting cycle. The implication is that negative supervision is not self-regulating; without an explicit benign anchor, the student can reinterpret its own distributional drift as evidence of additional flaws.

Experimental design

The experiments use Qwen3 models with 1.7B, 4B, and 8B parameters. MATH is used for training after discarding its gold labels for NSD, Intuitor, and TTRL. Training lasts two epochs. Evaluation covers seven mathematical reasoning benchmarks: AIME 2024, AIME 2025, AIME 2026, HMMT 2025, AMC 2023, OlympiadBench, and MATH-500.

The principal comparisons are:

• OPSD, which uses gold-answer-conditioned self-distillation;
• Intuitor, which uses internal confidence as a reward;
• TTRL, which uses majority-vote consensus as pseudo-supervision;
• NSD, which uses negative conditioning, selective unlikelihood, and reference regularization.

The comparisons are not perfectly supervision-matched because OPSD requires gold labels whereas NSD does not. The authors explicitly retain OPSD as a reference point rather than treating it as an equivalent label-free baseline.

Main accuracy results

NSD produces the strongest average improvement at all three model sizes. Its average absolute gains over the corresponding base models are:

Model	NSD average gain	OPSD	Intuitor	TTRL
Qwen3-1.7B	+2.3%	+1.1%	−0.5%	+0.3%
Qwen3-4B	+7.5%	+1.0%	+1.3%	+0.2%
Qwen3-8B	+6.0%	+0.3%	+1.9%	−0.1%

The reported confidence intervals for NSD exclude zero at every scale, with one-sided $p$-values of $p_t$0, less than $p_t$1, and less than $p_t$2 for the 1.7B, 4B, and 8B models, respectively. The largest gains occur for the 4B model, where NSD improves the average score by 7.5 percentage points.

The per-benchmark results indicate that the improvement is not confined to a single dataset. For example, the Qwen3-4B NSD model reaches 35.8% on AIME 2024, 31.3% on AIME 2025, 29.2% on AIME 2026, 16.3% on HMMT 2025, and 76.3% on AMC 2023. The Qwen3-8B model reaches 39.6% on AIME 2024 and 26.3% on AIME 2025, both substantial improvements over the corresponding base model.

The scaling pattern is itself a claim of the paper: NSD becomes more effective as the model becomes better at generating useful negative conditions. This interpretation is plausible but not fully isolated experimentally. The results establish that the method scales favorably over the tested range; they do not establish that negative-condition quality is the sole cause of the scaling behavior.

The pass@8 results reinforce the distinction between single-sample accuracy and exploratory solution generation. NSD improves the average pass@8 score by 7.2, 8.3, and 9.7 points for the 1.7B, 4B, and 8B models, respectively. This is particularly relevant to the paper’s thesis because pass@8 measures whether at least one sampled trajectory succeeds, thereby rewarding the preservation of useful diversity rather than only the probability of one canonical reasoning path.

Reflection behavior and reasoning dynamics

The paper evaluates reflection using the frequency of predefined tokens and phrases such as “wait,” “actually,” “let me reconsider,” and “let me verify.” On Qwen3-4B, the baseline averages 3.6 reflection markers per response across AIME 2024, AIME 2025, and HMMT 2025. OPSD reduces this to 2.2, while Intuitor reduces it to 0.8. NSD increases it to 7.5.

Method	AIME 2024	AIME 2025	HMMT 2025	Average
Baseline	6.8	2.2	1.7	3.6
OPSD	2.6	2.1	1.8	2.2
Intuitor	0.6	1.0	0.7	0.8
NSD	6.9	7.5	8.1	7.5

The implication is that NSD changes the behavioral regime of the model rather than simply increasing confidence in an existing solution style. The paper interprets the increased reflection frequency as evidence that negative training preserves uncertainty and encourages re-evaluation. However, reflection-token frequency is an indirect behavioral measure. It does not by itself prove that the resulting reflection is causally useful or that every additional marker corresponds to a valid correction.

The case study provides a qualitative illustration. On an AIME geometry problem, the base model and OPSD model both fail across eight samples, while NSD obtains four correct samples. The NSD trace abandons several unsuccessful numerical guesses, introduces an aggregate variable, derives a quadratic constraint, and verifies the positive root. This example is consistent with the proposed mechanism: the model does not merely imitate a more direct derivation but changes its response after detecting that a local strategy is failing.

Token selectivity, efficiency, and conditioning variants

The token-analysis experiments measure the ratio between average weights assigned to style tokens and task tokens. Lower values indicate better suppression of style-token noise. NSD gating yields ratios of 2.6x, 3.4x, and 3.5x for the wiki-irrelevant, solution-aware, and question-only conditions, respectively. These are lower than the entropy-weighted OPSD ratio of 3.9x and the standard OPSD ratio of 5.4x.

This result supports the claim that comparing negative and benign contexts is more selective than weighting tokens using entropy or an undifferentiated distillation loss. The conclusion is narrower than “NSD identifies reasoning tokens”: the analysis uses manually specified style/task categories and measures relative weighting, so it demonstrates improved alignment with that operational definition rather than an intrinsic semantic identification capability.

NSD also reduces sampling requirements. It uses one student rollout per problem, compared with eight rollouts for the GRPO-style Intuitor and TTRL baselines. The method avoids full-vocabulary logit alignment and requires only scalar probabilities for the sampled token. The paper reports that the default online NSD procedure takes approximately 68 seconds per training step in the measured configuration, while the wiki-irrelevant variant reduces this to approximately 54 seconds. The authors attribute the efficiency to single-sample rollout, parallelized reference and negative-teacher prefilling, and scalar-only loss computation.

attachment:figure4

Figure 4: Question-only and wiki-irrelevant conditioning remain competitive with the default online negative-conditioning strategy.

The conditioning study is notable because question-only negative conditioning achieves a 7.3% average gain on Qwen3-4B, close to the 7.8% gain of the default online strategy on the reported four-dataset comparison. Even irrelevant Wikipedia passages remain competitive. By contrast, offline solution-aware conditioning performs worse, which the paper attributes to stale negative conditions generated from outdated model states.

This result weakens the necessity of sophisticated adaptive attack-prompt generation. It suggests that the useful signal may derive partly from distributional interference itself rather than only from semantically precise descriptions of a reasoning flaw. At the same time, the wiki-irrelevant condition is not a neutral control: irrelevant long-context material may degrade the negative teacher in systematic ways, and the paper does not fully characterize which properties of the injected noise are responsible for the observed effect.

Gradient behavior and objective ablations

The gradient analysis contrasts three objectives: standard unlikelihood, the proposed gated unlikelihood, and OPSD. Standard unlikelihood produces a gradient that increases with the probability of the target token. The sigmoid-bounded objective instead peaks at intermediate probability and vanishes as the probability approaches one. OPSD exhibits the opposite tendency from NSD in the paper’s analysis, assigning comparatively larger gradients to high-probability tokens.

The authors interpret this as a mechanism-level explanation for why OPSD can overlearn stylistic or structural features. NSD, by reducing updates to high-confidence tokens, focuses optimization on token decisions that remain uncertain under the student’s current distribution. The argument is technically coherent, although the connection between token probability and semantic role is statistical rather than guaranteed. High-probability tokens are often structural, but they can also be decisive mathematical symbols or answer-bearing expressions.

The KL ablation is stronger evidence for the necessity of the full objective. Without KL regularization, the student’s divergence from the reference increases sharply, the gate activation later collapses, and the gated loss becomes less informative. The result shows that the gate depends on a relatively stable student distribution: once the student changes too far, the original negative-versus-benign comparison no longer provides a reliable monitor.

The paper also tests a policy-gradient formulation in which the negative NSD loss is treated as a negative token-level advantage. This version improves the 1.7B model by 5.0 percentage points on average under the tested comparison, but the direct loss is better for the 4B and 8B models. Wiki-irrelevant conditioning performs particularly well in the policy-gradient regime. Thus, NSD is better understood as a family of negative-signal objectives than as a single optimization procedure, but the results do not establish a uniform advantage for policy-gradient training.

Limitations and open questions

NSD depends on the model’s ability to generate a negative condition that produces a meaningful contrast. The paper concedes that very weak models may not possess sufficient capability to construct useful failure modes. This is a substantive limitation because the negative teacher is not externally validated: an uninformative or nonsensical negative condition can produce distributional changes that are unrelated to reasoning quality.

The experiments are restricted to mathematical reasoning, primarily using Qwen3 models and MATH-derived training data. It remains open whether the same gating assumptions transfer to code generation, scientific reasoning, long-horizon planning, or open-ended tasks where correctness cannot be verified by a compact symbolic evaluator. The benchmark results also use best checkpoints selected on a validation set within two epochs, so the reported gains do not by themselves characterize sensitivity to training duration or checkpoint-selection procedures.

The reflection analysis relies on lexical markers. A model can produce frequent “wait” or “let me verify” phrases without performing valid verification, while a correct self-correction may occur without any of the monitored expressions. A stronger evaluation would separate linguistic markers from demonstrable correction of an incorrect intermediate state.

The negative-condition variants raise an unresolved mechanistic question. Since irrelevant Wikipedia noise performs competitively with targeted conditions, the current evidence does not determine whether NSD learns specific reasoning flaws or instead benefits from a broader context-induced perturbation. Relatedly, the gate uses the probability difference for the sampled token, not a full distributional comparison, so its relationship to global behavioral divergence is only partially characterized.

Finally, the comparison with OPSD is informative but not supervision-equivalent: OPSD receives gold solutions, whereas NSD is label-free. The paper’s results show that NSD can outperform a stronger-supervision baseline under the chosen implementations, but they do not establish that negative supervision is intrinsically superior under equal compute, equal prompt quality, and fully optimized implementations of each method.

Conclusion

NSD proposes a technically coherent alternative to answer-conditioned self-distillation. Its main contribution is the combination of self-generated negative conditioning, probability-difference gating, bounded unlikelihood, and KL anchoring. Across seven mathematical reasoning benchmarks and three model sizes, it reports average gains of 2.3%, 7.5%, and 6.0%, with especially strong pass@8 improvements and substantially increased measured reflection behavior.

The empirical results support the paper’s narrower conclusion: for the tested mathematical reasoning setting, selectively suppressing context-sensitive negative behaviors can improve accuracy while preserving exploratory and self-corrective generation more effectively than several positive or confidence-based self-training baselines. The principal open issue is whether the gains arise from learning identifiable reasoning flaws or from a more general regularized perturbation process induced by negative conditioning.