Anti-Self-Distillation: Methods & Implications

• Anti-Self-Distillation (AntiSD) is a design principle that inverts conventional distillation by promoting repulsion between models to discourage straightforward capability transfer.
• It encompasses diverse methods—such as output de-correlation in ensembles, token-level signal reversal in reasoning RL, and output poisoning via trace rewriting—to achieve resistance against imitation.
• Empirical results demonstrate that AntiSD can improve reproducibility, accelerate training efficiency, and safeguard proprietary outputs in tasks like MNIST classification, GSM8K reasoning, and ridge regression.

Anti-Self-Distillation (AntiSD) denotes a family of methods that invert the usual logic of distillation. Instead of making a student imitate a teacher, or making jointly trained models agree, AntiSD either pushes models away from one another, or modifies outputs and architectures so that they are less useful for capability transfer. In the literature summarized here, the term spans several technically distinct settings: output de-correlation in deep ensembles for reproducibility (Shamir et al., 2020), divergence ascent in on-policy reasoning RL (Shen et al., 12 May 2026), decoding-time poisoning of reasoning traces (Savani et al., 17 Apr 2025), output purification against logit-based knowledge distillation (Fang et al., 3 Feb 2026), trace rewriting and watermarking for API-served models (Ma et al., 16 Feb 2026), architecture-level distillation resistance via constraint-coupled reasoning (Wei et al., 26 Mar 2026), and an analytically distinct use in ridge regression where negative self-distillation weights become optimal in over-regularized regimes (Dang et al., 19 Feb 2026). This multiplicity of meanings is central to the topic: AntiSD is not a single algorithm, but a broader design principle for discouraging agreement, transfer, or compression when ordinary distillation would otherwise be beneficial.

1. Conceptual scope and relation to ordinary distillation

Ordinary knowledge distillation trains a student to match a teacher’s outputs, typically soft targets, and self-distillation extends that idea to cases where teacher and student are closely related or even architecturally identical. The common structural feature is an agreement objective: models are encouraged to converge toward a shared solution. AntiSD reverses that orientation. In the ensemble setting of "Anti-Distillation: Improving reproducibility of deep networks" (Shamir et al., 2020), multiple models are jointly regularized to avoid agreement by penalizing correlated outputs. In reasoning RL, "Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information" (Shen et al., 12 May 2026) replaces the usual attractive teacher signal with a repulsive one derived from privileged-context PMI. In distillation defense work, "Antidistillation Sampling" (Savani et al., 17 Apr 2025) explicitly makes a model’s own traces bad training data for a student, while preserving the model’s practical utility.

This suggests two broad interpretations. One is agreement inversion: AntiSD alters the optimization objective so that student and teacher, or ensemble members, are pushed apart rather than together. The other is transfer resistance: AntiSD modifies outputs or architectures so that successful imitation becomes less effective or more expensive. The first interpretation is most explicit in reproducibility and reasoning-RL work; the second dominates API protection and distillation-resistance research (Fang et al., 3 Feb 2026).

A common misconception is that AntiSD always means training against a teacher in the same way. The literature does not support that simplification. In some papers AntiSD acts on logits inside a jointly trained ensemble (Shamir et al., 2020); in others it acts on generated traces at decoding time (Savani et al., 17 Apr 2025); in others it is a post-hoc linear transformation on output logits (Fang et al., 3 Feb 2026); and in the ridge-regression setting it means allowing the self-distillation mixing weight $\xi$ to be negative, which moves the student away from the pure-distilled solution along an affine path (Dang et al., 19 Feb 2026). The unifying idea is opposition to straightforward capability transfer, not a single canonical training loop.

2. AntiSD as disagreement regularization in deep ensembles

The earliest formulation in this set of papers is Anti-Distillation for deep-network reproducibility (Shamir et al., 2020). Its motivating problem is prediction irreproducibility: two nominally identical models, trained on the same data with the same architecture and hyperparameters, may have essentially the same average accuracy while producing very different predictions on individual unseen examples. The paper formalizes this with prediction-difference metrics such as

$P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$

and

$$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$

For binary classification with $M=2$, the paper often uses

$$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$

The method trains an ensemble of $J$ components with ordinary supervised losses and an auxiliary regularizer that penalizes correlation or covariance among component outputs over a minibatch. For logits $z_{j,\tau}$, the correlation matrix is

$$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$

and the core Anti-Distillation loss is

$$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$

which equals $\frac{1}{2}\sum_{i\neq j} C_{z;i,j}^2$. Minimizing this drives off-diagonal correlations toward zero. The total loss is

$P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$0

Inference remains standard ensemble averaging; the AntiSD term acts only during training.

The significance of this formulation is that it defines AntiSD not as anti-learning, but as explicit diversity induction. The paper reports that applying Anti-Distillation on logits substantially reduces prediction differences while keeping accuracy essentially unchanged on MNIST and with controllable trade-offs on a large CTR task (Shamir et al., 2020). On MNIST with $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$1, no AD gives $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$2, $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$3, $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$4, and $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$5 at 99.2% accuracy, whereas correlation-based AD on logits with $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$6 yields $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$7, $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$8, $P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},$9, and $$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$0–$$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$1 with the same 99.2% accuracy (Shamir et al., 2020). In this usage, AntiSD is best understood as de-correlation-based ensemble training whose aim is improved reproducibility rather than model protection.

3. AntiSD in reasoning reinforcement learning

A distinct and later meaning appears in reasoning RL (Shen et al., 12 May 2026). Here the starting point is on-policy self-distillation with privileged context. A student policy $$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$2 generates rollouts, while a teacher policy $$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$3 is the same network conditioned on privileged context $$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$4, such as a verified solution or feedback. Standard self-distillation adds a per-token KL term

$$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$5

which is combined with GRPO using

$$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$6

The paper’s main diagnosis is PMI-based. Defining

$$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$7

it shows that under parameter sharing

$$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$8

Standard self-distillation uses $$\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.$$9. The paper argues that privileged context inflates the teacher’s confidence on tokens already implied by the solution and deflates it on deliberation tokens such as “Wait”, “Let”, and “Maybe”. As a result, default self-distillation rewards shortcut tokens and penalizes deliberation.

AntiSD reverses this polarity by ascending a Jensen–Shannon divergence rather than descending reverse KL. With

$M=2$0

the AntiSD token-level advantage is

$M=2$1

and the combined RL advantage becomes

$M=2$2

Because $M=2$3 is strictly increasing and sign-preserving, this reverses the sign of the teacher signal. Because $M=2$4, it also yields a naturally bounded positive advantage on large negative $M=2$5, avoiding the instability of reverse-KL ascent.

The method is stabilized with an entropy-triggered gate. Let

$M=2$6

Then a binary gate $M=2$7 updates according to a warmup entropy $M=2$8 and deactivation threshold $M=2$9, with $$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$0 (Shen et al., 12 May 2026). This turns the AntiSD term off once teacher entropy collapses.

Empirically, the paper reports that across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline’s accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points (Shen et al., 12 May 2026). Standard self-distillation almost always underperforms GRPO, whereas AntiSD improves over GRPO for every model tested. In this setting, AntiSD is not a defensive wrapper but a token-level reasoning signal that favors exploration and deliberation over privileged-context shortcuts.

4. AntiSD as API-side anti-distillation

For black-box LLMs, AntiSD often denotes methods that make outputs themselves poor supervision for a student. Three mechanisms dominate the papers in this set: antidistillation sampling, trace rewriting, and output purification.

"Antidistillation Sampling" (Savani et al., 17 Apr 2025) perturbs a teacher’s next-token distribution using a proxy student and a downstream loss $$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$1. If a proxy student would update on token $$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$2 via

$$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$3

the paper defines a downstream-loss change $$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$4 and then biases sampling toward tokens that increase student loss: $$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$5 Because direct evaluation of $$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$6 is intractable, the deployed rule uses a first-order finite-difference approximation

$$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$7

yielding the operational distribution

$$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$8

The paper reports that on GSM8K, tuning $$\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.$$9 allows teacher accuracy around 70% while students trained on these traces perform below their undistilled baseline, whereas naive temperature sampling must reduce teacher accuracy to about 20% to achieve comparable student degradation (Savani et al., 17 Apr 2025).

"Protecting LLMs Against Unauthorized Distillation through Trace Rewriting" (Ma et al., 16 Feb 2026) instead rewrites teacher-generated reasoning traces $J$0 into $J$1, creating a rewritten teacher

$J$2

Its explicit anti-distillation objective is

$J$3

The strongest method is a two-stage LLM-based rewriting pipeline: the teacher first produces clean traces, then a rewriter transforms them using optimized instructions. The best prompt found by OPRO asks the assistant to "reformulate it utilizing an esoteric, formal, and densely technical lexicon, thereby obfuscating its clarity for a non-specialist audience" (Ma et al., 16 Feb 2026). The paper reports that optimized rewriting reduces student accuracy to below 20% on GSM8K and MATH, with up to 61.3% absolute reduction compared to clean-trace distillation, while teacher accuracy improves by about 3% on GSM8K and about 22% on MATH (Ma et al., 16 Feb 2026). It also supports API watermarking via trigger–target snippets with high true detection and essentially no false alarms.

"Towards Distillation-Resistant LLMs: An Information-Theoretic Perspective" (Fang et al., 3 Feb 2026) addresses logit-based rather than text-based distillation. It defines distillation-relevant information as conditional mutual information

$J$4

where $J$5 is the input, $J$6 the ground-truth token, and $J$7 the teacher logits. Using

$J$8

the paper motivates compressing logits with respect to contextual information while preserving predictive information about the label. It introduces a low-rank logit transformation

$J$9

and trains $z_{j,\tau}$0 with

$z_{j,\tau}$1

where $z_{j,\tau}$2 preserves task accuracy and

$z_{j,\tau}$3

penalizes alignment between distillation gradients induced by original and transformed logits. The paper reports teacher drops of only about 1 point on GSM8K and MMLU for Qwen2.5-7B and Llama-3.1-8B, while distilled students can fall below their SFT-only baseline; for example, a Qwen2.5-1.5B student on GSM8K goes from 62.93 with vanilla KD to 50.95 with the defense, compared with 58.83 for SFT-only (Fang et al., 3 Feb 2026). In this branch of the literature, AntiSD is an operational mechanism for poisoning the knowledge channel itself.

5. Architecture-level resistance and evaluation methodology

A more abstract strand treats AntiSD as an architectural goal rather than a decoding or post-processing method. "A Public Theory of Distillation Resistance via Constraint-Coupled Reasoning Architectures" (Wei et al., 26 Mar 2026) formalizes capability as a property of hidden-state trajectories

$z_{j,\tau}$4

and introduces four elements: bounded transition burden,

$z_{j,\tau}$5

path-load accumulation,

$z_{j,\tau}$6

dynamically evolving feasible regions,

$z_{j,\tau}$7

and a capability–stability coupling condition

$z_{j,\tau}$8

with $z_{j,\tau}$9 as $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$0. In this account, a model is distillation-resistant when preserving capability within small tolerance requires preserving the internal stability profile as well. The paper is explicitly theoretical and omits proprietary implementation details, but it gives AntiSD a precise architectural interpretation: make useful behavior depend materially on internal constraints that are difficult to reconstruct from outputs alone (Wei et al., 26 Mar 2026).

This suggests an important distinction. Output-level AntiSD methods sabotage the teaching signal visible to an attacker; architecture-level AntiSD seeks to make high capability inseparable from hidden stability structure. A plausible implication is that these approaches are complementary rather than mutually exclusive: output perturbation can reduce the value of black-box extraction, while constraint coupling can raise the cost of reproducing frontier capability even when extraction succeeds partially.

Evaluation itself has become a research topic. "What Does It Mean to Break a Distillation Defense?" (Libon et al., 23 Jun 2026) argues that output-perturbation defenses lack a shared threat model and proposes the tuple

$$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$1

where $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$2 is query budget, $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$3 data budget, and $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$4 interface profile. The paper’s central modeling choice is that only teacher API queries and input prompts count as attack cost; all local computation, including student training and post-processing, is treated as free. Using antidistillation sampling as a case study, it shows that defense effectiveness depends sharply on the assumed threat model. Under stronger but realistic attackers—free local post-processing, $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$5, matched-utility budgets, or a one-token prefill channel—student performance can almost fully recover the undefended baseline across the high-utility regime (Libon et al., 23 Jun 2026). The main significance for AntiSD is methodological: claims about “undistillability” are conditional, and defense evaluation must specify attacker capabilities explicitly.

6. Negative mixing and the statistical interpretation of AntiSD

A final, analytically separate meaning of AntiSD appears in "Optimal Unconstrained Self-Distillation in Ridge Regression" (Dang et al., 19 Feb 2026). Here self-distillation retrains ridge regression on a mixture of ground-truth labels and teacher predictions with mixing parameter $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$6: $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$7 Because ridge is linear in the response, the self-distilled predictor lies on an affine path

$$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$8

between the teacher and the pure-distilled student. Optimizing conditional squared prediction risk over $$C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},$$9 yields

$$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$0

with the sign rule

$$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$1

This means that in under-regularized regimes, $$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$2, but in over-regularized regimes, $$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$3 (Dang et al., 19 Feb 2026). The paper explicitly interprets negative mixing as what the data block calls Anti-Self-Distillation: the student moves away from the teacher along the same affine path that ordinary self-distillation would follow. The result is unusually strong: conditioned on the training data and without distributional assumptions, the optimally mixed student strictly improves upon the ridge teacher for every $$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$4 at which $$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$5 (Dang et al., 19 Feb 2026). In isotropic proportional asymptotics, AntiSD occurs exactly when $$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$6, where $$L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],$$7 is the asymptotically optimal ridge penalty (Dang et al., 19 Feb 2026).

This usage differs from LLM-oriented AntiSD, but the conceptual overlap is real. In both cases, ordinary self-distillation is not always directionally correct. In ridge regression the teacher signal can be beneficial only when given a negative coefficient; in reasoning RL the teacher signal can be beneficial only after reversing its sign via divergence ascent (Shen et al., 12 May 2026). The common theme is not “distill more weakly,” but “the correct update may point away from the teacher.”

7. Synthesis, misconceptions, and open directions

Across these papers, AntiSD can be organized around three axes. First is where the opposition occurs: in parameter space or output space, as in ensemble de-correlation (Shamir et al., 2020); in token-level policy gradients, as in reasoning RL (Shen et al., 12 May 2026); in API outputs and logits, as in sampling, rewriting, and purification (Savani et al., 17 Apr 2025); or in hidden-state dynamics, as in constraint-coupled architectures (Wei et al., 26 Mar 2026). Second is what is being protected or improved: reproducibility, reasoning quality, API intellectual property, watermark detectability, or governance-preserving transfer resistance. Third is how the anti-distillation signal is formalized: correlation penalties, PMI sign reversal, gradient-aligned poisoning, CMI minimization, or negative mixing along a regularization path.

Several misconceptions are directly contradicted by the record. AntiSD is not synonymous with adding random noise: structured perturbations outperform generic noise in antidistillation sampling (Savani et al., 17 Apr 2025), and optimized trace rewriting outperforms gradient-free direct generation of “esoteric” traces (Ma et al., 16 Feb 2026). AntiSD is not inherently incompatible with utility preservation: multiple papers report small teacher degradation or even teacher improvement under carefully designed defenses (Ma et al., 16 Feb 2026). Nor is AntiSD equivalent to a universal “distillation defense”: whether a defense is effective depends on the query budget, data budget, and interface profile assumed for the attacker (Libon et al., 23 Jun 2026).

The open problems identified across the papers are also convergent. Architectural theory lacks frontier-scale validation (Wei et al., 26 Mar 2026). Output-level defenses remain vulnerable to adaptive attackers unless threat models are explicit and stress-tested (Libon et al., 23 Jun 2026). Information-theoretic purification has been demonstrated mainly on specific tasks and model families (Fang et al., 3 Feb 2026). Trace rewriting has been evaluated primarily for SFT-based trace distillation, leaving RL-based or preference-based distillation as open territory (Ma et al., 16 Feb 2026). Reasoning-RL AntiSD has strong results in math and preliminary gains in code, but broader domains remain untested (Shen et al., 12 May 2026). This suggests that AntiSD is emerging less as a settled method than as a research program: determine when agreement is harmful, when transfer should be opposed, and which parts of model behavior can be made intrinsically resistant to being copied without equal access to the underlying structure.