---
title: 'Anti-Self-Distillation: Methods & Implications'
url: https://www.emergentmind.com/topics/anti-self-distillation-antisd
type: topic
---

# Anti-Self-Distillation: Methods & Implications

Anti-Self-Distillation (AntiSD) denotes a family of methods that invert the usual logic of distillation. Instead of making a student imitate a teacher, or making jointly trained models agree, AntiSD either pushes models away from one another, or modifies outputs and architectures so that they are less useful for capability transfer. In the literature summarized here, the term spans several technically distinct settings: output de-correlation in deep ensembles for reproducibility [2010.09923], divergence ascent in on-policy reasoning RL [2605.11609], decoding-time poisoning of reasoning traces [2504.13146], output purification against logit-based knowledge distillation [2602.03396], trace rewriting and watermarking for API-served models [2602.15143], architecture-level distillation resistance via constraint-coupled reasoning [2603.25022], and an analytically distinct use in ridge regression where negative self-distillation weights become optimal in over-regularized regimes [2602.17565]. This multiplicity of meanings is central to the topic: AntiSD is not a single algorithm, but a broader design principle for discouraging agreement, transfer, or compression when ordinary distillation would otherwise be beneficial.

## 1. Conceptual scope and relation to ordinary distillation

Ordinary knowledge distillation trains a student to match a teacher’s outputs, typically soft targets, and self-distillation extends that idea to cases where teacher and student are closely related or even architecturally identical. The common structural feature is an agreement objective: models are encouraged to converge toward a shared solution. AntiSD reverses that orientation. In the ensemble setting of "Anti-Distillation: Improving reproducibility of deep networks" [2010.09923], multiple models are jointly regularized to avoid agreement by penalizing correlated outputs. In reasoning RL, "Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information" [2605.11609] replaces the usual attractive teacher signal with a repulsive one derived from privileged-context PMI. In distillation defense work, "Antidistillation Sampling" [2504.13146] explicitly makes a model’s own traces bad training data for a student, while preserving the model’s practical utility.

This suggests two broad interpretations. One is **agreement inversion**: AntiSD alters the optimization objective so that student and teacher, or ensemble members, are pushed apart rather than together. The other is **transfer resistance**: AntiSD modifies outputs or architectures so that successful imitation becomes less effective or more expensive. The first interpretation is most explicit in reproducibility and reasoning-RL work; the second dominates API protection and distillation-resistance research [2602.03396].

A common misconception is that AntiSD always means training against a teacher in the same way. The literature does not support that simplification. In some papers AntiSD acts on logits inside a jointly trained ensemble [2010.09923]; in others it acts on generated traces at decoding time [2504.13146]; in others it is a post-hoc linear transformation on output logits [2602.03396]; and in the ridge-regression setting it means allowing the self-distillation mixing weight $\xi$ to be negative, which moves the student away from the pure-distilled solution along an affine path [2602.17565]. The unifying idea is opposition to straightforward capability transfer, not a single canonical training loop.

## 2. AntiSD as disagreement regularization in deep ensembles

The earliest formulation in this set of papers is Anti-Distillation for deep-network reproducibility [2010.09923]. Its motivating problem is *prediction irreproducibility*: two nominally identical models, trained on the same data with the same architecture and hyperparameters, may have essentially the same average accuracy while producing very different predictions on individual unseen examples. The paper formalizes this with prediction-difference metrics such as
\[
P_n = \frac{1}{M}\sum_{m=1}^M P_{n,m},
\]
and
\[
\Delta_p = \frac{1}{N}\sum_{n=1}^N \frac{1}{M}\sum_{m=1}^M \|P_{n,m} - P_n\|_p.
\]
For binary classification with $M=2$, the paper often uses
\[
\Delta_1 = \frac{1}{N}\sum_{n=1}^N |P_{n,1}(1) - P_{n,2}(1)|.
\]

The method trains an ensemble of $J$ components with ordinary supervised losses and an auxiliary regularizer that penalizes correlation or covariance among component outputs over a minibatch. For logits $z_{j,\tau}$, the correlation matrix is
\[
C_{z; i,j} = \frac{1}{k} \sum_{\tau=1}^k z_{i,\tau} z_{j,\tau},
\]
and the core Anti-Distillation loss is
\[
L_{\text{AD}(C_z)} = \frac{1}{2}\left[\|C_z\|_F^2 - \|\mathrm{diag}(C_z)\|_F^2\right],
\]
which equals $\frac{1}{2}\sum_{i\neq j} C_{z;i,j}^2$. Minimizing this drives off-diagonal correlations toward zero. The total loss is
\[
L = L_{\text{label}} + \lambda L_{\text{AD}}.
\]
Inference remains standard ensemble averaging; the AntiSD term acts only during training.

The significance of this formulation is that it defines AntiSD not as anti-learning, but as explicit diversity induction. The paper reports that applying Anti-Distillation on logits substantially reduces prediction differences while keeping accuracy essentially unchanged on MNIST and with controllable trade-offs on a large CTR task [2010.09923]. On MNIST with $J=2$, no AD gives $\Delta_1=0.0777$, $\Delta_2=0.0305$, $\Delta_1^L=1.83\%$, and $\Delta^H=0.64\%$ at 99.2% accuracy, whereas correlation-based AD on logits with $\lambda\in[0.02,0.5]$ yields $\Delta_1\approx0.0036$, $\Delta_2\approx0.0024$, $\Delta_1^L\approx0.51\%$, and $\Delta^H\approx0.35$–$0.38\%$ with the same 99.2% accuracy [2010.09923]. In this usage, AntiSD is best understood as de-correlation-based ensemble training whose aim is improved reproducibility rather than model protection.

## 3. AntiSD in reasoning reinforcement learning

A distinct and later meaning appears in reasoning RL [2605.11609]. Here the starting point is on-policy self-distillation with privileged context. A student policy $\pi_S(\cdot\mid x,y_{<t})$ generates rollouts, while a teacher policy $\pi_T(\cdot\mid x,c,y_{<t})$ is the same network conditioned on privileged context $c$, such as a verified solution or feedback. Standard self-distillation adds a per-token KL term
\[
\mathcal{L}_{\mathrm{SD}}(\theta) = \mathbb{E}_{x,\, y \sim \pi_S(\cdot \mid x)} \Big[ \sum_{t=1}^{T} D_{\mathrm{KL}}\big( \pi_S(\cdot \mid x, y_{<t}) \,\|\, sg[\pi_T(\cdot \mid x, y_{<t})] \big) \Big],
\]
which is combined with GRPO using
\[
A_{i,t} = A_i^{\mathrm{seq}} + \lambda \cdot \delta_t.
\]

The paper’s main diagnosis is PMI-based. Defining
\[
u_t := \log \pi_T(y_t \mid x, y_{<t}) - \log \pi_S(y_t \mid x, y_{<t}),
\]
it shows that under parameter sharing
\[
u_t = \log \frac{\pi_\theta(y_t \mid x, c, y_{<t})}{\pi_\theta(y_t \mid x, y_{<t})}
= \mathrm{PMI}(y_t; c \mid x, y_{<t}).
\]
Standard self-distillation uses $\delta_t=+u_t$. The paper argues that privileged context inflates the teacher’s confidence on tokens already implied by the solution and deflates it on deliberation tokens such as “Wait”, “Let”, and “Maybe”. As a result, default self-distillation rewards shortcut tokens and penalizes deliberation.

AntiSD reverses this polarity by ascending a Jensen–Shannon divergence rather than descending reverse KL. With
\[
\varphi(u) := \tfrac{1}{2}\big(\mathrm{softplus}(u)-\log 2\big),
\]
the AntiSD token-level advantage is
\[
A_t^{\mathrm{AntiSD}} = -\varphi(u_t),
\]
and the combined RL advantage becomes
\[
A_{i,t} = A_i^{\mathrm{seq}} - \lambda \,\varphi(u_{i,t}).
\]
Because $\varphi$ is strictly increasing and sign-preserving, this reverses the sign of the teacher signal. Because $\varphi(u)\ge -\tfrac{1}{2}\log 2$, it also yields a naturally bounded positive advantage on large negative $u$, avoiding the instability of reverse-KL ascent.

The method is stabilized with an entropy-triggered gate. Let
\[
H := \mathrm{median}_{i,t} H[\pi_T(\cdot \mid x_i, y_{i,<t})].
\]
Then a binary gate $g$ updates according to a warmup entropy $H_{\mathrm{warm}}$ and deactivation threshold $\tau_{\mathrm{down}} = 0.93\,H_{\mathrm{warm}}$, with $\lambda = g\cdot \lambda_{\max}$ [2605.11609]. This turns the AntiSD term off once teacher entropy collapses.

Empirically, the paper reports that across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline’s accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points [2605.11609]. Standard self-distillation almost always underperforms GRPO, whereas AntiSD improves over GRPO for every model tested. In this setting, AntiSD is not a defensive wrapper but a token-level reasoning signal that favors exploration and deliberation over privileged-context shortcuts.

## 4. AntiSD as API-side anti-distillation

For black-box LLMs, AntiSD often denotes methods that make outputs themselves poor supervision for a student. Three mechanisms dominate the papers in this set: antidistillation sampling, trace rewriting, and output purification.

"Antidistillation Sampling" [2504.13146] perturbs a teacher’s next-token distribution using a proxy student and a downstream loss $\ell(\theta_P)$. If a proxy student would update on token $x_{t+1}$ via
\[
\theta_P^+ = \theta_P + \eta \nabla_{\theta_P} \log p(x_{t+1}\mid x_{1:t};\theta_P),
\]
the paper defines a downstream-loss change $\Delta(x_{t+1}\mid x_{1:t})$ and then biases sampling toward tokens that increase student loss:
\[
x_{t+1} \sim \frac{1}{Z}\exp\left(\frac{1}{\tau}\log p(\cdot\mid x_{1:t};\theta_T) + \lambda\,\Delta(\cdot\mid x_{1:t})\right).
\]
Because direct evaluation of $\Delta$ is intractable, the deployed rule uses a first-order finite-difference approximation
\[
\widehat{\Delta}(\cdot \mid x_{1:t}) =
\frac{\log p(\cdot \mid x_{1:t}; \theta_P + \epsilon \nabla \ell(\theta_P)) - \log p(\cdot \mid x_{1:t}; \theta_P)}{\epsilon},
\]
yielding the operational distribution
\[
x_{t+1} \sim \frac{1}{Z}\exp\left(\frac{1}{\tau}\log p(\cdot \mid x_{1:t}; \theta_T) + \lambda \,\widehat{\Delta}(\cdot \mid x_{1:t}) \right).
\]
The paper reports that on GSM8K, tuning $\lambda$ allows teacher accuracy around 70% while students trained on these traces perform below their undistilled baseline, whereas naive temperature sampling must reduce teacher accuracy to about 20% to achieve comparable student degradation [2504.13146].

"Protecting Language Models Against Unauthorized Distillation through Trace Rewriting" [2602.15143] instead rewrites teacher-generated reasoning traces $r_i=\mathcal{T}(q_i)$ into $r_i'=\mathcal{R}(q_i,r_i)$, creating a rewritten teacher
\[
\mathcal{T}_{\mathcal{R}}(q)\equiv \mathcal{R}(q,\mathcal{T}(q)).
\]
Its explicit anti-distillation objective is
\[
\text{Acc}(\mathcal{S}_{\text{clean}}, \mathcal{D}) - \text{Acc}(\mathcal{S}_{\mathcal{R}}, \mathcal{D}) > \delta,
\qquad
\text{Acc}(\mathcal{T},\mathcal{D}) - \text{Acc}(\mathcal{T}_{\mathcal{R}},\mathcal{D}) \le \epsilon.
\]
The strongest method is a two-stage LLM-based rewriting pipeline: the teacher first produces clean traces, then a rewriter transforms them using optimized instructions. The best prompt found by OPRO asks the assistant to "reformulate it utilizing an esoteric, formal, and densely technical lexicon, thereby obfuscating its clarity for a non-specialist audience" [2602.15143]. The paper reports that optimized rewriting reduces student accuracy to below 20% on GSM8K and MATH, with up to 61.3% absolute reduction compared to clean-trace distillation, while teacher accuracy improves by about 3% on GSM8K and about 22% on MATH [2602.15143]. It also supports API watermarking via trigger–target snippets with high true detection and essentially no false alarms.

"Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective" [2602.03396] addresses logit-based rather than text-based distillation. It defines distillation-relevant information as conditional mutual information
\[
I(X;Z\mid Y),
\]
where $X$ is the input, $Y$ the ground-truth token, and $Z$ the teacher logits. Using
\[
I(X;Z\mid Y) = I(X;Z)-I(Z;Y),
\]
the paper motivates compressing logits with respect to contextual information while preserving predictive information about the label. It introduces a low-rank logit transformation
\[
Z' = MZ,\qquad M = E + AB,
\]
and trains $M$ with
\[
L_M = L_{\mathrm{CE}} + \lambda L_{\mathrm{grad}},
\]
where $L_{\mathrm{CE}}$ preserves task accuracy and
\[
L_{\mathrm{grad}} = \frac{\langle g, g' \rangle}{\|g\|_F\|g'\|_F}
\]
penalizes alignment between distillation gradients induced by original and transformed logits. The paper reports teacher drops of only about 1 point on GSM8K and MMLU for Qwen2.5-7B and Llama-3.1-8B, while distilled students can fall below their SFT-only baseline; for example, a Qwen2.5-1.5B student on GSM8K goes from 62.93 with vanilla KD to 50.95 with the defense, compared with 58.83 for SFT-only [2602.03396]. In this branch of the literature, AntiSD is an operational mechanism for poisoning the knowledge channel itself.

## 5. Architecture-level resistance and evaluation methodology

A more abstract strand treats AntiSD as an architectural goal rather than a decoding or post-processing method. "A Public Theory of Distillation Resistance via Constraint-Coupled Reasoning Architectures" [2603.25022] formalizes capability as a property of hidden-state trajectories
\[
h_{t+1} = F_\theta(h_t, x_t),
\]
and introduces four elements: bounded transition burden,
\[
b_t = \Psi(h_t, h_{t+1}, x_t),\qquad b_t \le B_t,
\]
path-load accumulation,
\[
L_t = \sum_{\tau=1}^{t}\alpha_\tau b_\tau,
\]
dynamically evolving feasible regions,
\[
\Omega_t = \Omega_0 \setminus \Gamma(L_t),
\qquad h_{t+1}\in \Omega_t,
\]
and a capability–stability coupling condition
\[
\Delta(\mathcal{K}(S), \mathcal{K}(T)) \le \varepsilon
\Longrightarrow
\Delta(\mathcal{R}(S), \mathcal{R}(T)) \le g(\varepsilon),
\]
with $g(\varepsilon)\to 0$ as $\varepsilon\to 0$. In this account, a model is distillation-resistant when preserving capability within small tolerance requires preserving the internal stability profile as well. The paper is explicitly theoretical and omits proprietary implementation details, but it gives AntiSD a precise architectural interpretation: make useful behavior depend materially on internal constraints that are difficult to reconstruct from outputs alone [2603.25022].

This suggests an important distinction. Output-level AntiSD methods sabotage the teaching signal visible to an attacker; architecture-level AntiSD seeks to make high capability inseparable from hidden stability structure. A plausible implication is that these approaches are complementary rather than mutually exclusive: output perturbation can reduce the value of black-box extraction, while constraint coupling can raise the cost of reproducing frontier capability even when extraction succeeds partially.

Evaluation itself has become a research topic. "What Does It Mean to Break a Distillation Defense?" [2606.25059] argues that output-perturbation defenses lack a shared threat model and proposes the tuple
\[
(Q, D, P),
\]
where $Q$ is query budget, $D$ data budget, and $P$ interface profile. The paper’s central modeling choice is that only teacher API queries and input prompts count as attack cost; all local computation, including student training and post-processing, is treated as free. Using antidistillation sampling as a case study, it shows that defense effectiveness depends sharply on the assumed threat model. Under stronger but realistic attackers—free local post-processing, $Q>D$, matched-utility budgets, or a one-token prefill channel—student performance can almost fully recover the undefended baseline across the high-utility regime [2606.25059]. The main significance for AntiSD is methodological: claims about “undistillability” are conditional, and defense evaluation must specify attacker capabilities explicitly.

## 6. Negative mixing and the statistical interpretation of AntiSD

A final, analytically separate meaning of AntiSD appears in "Optimal Unconstrained Self-Distillation in Ridge Regression" [2602.17565]. Here self-distillation retrains ridge regression on a mixture of ground-truth labels and teacher predictions with mixing parameter $\xi\in\mathbb{R}$:
\[
\hat\beta_{\mathrm{sd},\lambda}(\xi)
=
\arg\min_\beta
\Big\{
(1-\xi)\frac1n\|y-X\beta\|_2^2
+
\xi\frac1n\|\hat y_\lambda-X\beta\|_2^2
+
\lambda\|\beta\|_2^2
\Big\}.
\]
Because ridge is linear in the response, the self-distilled predictor lies on an affine path
\[
f_{\mathrm{sd},\lambda,\xi}(x)
=
(1-\xi)f_\lambda(x)+\xi f_{\mathrm{pd},\lambda}(x),
\]
between the teacher and the pure-distilled student. Optimizing conditional squared prediction risk over $\xi$ yields
\[
\xi^\*(\lambda)
=
-\frac{\lambda}{2}\frac{R'(\lambda)}{D(\lambda)},
\qquad
R_{\mathrm{sd}}^\*(\lambda)
=
R(\lambda)-\frac{\lambda^2}{4}\frac{(R'(\lambda))^2}{D(\lambda)},
\]
with the sign rule
\[
\operatorname{sign}(\xi^\*(\lambda))=-\operatorname{sign}(R'(\lambda)).
\]

This means that in under-regularized regimes, $\xi^\*(\lambda)>0$, but in over-regularized regimes, $\xi^\*(\lambda)<0$ [2602.17565]. The paper explicitly interprets negative mixing as what the data block calls Anti-Self-Distillation: the student moves away from the teacher along the same affine path that ordinary self-distillation would follow. The result is unusually strong: conditioned on the training data and without distributional assumptions, the optimally mixed student strictly improves upon the ridge teacher for every $\lambda>0$ at which $R'(\lambda)\neq 0$ [2602.17565]. In isotropic proportional asymptotics, AntiSD occurs exactly when $\lambda>\lambda^\*$, where $\lambda^\*=\gamma\sigma^2/r^2$ is the asymptotically optimal ridge penalty [2602.17565].

This usage differs from LLM-oriented AntiSD, but the conceptual overlap is real. In both cases, ordinary self-distillation is not always directionally correct. In ridge regression the teacher signal can be beneficial only when given a negative coefficient; in reasoning RL the teacher signal can be beneficial only after reversing its sign via divergence ascent [2605.11609]. The common theme is not “distill more weakly,” but “the correct update may point away from the teacher.”

## 7. Synthesis, misconceptions, and open directions

Across these papers, AntiSD can be organized around three axes. First is **where the opposition occurs**: in parameter space or output space, as in ensemble de-correlation [2010.09923]; in token-level policy gradients, as in reasoning RL [2605.11609]; in API outputs and logits, as in sampling, rewriting, and purification [2504.13146]; or in hidden-state dynamics, as in constraint-coupled architectures [2603.25022]. Second is **what is being protected or improved**: reproducibility, reasoning quality, API intellectual property, watermark detectability, or governance-preserving transfer resistance. Third is **how the anti-distillation signal is formalized**: correlation penalties, PMI sign reversal, gradient-aligned poisoning, CMI minimization, or negative mixing along a regularization path.

Several misconceptions are directly contradicted by the record. AntiSD is not synonymous with adding random noise: structured perturbations outperform generic noise in antidistillation sampling [2504.13146], and optimized trace rewriting outperforms gradient-free direct generation of “esoteric” traces [2602.15143]. AntiSD is not inherently incompatible with utility preservation: multiple papers report small teacher degradation or even teacher improvement under carefully designed defenses [2602.15143]. Nor is AntiSD equivalent to a universal “distillation defense”: whether a defense is effective depends on the query budget, data budget, and interface profile assumed for the attacker [2606.25059].

The open problems identified across the papers are also convergent. Architectural theory lacks frontier-scale validation [2603.25022]. Output-level defenses remain vulnerable to adaptive attackers unless threat models are explicit and stress-tested [2606.25059]. Information-theoretic purification has been demonstrated mainly on specific tasks and model families [2602.03396]. Trace rewriting has been evaluated primarily for SFT-based trace distillation, leaving RL-based or preference-based distillation as open territory [2602.15143]. Reasoning-RL AntiSD has strong results in math and preliminary gains in code, but broader domains remain untested [2605.11609]. This suggests that AntiSD is emerging less as a settled method than as a research program: determine when agreement is harmful, when transfer should be opposed, and which parts of model behavior can be made intrinsically resistant to being copied without equal access to the underlying structure.

Source: https://www.emergentmind.com/topics/anti-self-distillation-antisd