Antidistillation Sampling for Robust LLM Outputs

• Antidistillation sampling is a defense mechanism that modifies a model’s next-token distribution during decoding to preserve output utility while hindering effective distillation of reasoning traces.
• It leverages proxy student gradients to perturb teacher logits, carefully balancing output quality with reduced information for training downstream student models.
• Framed as a game-theoretic defense, this approach offers a trade-off between maintaining teacher accuracy and significantly degrading student performance, as demonstrated on benchmarks like GSM8K and MATH.

Antidistillation sampling is a family of inference-time defenses that modify a model’s released outputs so that they remain useful to end users while becoming less effective as training data for downstream distillation. In the reasoning-LLM literature, the term most commonly denotes a decoding-time perturbation of the teacher’s next-token distribution, guided by a proxy student and a downstream loss, so that released chain-of-thought traces actively degrade or fail to improve student learning (Savani et al., 17 Apr 2025). Subsequent work reinterprets the same gradient-aligned principle for fingerprint planting and detection (Xu et al., 3 Feb 2026), embeds it in a KL-constrained teacher–student minimax game (Allouah et al., 21 May 2026), and contrasts it with post-generation alternatives such as trace reformulation and sparse sentence deletion (Ding et al., 13 Oct 2025, Hartman et al., 25 Apr 2026). Separately, the same phrase has also been used in diffusion-model distillation for an adaptive timestep scheduler; that usage is conceptually distinct from the LLM defense literature (Wang et al., 16 Sep 2025).

1. Origins, threat model, and design objective

The motivating threat model is behavioral distillation from released reasoning traces. A model provider exposes answers together with intermediate reasoning, tool-use sequences, or other rich outputs; a third party then aggregates those outputs and fine-tunes a smaller student by standard next-token prediction. In this setting, final answers alone are a weaker attack surface, whereas token-level traces provide a substantially richer learning signal (Savani et al., 17 Apr 2025).

The central design objective is explicitly two-sided: preserve teacher utility while reducing distillability. In the original formulation, antidistillation sampling is a decoding-time defense for reasoning LLMs that “strategically modifies the next-token probability distribution during generation so that the traces are still highly likely under the original teacher model but actively harm a downstream student’s training dynamics” (Savani et al., 17 Apr 2025). Later formulations sharpen the same idea into a utility-constrained game in which the teacher suppresses outputs that are especially valuable for student learning, while the student may adapt by reweighting examples rather than training passively on all released data (Allouah et al., 21 May 2026).

This objective differs from provenance-oriented watermarking. Watermarking primarily aims at post-hoc detection of generated text, whereas antidistillation aims to reduce the usefulness of the outputs themselves as supervised fine-tuning data. The distinction is important because it leads to different perturbation targets: watermarking emphasizes detectability, while antidistillation emphasizes the student’s expected parameter update under fine-tuning (Savani et al., 17 Apr 2025, Xu et al., 3 Feb 2026).

2. Core decoding-time formulation for reasoning LLMs

In the original LLM formulation, the teacher model with parameters $\theta_T$ defines a next-token distribution $p_{\theta_T}(\cdot \mid x_{1:t})$. Standard temperature sampling draws from

$$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$

Antidistillation sampling instead introduces a proxy-student-derived token penalty and samples from a transformed distribution

$$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$

where $\phi = (\lambda,\epsilon,\tau)$ and $\hat{\Delta}$ is constructed from a proxy student model $\theta_P$ (Savani et al., 17 Apr 2025).

The ideal perturbation is defined through the change in a downstream proxy loss after one student update on a candidate token $x$:

$$\theta_P^{+} = \theta_P + \eta \nabla_{\theta_P} \log p_{\theta_P}(x \mid x_{1:t}),$$

$$\Delta(x \mid x_{1:t}) = \ell(\theta_P^{+}) - \ell(\theta_P).$$

In the small-step limit, the directional effect becomes

$p_{\theta_T}(\cdot \mid x_{1:t})$0

This shows that the perturbation targets the alignment between the token-induced student update and the gradient of a downstream loss: tokens whose supervised update would move the student toward higher downstream loss receive positive logit shifts, and tokens that would help the student are relatively suppressed (Savani et al., 17 Apr 2025).

Because evaluating $p_{\theta_T}(\cdot \mid x_{1:t})$1 for all vocabulary items is intractable, the method uses a finite-difference approximation:

$p_{\theta_T}(\cdot \mid x_{1:t})$2

Operationally, this yields a simple decoding hook: precompute $p_{\theta_T}(\cdot \mid x_{1:t})$3 once per decoding session, run two proxy forwards per step to obtain $p_{\theta_T}(\cdot \mid x_{1:t})$4, add $p_{\theta_T}(\cdot \mid x_{1:t})$5 to the teacher logits, and sample. No backpropagation is required during decoding, but the cost per token is approximately one teacher forward pass plus two proxy forward passes (Savani et al., 17 Apr 2025).

Utility preservation is controlled primarily by $p_{\theta_T}(\cdot \mid x_{1:t})$6. The same framework also allows an optional KL-bounded deployment view,

$p_{\theta_T}(\cdot \mid x_{1:t})$7

which can be enforced by shrinking $p_{\theta_T}(\cdot \mid x_{1:t})$8 per step. This suggests a continuum from nearly unmodified teacher decoding to aggressively poisoned traces (Savani et al., 17 Apr 2025).

3. Gradient alignment, student internalization, and fingerprinting

A later development repurposes the same gradient-based principle for fingerprinting rather than direct poisoning. In antidistillation fingerprinting, the objective is not to make student fine-tuning fail, but to bias teacher outputs toward tokens that a student will efficiently internalize so that the resulting student becomes statistically detectable after fine-tuning (Xu et al., 3 Feb 2026).

The setup introduces a teacher $p_{\theta_T}(\cdot \mid x_{1:t})$9, student $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$0, proxy $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$1, and a red-and-green-list hash $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$2 with window size $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$3 and green-list ratio $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$4. The fingerprint detectability statistic is the Average Green-list Token Probability (GTP). For open-weight students,

$$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$5

and for closed-weight students it becomes the empirical green-token fraction under one sample per context. Under the null hypothesis that student generations are independent of the secret key $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$6, a conservative p-value is

$$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$7

where $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$8 is the observed GTP (Xu et al., 3 Feb 2026).

The key advance is that antidistillation sampling is rederived from first-order student learning dynamics. With proxy logits $$x_{t+1} \sim q_{\mathrm{temp}}(\cdot \mid x_{1:t}) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau\}.$$9, distribution $$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$0, green list $$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$1, and per-step green-list probability

$$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$2

the token-level score is derived from the gradient inner product

$$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$3

Using an isotropic approximation $$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$4 in logit space, this simplifies to the ADS score

$$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$5

This score boosts high-probability green tokens and suppresses high-probability red tokens. Under the special case where the proxy has only a trainable final linear adaptation layer, the isotropic assumption is exact, with

$$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$6

The resulting sampling rule is

$$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$7

so the teacher distribution is biased by a proxy-derived estimate of which tokens most increase future detectability after student fine-tuning (Xu et al., 3 Feb 2026).

This recasting is significant because it makes explicit what heuristic watermarking ignores: students do not internalize arbitrary output biases uniformly. They internalize those biases through their fine-tuning gradients. Antidistillation sampling therefore becomes a general gradient-alignment mechanism whose target can be either student degradation or student detectability, depending on the downstream objective (Savani et al., 17 Apr 2025, Xu et al., 3 Feb 2026).

4. Game-theoretic interpretation and efficient defense templates

A more abstract formulation casts antidistillation as a KL-constrained teacher–student minimax game. Let $$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$8 be the reference teacher policy, $$q_{\theta_T}(\cdot \mid x_{1:t}; \phi) \propto \exp\{\log p_{\theta_T}(\cdot \mid x_{1:t})/\tau + \lambda \hat{\Delta}(\cdot \mid x_{1:t})\},$$9 the released policy, and $\phi = (\lambda,\epsilon,\tau)$0 the effective student training distribution after filtering or reweighting. With a scalar value function $\phi = (\lambda,\epsilon,\tau)$1 measuring how useful an output is for downstream distillation, the central objective is

$\phi = (\lambda,\epsilon,\tau)$2

Under active KL constraints, the one-sided best responses are exponential tilts:

$\phi = (\lambda,\epsilon,\tau)$3

The student-side rule implies adaptive evaluation: realistic attackers need not train uniformly on all released traces, but can upweight high-value examples via $\phi = (\lambda,\epsilon,\tau)$4 (Allouah et al., 21 May 2026).

Within this framework, gradient-based antidistillation sampling arises by choosing a value proxy

$\phi = (\lambda,\epsilon,\tau)$5

This is conceptually close to the original ADS objective, but it is computationally expensive because it requires gradients through an autoregressive proxy student during decoding (Allouah et al., 21 May 2026).

A cheaper defense template replaces the gradient proxy with the teacher–proxy-student likelihood gap

$\phi = (\lambda,\epsilon,\tau)$6

which yields the Product-of-Experts policy

$\phi = (\lambda,\epsilon,\tau)$7

At the token level, this becomes

$\phi = (\lambda,\epsilon,\tau)$8

or equivalently a convex mixture in logit space. This should be read as a soft suppression rule: outputs strongly favored by the teacher but not by the proxy student are downweighted, without the decoding-time backpropagation required by gradient-based ADS (Allouah et al., 21 May 2026).

5. Empirical trade-offs, Pareto frontiers, and adaptive evaluation

The original empirical claim for antidistillation sampling is a utility–poisoning advantage over ordinary stochastic decoding. On GSM8K and MATH, for equal teacher accuracy, students distilled on antidistilled traces are substantially worse than students distilled on temperature-sampled traces. In the GSM8K trade-off curve, pushing student accuracy below the undistilled baseline required temperature sampling to reduce teacher accuracy to about $\phi = (\lambda,\epsilon,\tau)$9, whereas antidistillation sampling achieved the same poisoning at approximately $\hat{\Delta}$0 teacher accuracy (Savani et al., 17 Apr 2025).

In the fingerprinting setting, the same gradient-aligned principle produces a different frontier: stronger detectability at smaller utility cost. On GSM8K, at similar teacher accuracy, ADFP reduces expected false positive rate by nearly an order of magnitude, with p-value $\hat{\Delta}$1 versus $\hat{\Delta}$2 for a red-and-green-list baseline. On OASST1, it yields lower NLL degradation at a fixed p-value. In the unsupervised, closed-weight, mismatched-proxy regime, ADFP achieves $\hat{\Delta}$3 at $\hat{\Delta}$4, compared with $\hat{\Delta}$5 for the baseline at the same FPR (Xu et al., 3 Feb 2026).

The game-theoretic evaluation work adds an important corrective: passive student training can materially understate leakage. On GSM8K, an ADS operating point with $\hat{\Delta}$6 yielded teacher accuracy $\hat{\Delta}$7, passive student accuracy $\hat{\Delta}$8, and adaptive student accuracy $\hat{\Delta}$9, a relative gain from adaptation of $\theta_P$0. A PoE operating point with $\theta_P$1 yielded teacher accuracy $\theta_P$2, passive student accuracy $\theta_P$3, and adaptive student accuracy $\theta_P$4. On MATH, the same pattern held: under adaptive evaluation, both ADS and PoE rose from about $\theta_P$5 passive student accuracy to about $\theta_P$6 adaptive student accuracy, while PoE incurred lower runtime overhead than ADS (Allouah et al., 21 May 2026).

These results support two broad conclusions. First, antidistillation sampling can materially improve the utility–distillability or utility–detectability frontier relative to untargeted decoding or heuristic watermarking. Second, the apparent robustness of a defense depends strongly on the student’s data-selection policy; passive evaluation is not a reliable upper bound on leakage (Savani et al., 17 Apr 2025, Allouah et al., 21 May 2026).

6. Related antidistillation methods and alternate uses of the term

Antidistillation sampling now sits within a broader landscape of defenses that intervene at different points in the generation pipeline.

Method	Intervention point	Salient property
ADS	Decoding time	Proxy-gradient logit perturbation
ADFP	Decoding time + audit	Gradient-aligned fingerprint planting
PoE	Decoding time	Forward-pass-only teacher–proxy mixture
PART	Post-generation trace rewrite	Removes self-talk, reorders sub-conclusions
TraceGuard	Post-generation trace edit	Deletes sparse high-importance sentences
Diffusion “Antidistillation Sampling”	Diffusion inference scheduler	Adaptive timestep allocation and stabilization

Post-generation reformulation methods are designed for settings where the teacher distribution itself should remain unchanged. PART rewrites reasoning traces by removing self-talk behaviors and reordering sub-conclusions into a conclusion-before-process format. The method is motivated by a discrepancy between how humans interpret reasoning traces and how LLM students exploit token-level and structural patterns under supervised fine-tuning. It uses a compact auxiliary reformulation model, adds less than $\theta_P$7 additional parameters and about $\theta_P$8 computational overhead relative to a large reasoning teacher, and was reported to degrade even a $\theta_P$9B student on AIME 2024 from $x$0 to $x$1, corresponding to a $x$2 degradation (Ding et al., 13 Oct 2025).

TraceGuard is a black-box alternative derived from a Stackelberg formulation. Rather than perturbing every token, it sparsely removes high-importance “thought anchor” sentences after generation. Its theoretical motivation is a detectability bound for sparse perturbations: for a single token with logit noise $x$3, the expected divergence satisfies

$x$4

and for $x$5 independent tokens the bound scales linearly as $x$6. This motivates poisoning fewer, more important locations rather than diffusely perturbing every token (Hartman et al., 25 Apr 2026).

A separate literature uses the same label in diffusion distillation. There, “Antidistillation Sampling” denotes an adaptive scheduler for consistency-distilled diffusion models, with target timestep selection based on an importance score derived from the SNR change rate, optimized alternating denoise/re-noise moves, and stabilization via smoothing clipping and color balancing. The method is explicitly described as applicable across PCM, LCM, CTM/TCD/TDD-style frameworks and reports large improvements in FID, CLIP score, and Inception Score on SDXL and SD v1-5 at $x$7, $x$8, and $x$9 steps under $$\theta_P^{+} = \theta_P + \eta \nabla_{\theta_P} \log p_{\theta_P}(x \mid x_{1:t}),$$0 (Wang et al., 16 Sep 2025). This diffusion usage shares the idea of correcting distillation-induced brittleness at inference time, but it addresses generative fidelity and scheduler rigidity rather than LLM trace leakage.

7. Limitations, failure modes, and open research directions

Several limitations recur across the literature. The original proxy-gradient formulation depends on access to a meaningful proxy student and a downstream loss gradient; it is most relevant when an attacker trains on full reasoning traces, and it has limited effect when only final answers are exposed (Savani et al., 17 Apr 2025). The fingerprinting reformulation depends on the approximation $$\theta_P^{+} = \theta_P + \eta \nabla_{\theta_P} \log p_{\theta_P}(x \mid x_{1:t}),$$1, which is exact only in a restricted adaptation regime and may not hold uniformly across architectures; low-rank or Fisher-preconditioned approximations are explicitly proposed as possible refinements (Xu et al., 3 Feb 2026).

Robustness against adaptive attackers remains unresolved. The minimax analysis shows that students can recover substantially more capability by reweighting high-value examples than passive training would suggest, narrowing the apparent gap between expensive gradient-based defenses and cheaper alternatives such as PoE (Allouah et al., 21 May 2026). Related concerns appear in trace-level defenses: sophisticated students may adapt to conclusion-before-process traces, retrieve over multiple diversified variants, or reduce their reliance on exposed reasoning altogether (Ding et al., 13 Oct 2025).

A second class of limitations concerns detectability, coherence, and deployment overhead. Large $$\theta_P^{+} = \theta_P + \eta \nabla_{\theta_P} \log p_{\theta_P}(x \mid x_{1:t}),$$2 can degrade teacher coherence or accuracy in ADS-style decoding (Savani et al., 17 Apr 2025). Sparse post-generation edits reduce detectability pressure but may miss critical training signal if the heuristics for high-importance sentence selection are weak (Hartman et al., 25 Apr 2026). Fingerprinting methods must also manage key security, context deduplication, and statistical calibration under open-weight and closed-weight access regimes (Xu et al., 3 Feb 2026).

Future directions in the cited work converge on hybridization and stronger evaluation. Proposed extensions include multi-proxy ensembles, adaptive online recomputation of proxy objectives under task drift, joint optimization of provenance and antidistillation, hybrid combinations of post-hoc reformulation with sampling-time defenses, and broader audits using ROC/AUC or adaptive student evaluation rather than passive fine-tuning alone (Savani et al., 17 Apr 2025, Xu et al., 3 Feb 2026, Allouah et al., 21 May 2026, Ding et al., 13 Oct 2025). A plausible implication is that antidistillation sampling is best understood not as a single algorithm, but as a general principle: inference-time output shaping should be aligned with the student’s actual learning dynamics, and any reported robustness should be interpreted relative to an attacker who may filter, reweight, or structurally adapt to those outputs.