What Does It Mean to Break a Distillation Defense?

Abstract: Black-box LLMs (accessible only via API) are vulnerable to distillation attacks, in which an attacker queries the model and trains a student on its outputs. A recent line of work proposes output perturbation defenses that modify the teacher's output to reduce student performance while preserving utility for legitimate users. As a relatively new family of approaches, output perturbation defenses lack a shared threat model, making it difficult to compare them, reason about composing them with other attacks, or evaluate their robustness against realistic adversaries. This underspecification matters beyond technical evaluation: when defenses are deployed to protect intellectual property or justify regulatory compliance, an imprecise threat model can create a false sense of security. We propose a threat model framework that describes attackers along three dimensions: a query budget, a data budget, and an interface profile that captures how attackers interact with the API. Using antidistillation sampling as a case study, we show that whether the defense is considered effective depends on the assumed threat model. We argue that future work on distillation defenses, along with any governance or policy frameworks built around them, should explicitly specify and stress-test attacker capabilities along our three dimensions.

• The paper introduces a formal threat model framework for evaluating output perturbation defenses by separating query, data, and interface budgets.
• It demonstrates that minor changes in attacker capabilities, such as adding a prefill token or increasing query budget, can nearly nullify defenses like ADS.
• The evaluation reveals that defense effectiveness is highly context-dependent, emphasizing the need for explicit threat model specifications in security claims.

Summary of “What Does It Mean to Break a Distillation Defense?” (2606.25059)

Motivation and Scope

The paper analyzes output perturbation defenses against distillation attacks on API-accessed LLMs. Distillation allows adversaries to replicate proprietary model behaviors by querying the API and training a student model on returned outputs, posing a significant IP risk for frontier providers. Recent research has focused on output perturbation defenses—mechanisms that intentionally degrade output quality for attackers to impede distillation while maintaining utility for legitimate users. However, lacking a unified, explicit threat model undermines meaningful evaluation, technical comparison, and policy development for these defenses.

Threat Model Framework

The central contribution is a formal threat model framework for distillation defense evaluation. Attackers are characterized along three axes:

• Query budget: total allowed API calls.
• Data budget: number of distinct prompts.
• Interface profile: describes available input channels (e.g., prefill, sampling parameters), provider-side processing (e.g., output filtering, trace summarization), and output channels (e.g., text, logprobs).

Separating query from data budgets enables modeling strategic query allocation (e.g., requerying high-value prompts), and the interface profile exposes the degree to which attacker interaction can bypass output defenses. This (Q, D, P) tuple fully specifies threat scenarios, aligning evaluation setups with practical adversary capabilities and API design choices.

Case Study: Antidistillation Sampling (ADS)

ADS is a decoding-time defense perturbing the teacher’s next-token distribution to maximize student model loss, using a proxy student to guide the perturbation. The strength of perturbation ($\lambda$) trades off teacher utility for student degradation; high-utility regimes retain $\geq$70% teacher accuracy.

The evaluation demonstrates stark dependence on threat model specification:

• Black-box minimal profile, Q = D: Simple attacker-side post-processing (e.g., repetition deletion) recovers substantial student accuracy, exceeding the originally reported baseline. Further, query reallocation (requerying incorrectly answered prompts) provides marginal gains.
• Scaling query budget: Increasing Q relative to D (reflecting attacks using multiple resampling per prompt) nearly neutralizes ADS across most $\lambda$ values. The “matched-utility” budget, equal to the expected queries a legitimate user performs to overcome degraded outputs, results in near-complete recovery of student performance.
• Prefill input channel: Adding a prefill token (“First”) to each generation further mitigates ADS’s effect due to reduced output entropy, enabling the student to almost fully recover temperature-sampling performance.

These results underline that defense efficacy is highly contingent on subtle attacker capabilities and interface design choices. For instance, prefill access—which some APIs permit and others restrict—renders ADS ineffective, demonstrating that security depends not just on output perturbation’s design, but on the deployment context.

Implications

The paper makes several bold claims:

• Output perturbation defenses can be rendered ineffective by minor threat model changes (e.g., single-token prefix, increased query allocation).
• Evaluations anchored on implicit, underspecified attacker profiles overstate robustness and risk fostering a false sense of IP security.
• Practical security and governance depend critically on explicit, standardized threat models.

The broader implication is that current practices in reporting and deploying distillation defenses risk creating a gap between perceived and actual protection. Without explicit attacker specification and stress-testing across the threat model dimensions, any apparent security is illusory. This carries tangible consequences for regulatory and IP enforcement strategies as well as investment in genuine defense development.

Future Directions

The authors advocate for:

• Adopting explicit threat model reporting as a minimum standard for output perturbation defense work.
• Evaluating defenses across the full threat model space (varying Q, D, and P), mirroring adversarial ML’s adaptive attack conventions.
• Regular re-evaluation of deployed defenses in response to API changes.
• Using budgets and interface profiles grounded in realistic attacker constraints, including behavior mirroring legitimate-users.

This framework—defense claims bound to (Q, D, P) tuples—provides the vocabulary needed for responsible research, deployment, and policy discussions. As distillation attacks become a central concern for providers and regulators, robust evaluation standards are fundamental for industry coordination and for ensuring policy measures reflect actual security properties.

Conclusion

Output perturbation distillation defenses remain vulnerable to adaptive adversaries and strategic exploitation of API interfaces. Defense efficacy is not an inherent property, but a function of the assumed attacker, budget constraints, and API configuration. The paper’s threat model framework sets a new benchmark for evaluating and deploying distillation defenses, emphasizing that only security claims explicitly tied to threat model specification are meaningful for both technical and governance purposes. Future developments in AI security should integrate this methodology to avoid the pitfalls of false assurances and to guide the evolution of robust protection mechanisms.