Asking Back: Interaction-Layer Antidistillation Watermarks

Abstract: Detecting unauthorized knowledge distillation from a deployed LLM API is hard because the defender controls neither the attacker's training pipeline nor the next-token logits. Existing defenses operate on the teacher's output tokens -- biasing the next-token distribution (green-list watermarks, cryptographic schemes, antidistillation sampling) or rewriting outputs after generation. Recent work shows a paraphrasing attacker can strip these signals without losing the underlying knowledge. We propose interaction-layer antidistillation watermarks, which move the trace one layer higher, into the teacher's interaction behavior: the defender wraps the teacher with a system prompt that intermittently induces a behavioral marker -- an explicit follow-up question, a low-frequency variant, or a declarative restatement. An oblivious distiller inherits the behavior, and the defender audits via black-box queries with a human-validated LLM-as-judge (Cohen's kappa = 0.84/0.78 on strong/style rubrics). Across 63 LoRA-distilled students under a Llama-3.3-70B-Instruct teacher (35,343 judged samples), behavioral watermarks transfer at 88.9% (Gemma) / 80.9% (OLMo) / 45.2% (Qwen) relative fidelity (H1, H2). Under non-adaptive DIPPER paraphrasing, robustness decomposes into a teacher-self ceiling (about 66.4%) and student-relative retention of 21-112%, with OLMo preserving the watermark above the teacher itself (H3, F-Amp). Low-density (about 20%) explicit and implicit declarative variants transfer above per-family baseline (H4, F-Style). An N=20 in-lab study (pre-registered Latin-square) shows all marker variants within 0.22 Likert step of baseline; TOST, Friedman, and Bonferroni-Wilcoxon support H5. The interaction layer is a viable design locus for antidistillation watermarking, complementary to token-, model-, and reasoning-trace-layer defenses.

• The paper introduces a novel defense mechanism that injects behavioral markers into LLM responses to robustly trace unauthorized knowledge distillation.
• It evaluates marker transfer rates, paraphrase robustness, and amplification effects across multiple model families using controlled experimental designs.
• Empirical results show that behavioral watermarks preserve user experience, complement token-level defenses, and offer a tunable stealth–detection trade-off.

Interaction-Layer Antidistillation Watermarks: Behavioral Tracing for API-Served LLMs

Motivation and Problem Context

Modern LLMs, particularly frontier models, are predominantly accessed via paid black-box APIs. This deployment model exposes model owners to unauthorized knowledge distillation attacks, where adversaries harvest prompt–response pairs, fine-tune a student model, and release the clone without authorization. Traditional antidistillation watermarks involve modifying the teacher’s token-level output statistics, model weights, or chain-of-thought traces so that unauthorized distillation can be audited post hoc. However, recent work demonstrates that such signals are vulnerable: paraphrasing or rewriting harvested outputs before student training can defeat most prior watermarking strategies [krishna2023paraphrasing], [pan2025watermarks].

This paper introduces a fundamentally different approach—interaction-layer antidistillation watermarks—where the teacher’s behavioral tendencies rather than response tokens are watermarked. Rather than relying exclusively on statistical or cryptographic marks, a wrapper injects intermittent prompts that produce distinctive behavioral markers (e.g., follow-up questions or conditional restatements) in the teacher’s responses. This emerges as a complementary defense locus distinct from existing token-, model-, or chain-of-thought-layer approaches.

Figure 1: The three layers of the Asking-Back threat model: attacker harvests prompt–response pairs (top); defender injects a behavioral marker at chosen density (middle); defender audits the student model via black-box LLM-judge assessment (bottom).

Methodology: Designing and Formalizing Interaction-Layer Watermarks

The defender wraps the teacher API using a system prompt that intermittently triggers a behavioral marker $B$, which is expressed in one of three forms:

• Strong (100% density): Trailing explicit follow-up questions.
• Soft (≈20% density): Occasional trailing follow-up questions.
• Style-Control (≈20% density): Declarative, scenario-scoped meta-comments embedded in the response.

The attacker, lacking access to the system prompt or trigger policy, unknowingly harvests prompt–response pairs containing $B$. Standard supervised distillation is then performed to train the student model on these marked outputs. Subsequent auditing is black-box: querying the suspect model and using an LLM-as-judge to assess the frequency of $B$ relative to a calibrated teacher baseline.

The formalism characterizes the watermark as a tuple $(B,\,\Pi_B,\,\rho)$, where $\Pi_B$ is the trigger policy and $\rho$ is the achieved marker rate. Distillation transfer is captured by comparing the marker’s prevalence in the teacher and student ($\tau$, $\tau_{\mathrm{rel}}$), and robustness to paraphrasing is factored into teacher-self and student-relative terms $(R_T,\,R_{\mathrm{rel}})$ to separate the ceiling imposed by paraphrasing on the teacher from robustness internal to the student pipeline.

Experimental Design

A comprehensive cross-family study was conducted: three base model families (Qwen3.5-0.8B-Base, Gemma-3-1B-pt, and OLMo-2-0425-1B), each distilled from a Llama-3.3-70B-Instruct teacher over seven conditions (including paraphrase attacks), across three seeds for a total of 63 LoRA students and over 35,000 human-validated samples. Key aspects include:

• Teacher outputs drawn from a 4,500-prompt mixture covering diverse conversational domains.
• LoRA adapters with fixed rank ensured capacity control and minimized overfitting.
• Robust auditing using structured-output judgments from GPT-OSS-120B, validated against human annotators (Cohen's $\kappa = 0.84/0.78$ on strong/style rubrics).
• Paraphrase robustness was tested using DIPPER (lex=60, order=60).
• User impact (stealth) was measured by an in-lab, Latin-square-controlled $B$0 user study assessing subjective interaction quality.

Empirical Results

Transferability and Cross-Family Generalization

Strong behavioral watermarks transfer robustly through distillation:

• Strong marker relative transfer rates: Gemma: 88.9%, OLMo: 80.9%, Qwen: 45.2% (against a 90.93% teacher baseline).
• Intermediate retention for soft markers and lower—but still significantly above baseline—for style-control (implicit) markers, indicating learnability of both explicit and implicit interaction patterns.

  Figure 2: Detection rates across the $B$1 matrix: Strong, Soft, and Style-Control markers are detectable above baseline for every family; OLMo soft student exceeds the teacher’s soft rate (amplification effect).

Robustness to Paraphrasing

Marked behaviors retain partial robustness to prompt-side paraphrasing:

• Teacher-side paraphrasing degrades marker rates from 90.93%→60.37% (strong) and 17.85%→11.93% (soft).
• Student retention of the marker under paraphrased training data (strong): Gemma: 89%, OLMo: 112% (amplifies the marker relative to teacher), Qwen: 65%; for soft: retention is 21–81%.

Figure 3: Left: Paraphrase robustness against DIPPER; teacher paraphrased rates are shown as reference lines.

Density–Detection Trade-off and Amplification

Detection is monotonic in the density $B$2 of marker injection—students reliably exceed per-family baseline rates even at 20% marker density. Remarkably, OLMo low-density students exhibited amplification, expressing the marker more frequently than the teacher itself (31.99% vs. 17.85%). This family-specific anomaly is interpreted as a potential marginal collapse in capacity-constrained students unable to track the teacher’s trigger predicate, but overfitting to marker-carrying tokens.

Figure 4: Left: Dose-response of student detection vs. teacher-side density; Right: Stealth–robustness–detection trade-off highlighting practical operating regions.

Stealth and User Perception

The $B$3 Latin-square user study found that all low-density marker conditions fell within $B$4 Likert steps of baseline, with no statistically significant user-disfavored degradation of interaction quality, validating the approach’s stealth in practical deployments.

(Figure 3, right panel)

Figure 3: H5 in-lab study: Mean Likert ratings per condition reveal negligible perceived difference versus baseline.

Qualitative and Statistical Validation

Human verification yielded Cohen’s $B$5 on explicit questioning and 0.78 on style-control rubrics, ensuring the black-box judge’s reliability for deployment scenarios.

Theoretical and Practical Implications

The results challenge the notion that distillation exclusively propagates surface-level statistical patterns; both explicit and subtle interactional behaviors propagate robustly, supporting the thesis that internalization of discourse-level competencies occurs even with limited capacity/adapter-based transfer.

Key practical implications:

• Behavioral auditing provides a robust, black-box detection channel, sidestepping vulnerabilities to response-level paraphrasing that erode token-level watermarks.
• Density–robustness–stealth trade-off is tunable, with family-dependent optimal densities suggestive of future adaptive defenses.
• Amplification: Small models under low-density behavioral marking can inadvertently exceed teacher’s marker prevalence, complicating evasion efforts for attackers.

Complementarity with Prior Defenses

Behavioral watermarks do not replace token-, model-, or chain-of-thought-level mechanisms but are strictly complementary. Joint deployment increases audit resilience, targeting disjoint adversarial pipelines (paraphrasing, rewriting, or filtering).

Open Problems and Future Directions

The study identifies several limitations and directions:

• Evaluation limited to prompt-side, non-adaptive paraphrasing; adaptive or response-side rewriting attacks remain open.
• Only English-language, single-tokenizer/family assessed.
• Small $B$6 user study; replication at larger scale is necessary.
• Mechanistic basis of the amplification phenomenon requires further analysis (multi-point $B$7, capacity scans).
• Theoretical characterization of the stealth–detection–robustness Pareto frontier is outstanding.

Conclusion

This work establishes interaction-layer antidistillation watermarks as an effective, complementary approach for defending API-served LLMs against unauthorized distillation. Injection of behavioral markers at the interaction layer survives knowledge transfer across families, shows nontrivial robustness to paraphrasing, preserves user experience at low densities, and exposes new transfer phenomena (e.g., amplification in small students). Collectively, these findings recommend behavioral watermarking as an operationally practical and theoretically compelling addition to the AI provenance and model auditing toolbox.

---

• "Paraphrasing Evades Detectors of AI-generated Text, but Retrieval is an Effective Defense" (Krishna et al., 2023)
• "Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?" [2025.acl-long.648]
• "A Watermark for LLMs" (Kirchenbauer et al., 2023)
• "Antidistillation Sampling" [Vo2UHqMu8t]
• "Radioactive Data: Tracing through Training" (Sablayrolles et al., 2020)
• [Full paper: "Asking Back: Interaction-Layer Antidistillation Watermarks" (2605.16462)]