Interaction-Layer Antidistillation Watermarks

• Interaction-layer antidistillation watermarks are behavioral schemes that embed markers in dialogue responses rather than in token-level statistics.
• They utilize system prompt wrapping to induce follow-up questions or declarative statements, enabling effective transfer of watermark signals during distillation.
• Empirical results demonstrate high relative fidelity across model families, ensuring robust detection even under paraphrasing and other transformations.

Interaction-layer antidistillation watermarks are behavioral watermarking schemes for LLM APIs that place the watermark not in logits, token-frequency biases, or internal parameters, but in the model’s observable interaction behavior. In the formulation introduced by “Asking Back: Interaction-Layer Antidistillation Watermarks,” the defender wraps a teacher model with a system prompt that intermittently induces a behavioral marker—such as an explicit follow-up question, a low-frequency variant of that question-asking behavior, or a declarative restatement embedded in the answer. An oblivious distiller that trains on harvested prompt–response pairs inherits this behavior, and the defender later audits the suspect student through black-box queries with an LLM-as-judge rubric, without requiring access to logits or weights (Yang et al., 15 May 2026).

1. Conceptual position and motivation

The motivating threat model is unauthorized knowledge distillation from a deployed LLM API. An attacker queries a proprietary teacher, collects prompt–response pairs, and fine-tunes a smaller student on that corpus. The defender cannot prevent logging of outputs, does not control the attacker’s training pipeline, and often cannot rely on access to the student’s logits or internal states. The defensive objective is therefore not necessarily to prevent distillation, but to make it auditable after the fact through black-box interaction with the suspect model (Yang et al., 15 May 2026).

This positioning differs from token-distribution and logit-space watermarking. Output-level schemes modify the next-token distribution or logits, including distortion-free watermark ensembles, gradient-aligned antidistillation sampling, and antidistillation fingerprinting (Chen et al., 12 Feb 2026, Savani et al., 17 Apr 2025, Xu et al., 3 Feb 2026). Those approaches encode a signal in token statistics or in structured logit perturbations. Interaction-layer schemes instead move the signal “one layer higher,” to recurrent dialogue behavior visible in plain-text responses (Yang et al., 15 May 2026).

The move is motivated by a specific weakness of token-level defenses: paraphrasing and rewriting can preserve semantic knowledge while destroying the fine-grained token patterns on which detection depends. The paper explicitly situates interaction-layer watermarking against green-list watermarks, cryptographic or undetectable schemes, antidistillation sampling, and post-generation rewriting defenses, arguing that these are vulnerable because their signal depends on precise token statistics rather than on a distinct conversational behavior (Yang et al., 15 May 2026).

2. Behavioral watermark construction

The formal watermark object is a triple $(B,\Pi_B,\rho)$, where $B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$ is a marker function on prompt–response pairs, $\Pi_B$ is a trigger policy selecting the system prompt used for a given prompt, and $\rho$ is the induced marker rate

$$\rho = \mathbb{E}_{p \sim \mathcal{P},\, s_B \sim \Pi_B(p)} \big[ B(p,\, T_{s_B}(p)) \big] \in [0,1].$$

Here $T_{s_B}(p)$ denotes the teacher response under the selected system prompt (Yang et al., 15 May 2026).

In implementation, the watermark is realized entirely by system-prompt wrapping. No model weights are changed, and no decoding modification is introduced. The teacher answers normally except that, on some fraction of prompts determined by $\Pi_B$, it exhibits a prescribed dialogue behavior (Yang et al., 15 May 2026).

Marker type	Density	Behavioral form
Strong	100%	Final-sentence follow-up question ending with ?
Soft	about 20%	Same explicit follow-up question, but only on a fixed 20% subset
Style-control	about 20%	Declarative, advisory restatement integrated into the answer

The strong marker uses a system prompt instructing the model to answer directly and completely, then append a follow-up question as the final sentence, ending with a question mark, concerning the user’s use case, technical environment, or intended application. The soft marker uses the same instruction but only on a fixed subset of prompt IDs, while the remaining prompts use a baseline assistant prompt. The style-control marker replaces the interrogative signature with a declarative sentence such as “If you’re aiming for <goal>, <guidance>,” “In practice, this works best when <condition>,” “Worth noting: <context-specific consideration>,” or “This approach is optimal for <scenario>,” and explicitly forbids phrasing it as a question (Yang et al., 15 May 2026).

The low-density regimes are operationally important. In the reported setup, all seven experimental conditions share the same 3,009 prompt IDs after alignment and filtering, and 582 IDs, or 19.34%, define the low-density subset used for the soft and style-control variants. This makes the trigger policy deterministic but hidden from the attacker (Yang et al., 15 May 2026).

3. Audit formalism and black-box detection

Detection is framed as black-box behavioral auditing. The defender queries a suspect model $S$ on held-out prompts, without supplying the watermarking system prompt, and estimates how often the suspect spontaneously exhibits the marker behavior. For a marker $B$, the transfer rate is

$$\tau(S,B) = \mathbb{E}_{p \sim \mathcal{P}_{\mathrm{held}}} [ B(p, S(p)) ],$$

and the relative fidelity to the teacher is

$B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$0

The first quantity measures absolute marker incidence in the student; the second normalizes it by the teacher’s own marker rate and is the paper’s main cross-family comparison metric (Yang et al., 15 May 2026).

For paraphrase robustness, the paper studies a prompt-side paraphraser $B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$1 and decomposes robustness into teacher-side and student-side factors:

$B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$2

$B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$3

$B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$4

with

$B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$5

This decomposition separates the intrinsic robustness of the teacher-side marker under paraphrased prompts from the student’s ability to retain whatever signal remains (Yang et al., 15 May 2026).

The marker classifier is an LLM-as-judge rather than a regex. The reported system uses gpt-oss-120b, served via vLLM with reasoning_effort=high, and returns constrained JSON fields verdict, confidence, evidence, and reason. For strong markers, the rubric checks whether the response follows a system prompt requiring a complete answer followed by a final-sentence follow-up question ending in ?. For style-control markers, the rubric checks for a declarative, scenario-scoped advisory sentence rather than a question (Yang et al., 15 May 2026).

Judge calibration was validated on 200 items with three human annotators. Cohen’s $B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$6 between judge and human majority was reported as $B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$7 for the strong rubric and $B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$8 for the style rubric; Fleiss’ $B : \mathcal{X}\times\mathcal{Y}\to\{0,1\}$9 among humans was $\Pi_B$0 and $\Pi_B$1, respectively. The paper further notes that relative metrics such as $\Pi_B$2 and $\Pi_B$3 are invariant to multiplicative judge bias, because the same calibration factor cancels between numerator and denominator (Yang et al., 15 May 2026).

4. Transfer through distillation and robustness under paraphrasing

The main experimental teacher is Llama-3.3-70B-Instruct, queried with temperature $\Pi_B$4, top-$\Pi_B$5, max tokens $\Pi_B$6, repetition penalty $\Pi_B$7, and seed $\Pi_B$8. Students are three base, non-Instruct families from disjoint organizations—Qwen3.5-0.8B-Base, Gemma-3-1B-pt, and OLMo-2-0425-1B—fine-tuned with rank-$\Pi_B$9 LoRA adapters on seven projections $\rho$0, LoRA $\rho$1, learning rate $\rho$2, one epoch, bf16, and max sequence length $\rho$3. Loss is masked to response tokens only. Across three families, seven training conditions, and three seeds, the study trains $\rho$4 students and audits $\rho$5 judged samples on $\rho$6 shared held-out prompts (Yang et al., 15 May 2026).

The headline result is that behavioral watermarks do transfer across families. Under the main strong-marker conditions, transfer reaches 88.9% relative fidelity for Gemma, 80.9% for OLMo, and 45.2% for Qwen, supporting the paper’s learnability and cross-family generalization hypotheses. A within-family sanity check in a more favorable Qwen-to-Qwen setting found that a strong-marked student ended responses with a question on 450 of 452 held-out prompts, or 99.56%, whereas a baseline student did so on 23 of 452 prompts, or 5.09% (Yang et al., 15 May 2026).

Low-density markers also transfer above family-specific baselines. For style-control at about 20% density, the reported student rates are 11.71% for Gemma, 6.66% for OLMo, and 7.38% for Qwen, versus baselines in the 0.20–1.20% range. For soft explicit markers, the teacher’s clean rate on the marker subset is 17.85%, while students still display substantial learned incidence; OLMo is notable for exceeding the teacher’s own rate in one such setting (Yang et al., 15 May 2026).

Robustness was evaluated under non-adaptive DIPPER prompt-side paraphrasing with lex=60 and order=60. For strong markers, the clean teacher rate is 90.93%, the teacher rate under paraphrased prompts is 60.37%, and teacher-side retention is therefore about 66.4%. Student-relative retention then varies by family: about 89% for Gemma, about 112% for OLMo, and about 65% for Qwen. The OLMo result places it in the paper’s “amplification” regime, where the student preserves the watermark better than the teacher itself under the paraphrase condition. For low-density soft markers, student-relative retention is lower, in the reported 21–81% range, indicating that sparse markers are more fragile under prompt shift (Yang et al., 15 May 2026).

The paper interprets the OLMo amplification effect as a consequence of hidden trigger policies and limited student capacity: a student that cannot reconstruct the teacher’s exact 20% trigger subset may internalize a simpler marginal rule and over-generalize the behavior. This interpretation is presented as a hypothesis rather than a settled mechanism (Yang et al., 15 May 2026).

5. Relation to token-, model-, and parameter-layer antidistillation research

Interaction-layer watermarking belongs to a broader antidistillation landscape. At the token-distribution layer, “More Haste, Less Speed: Weaker Single-Layer Watermark Improves Distortion-Free Watermark Ensembles” studies recursive application of distortion-free watermarks and shows that strong early watermark layers reduce entropy and weaken later layers, leading to a weaker ensemble signal. Its remedy is weaker per-layer watermarking via $\rho$7, which preserves more entropy for subsequent layers (Chen et al., 12 Feb 2026). This work remains squarely at the next-token distribution layer rather than the interaction layer.

“Antidistillation Sampling” defines a modified sampling distribution

$\rho$8

where $\rho$9 estimates which next-token choices most harm a distiller’s downstream learning. The signal is a secret, structured logit transform derived from a proxy gradient on an evaluation loss, and the paper explicitly notes that the mechanism can be repurposed as a watermark-like pattern in token-level logit adjustments (Savani et al., 17 Apr 2025). “Antidistillation Fingerprinting” takes a closely related route for distillation detection: it is described as an output-level, logit-space fingerprinting method that aligns the perturbation objective with student learning dynamics using a proxy, rather than relying on heuristic green-list boosting. The paper also states that it does not explicitly construct interaction-layer or intermediate-representation watermarks, although its gradient-inner-product derivation could in principle be extended in that direction (Xu et al., 3 Feb 2026).

At the model and parameter layers, earlier antidistillation work pursued tighter coupling between the watermark and task-relevant representations. “Effectiveness of Distillation Attack and Countermeasure on Neural Network Watermarking” argues that distillation removes watermarks that are redundant to the main task and introduces ingrain, which regularizes a network against an ingrainer model so that watermark-related information is carried by the same logits and representations used for the benign task (Yang et al., 2019). “Embedding Watermarks into Deep Neural Networks” embeds a secret bitstring into internal convolutional parameters using a watermark regularizer and explicitly analyzes train-to-embed, fine-tune-to-embed, and distill-to-embed scenarios (Uchida et al., 2017). These model- and parameter-space approaches are not interaction-layer methods, but they share the design principle that a watermark is more robust when it is entangled with the model behavior that a student must preserve.

Within that taxonomy, interaction-layer watermarks are best understood as complementary. They do not require logit access, do not require teacher retraining, and are auditable through plain-text black-box interaction. Their signal is behavioral rather than statistical or parametric (Yang et al., 15 May 2026).

6. Limitations, attack surface, and open problems

The method’s stated threat model is an oblivious distiller. The attacker harvests teacher outputs and fine-tunes normally, without explicitly trying to suppress follow-up questions or stylistic declaratives. Robustness experiments cover only non-adaptive prompt-side paraphrasing with DIPPER. The paper does not evaluate response-side paraphrasing of the teacher outputs, adaptive rewriting aimed at removing markers, student-side fine-tuning to suppress the behavioral trait, or multilingual and cross-tokenizer settings (Yang et al., 15 May 2026).

The marker family is intentionally narrow. Only three behavioral variants are studied: strong explicit follow-up questions, low-density explicit questions, and low-density implicit declarative restatements. The evaluation teacher is restricted to Llama-3.3-70B-Instruct, and students are small 0.8B–1B base checkpoints fine-tuned with rank-$$\rho = \mathbb{E}_{p \sim \mathcal{P},\, s_B \sim \Pi_B(p)} \big[ B(p,\, T_{s_B}(p)) \big] \in [0,1].$$0 LoRA. The paper therefore does not establish how the transfer pattern changes under full-parameter student training, larger students, or substantially different teaching models (Yang et al., 15 May 2026).

The held-out evaluation set is also narrowed by truncation and filtering. Although the original evaluation pool contains 1,000 prompts from Alpaca, OpenAssistant, math_train, and MBPP, only 561 shared prompts remain after filtering for length and family-specific truncation issues, especially Qwen’s longer responses. This suggests that the current evidence is strongest on a shorter-prompt regime (Yang et al., 15 May 2026).

Stealth is supported, but only within a limited user study. The paper reports that all marker variants lie within 0.22 Likert steps of baseline and that TOST, Friedman, and Bonferroni-Wilcoxon analyses support the claim that moderate-density markers do not clearly degrade user experience. The detailed study description presents a small in-lab, within-subject Latin-square design, so the evidence is best read as preliminary rather than definitive (Yang et al., 15 May 2026).

A recurrent misconception is that antidistillation watermarking must operate at the token or logit level. The interaction-layer proposal directly contradicts that assumption: it treats conversational behavior itself as the carrier. A second misconception is that such watermarks are meant to prevent distillation. In the paper’s formulation, the primary function is auditability: the defender later detects inherited behavior in a suspect student. A plausible implication is that robust provenance systems will combine interaction-layer markers with token-level, model-level, and reasoning-trace-layer defenses, because the paper explicitly presents the interaction layer as a viable and complementary design locus rather than as a replacement for the others (Yang et al., 15 May 2026).