PromptCOS: Black-box Prompt Copyright Auditing

• The paper introduces PromptCOS, a system prompt watermarking method that enables copyright auditing using only generated text outputs.
• It jointly optimizes a watermarked prompt with auxiliary and cover tokens, a secret verification query, and a content-level signal mark to balance effectiveness, fidelity, and robustness.
• Empirical evaluations show high watermark similarity, minimal accuracy degradation, and significant efficiency gains over prior methods in various benchmark tasks.

PromptCOS is a system prompt watermarking and copyright-auditing method for LLMs that is designed for the realistic setting in which a verifier can observe only a suspicious application’s text outputs, rather than logits, hidden states, or source prompts. It embeds a watermark into a system prompt by jointly optimizing a watermarked prompt, a secret verification query, and a content-level signal mark, then verifies suspected misuse by measuring similarity between the suspicious output and the signal mark under that query. The method is motivated by the premise that system prompts are valuable engineered assets whose theft or unauthorized reuse can damage the interests of prompt owners, while prior prompt-protection methods are often impractical because they rely on intermediate outputs that deployed applications typically do not expose (Yang et al., 3 Sep 2025).

1. Problem setting and conceptual basis

PromptCOS addresses system prompt copyright auditing. In the formulation used by the paper, a prompt owner starts from an original prompt $P$ and applies a watermark embedding function $W$ to obtain a watermarked prompt $P_w$ and a verification query $V$:

$P_w, V = W(P, \theta).$

A suspicious application uses an unknown prompt $P_s$ with backend model $\theta_s$, and the verifier issues the verification query:

$O = \theta_s(P_s, V).$

The auditing task is then to determine, from the text output $O$ alone, whether the suspicious application is using the protected prompt (Yang et al., 3 Sep 2025).

The paper defines four requirements for a useful auditing scheme: effectiveness, distinctiveness, fidelity, and robustness. Effectiveness requires the watermark to be detectable when the prompt is stolen. Distinctiveness requires benign prompts not to trigger similar verification outcomes. Fidelity requires watermarking not to substantially harm the prompt’s original task performance. Robustness requires the watermark to resist removal or evasion attempts. PromptCOS is built around the claim that these goals can be achieved in a content-only access model, which the paper treats as both more realistic and more difficult than logit-based verification (Yang et al., 3 Sep 2025).

A central conceptual distinction is between logit-dependent prompt protection and content-level auditing. Prior methods such as PromptCare-style approaches are described as relying on intermediate outputs, especially logits, by inducing a distinctive next-token distribution after a hidden trigger. PromptCOS instead makes the watermark manifest as a distinctive content-level pattern in generated text, so verification can be performed from ordinary API or interface outputs. This design is explicitly motivated by practical deployment conditions in chatbots, search assistants, and general LLM services, where only final text is typically accessible (Yang et al., 3 Sep 2025).

2. Watermark embedding architecture

PromptCOS embeds the watermark through four coupled design elements: a special verification query, a content-level signal mark, cyclic output signals, and two classes of modified prompt tokens called auxiliary tokens and cover tokens. The watermark is therefore not a visible tag inserted into the prompt, but a latent behavior: under a secret query, the protected prompt tends to emit a characteristic output string (Yang et al., 3 Sep 2025).

The signal mark is a token sequence $S_m$ selected from the prompt itself, but constrained to exclude verification-query tokens. The candidate set is defined as

$W$0

and the mark is chosen by minimizing the effectiveness loss over candidates from this set. This construction is meant to balance effectiveness and distinctiveness: tokens already present in the prompt are easier for the model to reproduce, while excluding tokens from $W$1 reduces the chance that a benign model emits the mark merely because the query contains it (Yang et al., 3 Sep 2025).

To stabilize generation, PromptCOS does not optimize toward a single one-shot target string. Instead it uses cyclic output signals of the form

$W$2

where $W$3 is the signal strength. The paper’s effectiveness objective is

$W$4

The purpose of the cyclic structure is to give the model multiple chances to emit the signal mark during stochastic autoregressive generation, reducing brittleness relative to one-shot watermark targets (Yang et al., 3 Sep 2025).

The auxiliary tokens $W$5 are additional prompt tokens inserted into the original prompt and optimized to carry watermark semantics while reducing direct perturbation of the original prompt text. Their role is primarily fidelity-oriented: watermarking pressure is concentrated in a small injected token set rather than distributed across the entire prompt. Fidelity is modeled with

$W$6

and the basic watermark optimization is

$W$7

The benchmark set $W$8 is used here to preserve the prompt’s original task behavior (Yang et al., 3 Sep 2025).

The cover tokens $W$9 are optimized original prompt tokens that make malicious deletion of $P_w$0 costly. Their associated deletion penalty is

$P_w$1

The final robust prompt optimization is

$P_w$2

This mechanism does not make deletion impossible; rather, it entangles auxiliary-token removal with a drop in prompt utility, so clean removal becomes difficult (Yang et al., 3 Sep 2025).

3. Optimization procedure

PromptCOS uses alternating token optimization with greedy gradient search over discrete tokens. The paper separates optimization into prompt-side variables and verification-side variables. For the prompt-embedded components, it groups losses as

$P_w$3

where $P_w$4 is described as a typographic inconsistency and corresponds to the fidelity term. For verification variables, the paper uses

$P_w$5

Thus prompt-side tokens are optimized for effectiveness, fidelity, and robustness, whereas the verification query and signal mark are optimized primarily for effectiveness (Yang et al., 3 Sep 2025).

The appendix specifies an attention-based heuristic for choosing where to inject and which original tokens to treat as covers. If $P_w$6 is the prompt-token attention score vector, the insertion position for auxiliary tokens is

$P_w$7

so the injected tokens enter a low-attention region to minimize semantic disruption. Cover tokens are selected from the highest-attention positions:

$P_w$8

This strategy makes auxiliary tokens easy to insert while making their removal costly through dependency on important original tokens (Yang et al., 3 Sep 2025).

The discrete search primitive is defined for an input sequence $P_w$9 and target generation sequence $V$0 by

$V$1

For a token $V$2, the ideal discrete update is

$V$3

and greedy gradient search approximates this through the embedding-layer gradient:

$V$4

Top-$V$5 candidates are then explicitly evaluated and the best improving replacement is kept. This search primitive is used alternately over auxiliary tokens, cover tokens, verification-query tokens, and signal-mark tokens (Yang et al., 3 Sep 2025).

The paper gives concrete default hyperparameters: batch size for $V$6 is 4, top-$V$7 is 100, the number of auxiliary tokens is 3, verification query length is 5 tokens, signal mark length is 3 tokens, $V$8, $V$9, signal strength is 2, the number of cover tokens is 5, and experiments are run on an NVIDIA A6000 GPU. The appendix also specifies a simple template skeleton with prefix "System: ", infix " Input: ", and suffix " Output: " (Yang et al., 3 Sep 2025).

4. Verification procedure

The verification phase is intentionally simple and black-box. The verifier needs only the secret verification query $P_w, V = W(P, \theta).$0, the secret signal mark $P_w, V = W(P, \theta).$1, and access to the suspicious application’s generated text. No logits, hidden states, source code, or prompt text are required (Yang et al., 3 Sep 2025).

Given generated content $P_w, V = W(P, \theta).$2 of length $P_w, V = W(P, \theta).$3 and a signal mark of length $P_w, V = W(P, \theta).$4, the paper defines all sliding windows

$P_w, V = W(P, \theta).$5

For each window, character-level similarity is computed as

$P_w, V = W(P, \theta).$6

and the final watermark score is the maximum over windows:

$P_w, V = W(P, \theta).$7

This sliding-window maximum addresses uncertain position, uncertain frequency, and minor local deviations in generated text (Yang et al., 3 Sep 2025).

The decision rule is threshold-based: if and only if the watermark similarity exceeds an empirical threshold, the suspicious application is treated as infringing. The paper does not specify a universal threshold in the body text. Instead, it assesses separation between true and false cases through True-WS, False-WS, and MDWS, where MDWS is the minimum distance between the watermark similarity achieved by protected prompts and that achieved by normal prompts (Yang et al., 3 Sep 2025).

Two common misconceptions are explicitly countered by the paper’s design. First, the watermark is not a human-readable ownership label; optimized signal marks are typically odd strings such as educaçãoscrapy, WORLDalipicker, or Să Więmedad pessi. Second, content-only verification does not eliminate secrecy assumptions: the method assumes the adversary does not know the verification query or signal mark, because leakage of either would make filtering attacks more plausible (Yang et al., 3 Sep 2025).

5. Empirical evaluation

PromptCOS is evaluated on three representative domains: BIGBENCH-II for question answering, GSM8K for math reasoning, and HumanEval for coding. The evaluated LLMs are Deepseek-distilled-qwen-1.5b, Gemma2-2b-it, TinyLlama-1.1b-chat, CodeGemma-2B, and CodeGemma-7B. Prompt generation is based on Automatic Prompt Engineer (APE) so that protected prompts are automatically produced rather than manually authored, and optimization is carried out over randomly selected prompts from that pool (Yang et al., 3 Sep 2025).

The baselines are PC*, a PromptCare-style method under logits-available conditions; PCG, a content-only PromptCare adaptation; and PR, a repetition-based content-only method. Evaluation is organized around Watermark Similarity (WS), True-WS, False-WS, MDWS, Accuracy degradation, BERTScore, and watermark embedding time (Yang et al., 3 Sep 2025).

The main reported results are that PromptCOS achieves 99.3% average watermark similarity, 60.8% greater distinctiveness than the best baseline, accuracy degradation of no more than 0.58%, robustness against three types of potential attacks, and up to 98.1% reduction in computational cost (Yang et al., 3 Sep 2025). In the paper’s aggregate comparison, PromptCOS obtains average True-WS = 0.99, False-WS = 0.17, and MDWS = 0.74. The corresponding averages for baselines are PC*: True-WS 0.52, False-WS 0.04, MDWS 0.46; PCG: True-WS 0.50, False-WS 0.14, MDWS 0.33; and PR: True-WS 0.83, False-WS 0.73, MDWS 0.02. The paper interprets this as strong content-only effectiveness with substantially better separation between protected and benign prompts (Yang et al., 3 Sep 2025).

The fidelity results are similarly favorable. PromptCOS reports average Accuracy degradation = -0.13% and BERTScore = 0.67, compared with PC* at 0.54% / 0.70 and PCG at 1.94% / 0.50. The negative average degradation indicates that some watermarked prompts slightly improved task performance, which the paper attributes to the fidelity objective behaving like additional prompt tuning. Qualitative examples contrast this with PCG, which can induce wrong tense, multilingual nonsense, mathematical mistakes, or unusable code outputs such as <|file_separator|> (Yang et al., 3 Sep 2025).

Robustness is evaluated under redundancy, constraint, and removal attacks. Average watermark similarity remains 0.98 under redundancy and 0.99 under constraint, indicating that addition-based attacks have little effect. Under removal, watermark similarity falls to 0.21, but the corresponding prompt-quality damage is severe, with Accuracy degradation = 2.89 and BERTScore = 0.47. The paper presents this as evidence that cover-token design makes removal costly in utility terms even when it reduces detectability (Yang et al., 3 Sep 2025).

Efficiency is reported in watermark embedding time. PromptCOS requires 20.8 minutes on TinyLlama, 11.1 on Deepseek-d-qwen, 36.0 on Gemma2-it, 27.6 on CodeGemma-2B, and 43.2 on CodeGemma-7B. The corresponding PCG times are 281.7, 253.9, 461.5, 1014.2, and 2351.7 minutes. The largest gain is on CodeGemma-7B, where PromptCOS reduces embedding time from 2351.7 to 43.2 minutes, i.e. about 98.1% reduction (Yang et al., 3 Sep 2025).

Ablations identify the method’s core mechanisms. Removing cyclic signals drops True-WS from 1.00 to 0.42 and MDWS from 0.79 to 0.14, showing that repeated mark emission is essential for content-only verification. Removing auxiliary tokens raises False-WS to 0.93, collapses MDWS to 0.03, worsens AcD to 0.63, and lowers BS to 0.57, indicating that auxiliary tokens are central to both distinctiveness and fidelity. Increasing signal strength from 2 to 3 preserves high True-WS but worsens False-WS and fidelity, while increasing auxiliary tokens from 3 to 5 improves False-WS to 0.06, MDWS to 0.89, AcD to -0.38, and BS to 0.77 (Yang et al., 3 Sep 2025).

6. Interpretation, limitations, and place in the literature

PromptCOS is significant because it relocates prompt copyright auditing from hidden model internals to ordinary content outputs. In the paper’s framing, this matters because suspicious third-party applications usually expose only generated text, not logits, and because exposing logits would itself increase risks such as model extraction, knowledge stealing, and data leakage. PromptCOS therefore treats prompt ownership verification as a black-box, content-only inference problem, rather than a white-box watermark-detection problem (Yang et al., 3 Sep 2025).

Its design also reframes what a prompt watermark is. The watermark is not a static string embedded in the prompt text, nor merely a next-token bias. It is a conditional behavior induced by the joint optimization of $P_w, V = W(P, \theta).$8, $P_w, V = W(P, \theta).$9, and $P_s$0: under a secret query, the protected prompt tends to emit a distinctive signal mark. This suggests a broader view of prompt watermarking as behavioral prompt instrumentation rather than literal prompt annotation. A plausible implication is that prompt ownership can be operationalized at the level of induced model behavior, even when the protected artifact itself is not directly inspectable (Yang et al., 3 Sep 2025).

The paper is also explicit about limitations. Signal marks are usually odd strings and are not strongly semantically meaningful copyright identifiers. Verification uses character-level similarity, which is simple and effective but limited in expressivity. The method is model-specific in practice; cross-backend transfer is not the main target. Threshold selection is empirical rather than theorem-backed. Finally, the robustness story depends on secrecy: if the verification query or signal mark were leaked, stronger adaptive attacks would become more plausible (Yang et al., 3 Sep 2025).

Another important limitation is that the robustness guarantee is practical rather than absolute. Removal attacks can indeed reduce watermark similarity, but the method is designed so that successful removal is entangled with a loss of prompt value. That is a weaker claim than invulnerability. It means PromptCOS is best understood as raising the cost of prompt theft and deletion, not as providing cryptographic prompt ownership proofs (Yang et al., 3 Sep 2025).

Within the broader prompt-engineering landscape, PromptCOS occupies the protection side of a field otherwise dominated by optimization, refinement, and deployment trade-offs. Its underlying motivation is that prompts are expensive to create because they encode task instructions, style, constraints, and behavior-shaping knowledge, often requiring expert labor, proprietary data, prompt optimization loops, and substantial compute (Yang et al., 3 Sep 2025). PromptCOS therefore complements prompt-design research by treating the resulting system prompt as an asset that can be audited after deployment, under the same black-box conditions in which prompt theft would typically occur (Yang et al., 3 Sep 2025).