---
title: 'PromptCOS: Black-box Prompt Copyright Auditing'
url: https://www.emergentmind.com/topics/promptcos
type: topic
---

# PromptCOS: Black-box Prompt Copyright Auditing

PromptCOS is a system prompt watermarking and copyright-auditing method for large language models that is designed for the realistic setting in which a verifier can observe only a suspicious application’s text outputs, rather than logits, hidden states, or source prompts. It embeds a watermark into a system prompt by jointly optimizing a watermarked prompt, a secret verification query, and a content-level signal mark, then verifies suspected misuse by measuring similarity between the suspicious output and the signal mark under that query. The method is motivated by the premise that system prompts are valuable engineered assets whose theft or unauthorized reuse can damage the interests of prompt owners, while prior prompt-protection methods are often impractical because they rely on intermediate outputs that deployed applications typically do not expose [2509.03117].

## 1. Problem setting and conceptual basis

PromptCOS addresses **system prompt copyright auditing**. In the formulation used by the paper, a prompt owner starts from an original prompt \(P\) and applies a watermark embedding function \(W\) to obtain a watermarked prompt \(P_w\) and a verification query \(V\):
$$
P_w, V = W(P, \theta).
$$
A suspicious application uses an unknown prompt \(P_s\) with backend model \(\theta_s\), and the verifier issues the verification query:
$$
O = \theta_s(P_s, V).
$$
The auditing task is then to determine, from the text output \(O\) alone, whether the suspicious application is using the protected prompt [2509.03117].

The paper defines four requirements for a useful auditing scheme: **effectiveness**, **distinctiveness**, **fidelity**, and **robustness**. Effectiveness requires the watermark to be detectable when the prompt is stolen. Distinctiveness requires benign prompts not to trigger similar verification outcomes. Fidelity requires watermarking not to substantially harm the prompt’s original task performance. Robustness requires the watermark to resist removal or evasion attempts. PromptCOS is built around the claim that these goals can be achieved in a **content-only** access model, which the paper treats as both more realistic and more difficult than logit-based verification [2509.03117].

A central conceptual distinction is between **logit-dependent prompt protection** and **content-level auditing**. Prior methods such as PromptCare-style approaches are described as relying on intermediate outputs, especially logits, by inducing a distinctive next-token distribution after a hidden trigger. PromptCOS instead makes the watermark manifest as a distinctive content-level pattern in generated text, so verification can be performed from ordinary API or interface outputs. This design is explicitly motivated by practical deployment conditions in chatbots, search assistants, and general LLM services, where only final text is typically accessible [2509.03117].

## 2. Watermark embedding architecture

PromptCOS embeds the watermark through four coupled design elements: a **special verification query**, a **content-level signal mark**, **cyclic output signals**, and two classes of modified prompt tokens called **auxiliary tokens** and **cover tokens**. The watermark is therefore not a visible tag inserted into the prompt, but a latent behavior: under a secret query, the protected prompt tends to emit a characteristic output string [2509.03117].

The **signal mark** is a token sequence \(S_m\) selected from the prompt itself, but constrained to exclude verification-query tokens. The candidate set is defined as
$$
T_{\text{mark}} \triangleq \{ t \mid (t \in P_w) \land (t \notin V) \},
$$
and the mark is chosen by minimizing the effectiveness loss over candidates from this set. This construction is meant to balance effectiveness and distinctiveness: tokens already present in the prompt are easier for the model to reproduce, while excluding tokens from \(V\) reduces the chance that a benign model emits the mark merely because the query contains it [2509.03117].

To stabilize generation, PromptCOS does not optimize toward a single one-shot target string. Instead it uses **cyclic output signals** of the form
$$
\{[\text{mark}][\text{pad}]\} \times s,
$$
where \(s\) is the signal strength. The paper’s effectiveness objective is
$$
\mathcal{L}_e = \log(1 - \mathbb{P}_{\theta}(S \times s \mid P_w + V)).
$$
The purpose of the cyclic structure is to give the model multiple chances to emit the signal mark during stochastic autoregressive generation, reducing brittleness relative to one-shot watermark targets [2509.03117].

The **auxiliary tokens** \(P_a\) are additional prompt tokens inserted into the original prompt and optimized to carry watermark semantics while reducing direct perturbation of the original prompt text. Their role is primarily fidelity-oriented: watermarking pressure is concentrated in a small injected token set rather than distributed across the entire prompt. Fidelity is modeled with
$$
\mathcal{L}_f = \sum_{i=1}^{|B|} \log(1 - \mathbb{P}_{\theta}(B_{\text{output}^{(i)}} \mid P_w + B_{\text{input}^{(i)}})),
$$
and the basic watermark optimization is
$$
P_w \leftarrow \underset{P_a}{\arg\min} (\mathcal{L}_e + r_{1}\mathcal{L}_f).
$$
The benchmark set \(B\) is used here to preserve the prompt’s original task behavior [2509.03117].

The **cover tokens** \(P_c\) are optimized original prompt tokens that make malicious deletion of \(P_a\) costly. Their associated deletion penalty is
$$
\mathcal{L}_d = - \sum_{i=1}^{n} \log(1 - \mathbb{P}_{\theta}(B_{i,\text{output}} \mid (P_w - P_a) + B_{i,\text{input}})).
$$
The final robust prompt optimization is
$$
P_w \leftarrow \underset{P_a, P_c}{\arg\min} (\mathcal{L}_e + r_{1}\mathcal{L}_f + r_{2}\mathcal{L}_d).
$$
This mechanism does not make deletion impossible; rather, it entangles auxiliary-token removal with a drop in prompt utility, so clean removal becomes difficult [2509.03117].

## 3. Optimization procedure

PromptCOS uses **alternating token optimization** with **greedy gradient search** over discrete tokens. The paper separates optimization into prompt-side variables and verification-side variables. For the prompt-embedded components, it groups losses as
$$
\begin{aligned}
\mathcal{L}_1 = \mathcal{L}_e + r_1 \mathcal{L}_h + r_2 \mathcal{L}_d, \\
P_w \leftarrow \underset{P_a, P_c}{\arg\min} \mathcal{L}_1,
\end{aligned}
$$
where \(\mathcal{L}_h\) is described as a typographic inconsistency and corresponds to the fidelity term. For verification variables, the paper uses
$$
\begin{aligned}
\mathcal{L}_2 &= \mathcal{L}_e, \\
V, S \leftarrow \underset{V, S_m}{\arg\min} \mathcal{L}_2.
\end{aligned}
$$
Thus prompt-side tokens are optimized for effectiveness, fidelity, and robustness, whereas the verification query and signal mark are optimized primarily for effectiveness [2509.03117].

The appendix specifies an attention-based heuristic for choosing where to inject and which original tokens to treat as covers. If \(A = [a_1, a_2, \dots, a_n]\) is the prompt-token attention score vector, the insertion position for auxiliary tokens is
$$
p = \underset{1 \leq i < n}{\operatorname{arg\,min}} \left( a_i + a_{i+1} \right),
$$
so the injected tokens enter a low-attention region to minimize semantic disruption. Cover tokens are selected from the highest-attention positions:
$$
P_c = \left\{ t_j \mid j \in \underset{1 \leq j \leq n}{\operatorname{arg\,topk}} \, a_j, \, |P_c| = m \right\}.
$$
This strategy makes auxiliary tokens easy to insert while making their removal costly through dependency on important original tokens [2509.03117].

The discrete search primitive is defined for an input sequence \(Q\) and target generation sequence \(G\) by
$$
\mathcal{L} = \sum_{i=1}^{n} \log(1 - p(g_i \mid Q, g_1, g_2, \ldots, g_{i-1})).
$$
For a token \(q_i\), the ideal discrete update is
$$
q_i = \mathop{\arg\min}\limits_{q_i} \mathcal{L},
$$
and greedy gradient search approximates this through the embedding-layer gradient:
$$
e'_i = \underset{e'_i \in V}{\arg\min} \left( e'_i - e_i \right)^T \nabla_{q_i} \mathcal{L}.
$$
Top-\(k\) candidates are then explicitly evaluated and the best improving replacement is kept. This search primitive is used alternately over auxiliary tokens, cover tokens, verification-query tokens, and signal-mark tokens [2509.03117].

The paper gives concrete default hyperparameters: batch size for \(\mathcal{L}_f\) is **4**, top-\(k\) is **100**, the number of auxiliary tokens is **3**, verification query length is **5 tokens**, signal mark length is **3 tokens**, \(r_1 = 0.5\), \(r_2 = 0.5\), signal strength is **2**, the number of cover tokens is **5**, and experiments are run on an **NVIDIA A6000 GPU**. The appendix also specifies a simple template skeleton with prefix `"System: "`, infix `" Input: "`, and suffix `" Output: "` [2509.03117].

## 4. Verification procedure

The verification phase is intentionally simple and black-box. The verifier needs only the secret verification query \(V\), the secret signal mark \(S_m\), and access to the suspicious application’s generated text. No logits, hidden states, source code, or prompt text are required [2509.03117].

Given generated content \(C\) of length \(L\) and a signal mark of length \(l\), the paper defines all sliding windows
$$
\begin{aligned}
C_w \in \big\{ c_i, c_{i+1}, \ldots, c_{i+l-1} \mid i \in [0, L - l) \big\}.
\end{aligned}
$$
For each window, character-level similarity is computed as
$$
\begin{aligned}
\text{Similarity}(S_m, C_w) &= \frac{1}{l} \sum_{k=0}^{l-1} 1\left(S_m^{(i+k)} = C_w^{(i+k)}\right),
\end{aligned}
$$
and the final watermark score is the maximum over windows:
$$
\begin{aligned}
\text{Watermark Similarity}(S_m, C) = \max_{C_w} \big( \text{Similarity}(S_m, C_w) \big).
\end{aligned}
$$
This sliding-window maximum addresses uncertain position, uncertain frequency, and minor local deviations in generated text [2509.03117].

The decision rule is threshold-based: if and only if the watermark similarity exceeds an empirical threshold, the suspicious application is treated as infringing. The paper does not specify a universal threshold in the body text. Instead, it assesses separation between true and false cases through **True-WS**, **False-WS**, and **MDWS**, where MDWS is the minimum distance between the watermark similarity achieved by protected prompts and that achieved by normal prompts [2509.03117].

Two common misconceptions are explicitly countered by the paper’s design. First, the watermark is **not** a human-readable ownership label; optimized signal marks are typically odd strings such as `educaçãoscrapy`, `WORLDalipicker`, or `Să Więmedad pessi`. Second, content-only verification does **not** eliminate secrecy assumptions: the method assumes the adversary does not know the verification query or signal mark, because leakage of either would make filtering attacks more plausible [2509.03117].

## 5. Empirical evaluation

PromptCOS is evaluated on three representative domains: **BIGBENCH-II** for question answering, **GSM8K** for math reasoning, and **HumanEval** for coding. The evaluated LLMs are **Deepseek-distilled-qwen-1.5b**, **Gemma2-2b-it**, **TinyLlama-1.1b-chat**, **CodeGemma-2B**, and **CodeGemma-7B**. Prompt generation is based on **Automatic Prompt Engineer (APE)** so that protected prompts are automatically produced rather than manually authored, and optimization is carried out over randomly selected prompts from that pool [2509.03117].

The baselines are **PC\***, a PromptCare-style method under logits-available conditions; **PCG**, a content-only PromptCare adaptation; and **PR**, a repetition-based content-only method. Evaluation is organized around **Watermark Similarity (WS)**, **True-WS**, **False-WS**, **MDWS**, **Accuracy degradation**, **BERTScore**, and watermark embedding time [2509.03117].

The main reported results are that PromptCOS achieves **99.3% average watermark similarity**, **60.8% greater distinctiveness than the best baseline**, **accuracy degradation of no more than 0.58%**, robustness against **three types of potential attacks**, and **up to 98.1% reduction in computational cost** [2509.03117]. In the paper’s aggregate comparison, PromptCOS obtains average **True-WS = 0.99**, **False-WS = 0.17**, and **MDWS = 0.74**. The corresponding averages for baselines are **PC\***: True-WS 0.52, False-WS 0.04, MDWS 0.46; **PCG**: True-WS 0.50, False-WS 0.14, MDWS 0.33; and **PR**: True-WS 0.83, False-WS 0.73, MDWS 0.02. The paper interprets this as strong content-only effectiveness with substantially better separation between protected and benign prompts [2509.03117].

The fidelity results are similarly favorable. PromptCOS reports average **Accuracy degradation = -0.13%** and **BERTScore = 0.67**, compared with **PC\*** at 0.54% / 0.70 and **PCG** at 1.94% / 0.50. The negative average degradation indicates that some watermarked prompts slightly improved task performance, which the paper attributes to the fidelity objective behaving like additional prompt tuning. Qualitative examples contrast this with PCG, which can induce wrong tense, multilingual nonsense, mathematical mistakes, or unusable code outputs such as `<|file_separator|>` [2509.03117].

Robustness is evaluated under **redundancy**, **constraint**, and **removal** attacks. Average watermark similarity remains **0.98** under redundancy and **0.99** under constraint, indicating that addition-based attacks have little effect. Under removal, watermark similarity falls to **0.21**, but the corresponding prompt-quality damage is severe, with **Accuracy degradation = 2.89** and **BERTScore = 0.47**. The paper presents this as evidence that cover-token design makes removal costly in utility terms even when it reduces detectability [2509.03117].

Efficiency is reported in watermark embedding time. PromptCOS requires **20.8** minutes on TinyLlama, **11.1** on Deepseek-d-qwen, **36.0** on Gemma2-it, **27.6** on CodeGemma-2B, and **43.2** on CodeGemma-7B. The corresponding PCG times are **281.7**, **253.9**, **461.5**, **1014.2**, and **2351.7** minutes. The largest gain is on CodeGemma-7B, where PromptCOS reduces embedding time from **2351.7** to **43.2** minutes, i.e. about **98.1% reduction** [2509.03117].

Ablations identify the method’s core mechanisms. Removing cyclic signals drops **True-WS** from **1.00** to **0.42** and **MDWS** from **0.79** to **0.14**, showing that repeated mark emission is essential for content-only verification. Removing auxiliary tokens raises **False-WS** to **0.93**, collapses **MDWS** to **0.03**, worsens **AcD** to **0.63**, and lowers **BS** to **0.57**, indicating that auxiliary tokens are central to both distinctiveness and fidelity. Increasing signal strength from 2 to 3 preserves high True-WS but worsens False-WS and fidelity, while increasing auxiliary tokens from 3 to 5 improves False-WS to **0.06**, MDWS to **0.89**, AcD to **-0.38**, and BS to **0.77** [2509.03117].

## 6. Interpretation, limitations, and place in the literature

PromptCOS is significant because it relocates prompt copyright auditing from hidden model internals to ordinary content outputs. In the paper’s framing, this matters because suspicious third-party applications usually expose only generated text, not logits, and because exposing logits would itself increase risks such as model extraction, knowledge stealing, and data leakage. PromptCOS therefore treats prompt ownership verification as a **black-box, content-only** inference problem, rather than a white-box watermark-detection problem [2509.03117].

Its design also reframes what a prompt watermark is. The watermark is not a static string embedded in the prompt text, nor merely a next-token bias. It is a conditional behavior induced by the joint optimization of \(P_w\), \(V\), and \(S_m\): under a secret query, the protected prompt tends to emit a distinctive signal mark. This suggests a broader view of prompt watermarking as **behavioral prompt instrumentation** rather than literal prompt annotation. A plausible implication is that prompt ownership can be operationalized at the level of induced model behavior, even when the protected artifact itself is not directly inspectable [2509.03117].

The paper is also explicit about limitations. Signal marks are usually odd strings and are **not strongly semantically meaningful copyright identifiers**. Verification uses **character-level similarity**, which is simple and effective but limited in expressivity. The method is **model-specific** in practice; cross-backend transfer is not the main target. Threshold selection is empirical rather than theorem-backed. Finally, the robustness story depends on secrecy: if the verification query or signal mark were leaked, stronger adaptive attacks would become more plausible [2509.03117].

Another important limitation is that the robustness guarantee is practical rather than absolute. Removal attacks can indeed reduce watermark similarity, but the method is designed so that successful removal is entangled with a loss of prompt value. That is a weaker claim than invulnerability. It means PromptCOS is best understood as raising the cost of prompt theft and deletion, not as providing cryptographic prompt ownership proofs [2509.03117].

Within the broader prompt-engineering landscape, PromptCOS occupies the protection side of a field otherwise dominated by optimization, refinement, and deployment trade-offs. Its underlying motivation is that prompts are expensive to create because they encode task instructions, style, constraints, and behavior-shaping knowledge, often requiring expert labor, proprietary data, prompt optimization loops, and substantial compute [2509.03117]. PromptCOS therefore complements prompt-design research by treating the resulting system prompt as an asset that can be audited after deployment, under the same black-box conditions in which prompt theft would typically occur [2509.03117].

Source: https://www.emergentmind.com/topics/promptcos