Signature filtering: a lightweight enhancement for statistical watermark detection in large language models

Abstract: Statistical watermarks help organizations attribute LLM outputs, yet existing detectors often struggle when watermark signals are weak, texts are repetitive, or watermarks are edited. We propose signature filtering, a detection-time module that enhances watermark detection without modifying watermark embedding and text generation. It learns a small set of ``signature'' tokens whose presence makes watermark tests unreliable, and removes these tokens before detection. The signatures are obtained by solving a mixed-integer linear program on a small training set, with constraints that maximize the true positive rate. We additionally derive finite-sample and asymptotic bounds under several attacker models (color-blind, color-adaptive, and distributionally correlated). On four well-known watermark families (Kgw, Sweet, Unigram, Exp), four benchmark corpora (C4, MBPP, HumanEval, Code-Search-Net), and six LLMs (Opt-1.3b, Opt-6.7b, Llama2-13b, Llama3.1-8b, Qwen2.5-14b, Phi-3-medium-14b), 2- and 3-gram signatures raise detection rates in weak-signal and low-entropy settings from 8~31% without filtering to 78~99% with filtering, while keeping false positives controllable and often negligible. In stress tests where we scramble sentences and perturb 25~50% of tokens by dilution, deletions, and substitutions, 2-gram filters for Kgw-style watermarks preserve most of the clean-text detection gains, often matching or outperforming the advanced WinMax watermark detector. Signature filtering thus provides a simple, scalable, and model-agnostic add-on to strengthen watermark-based provenance checks for LLM text in information processing workflows.

• The paper demonstrates that signature filtering increases true positive rates from 8–31% to 78–99% in weak-signal regimes with minimal false positive rate inflation.
• Methodology employs a two-stage process using an MILP formulation to remove disruptive n-gram tokens, thereby enhancing standard statistical tests.
• Empirical results show robust performance across various watermark schemes, LLM architectures, and adversarial text modifications, supporting large-scale, compliance-critical applications.

Signature Filtering for Robust Statistical Watermark Detection in LLMs

Introduction and Problem Context

The paper "Signature filtering: a lightweight enhancement for statistical watermark detection in LLMs" (2606.18430) introduces a detection-time technique—signature filtering—for improving the reliability of statistical watermark detectors in LLMs across weak-signal, repetitive, and adversarially edited texts. Conventional statistical watermark detectors (Kgw, Exp, Unigram, Sweet, etc.) suffer reduced true positive rates (TPR) in these "borderline" scenarios due to necessity to minimize signal bias for text quality, the frequent presence of low-entropy segments, and vulnerability to common text modifications. The motivation is thus to provide a modular enhancement that preserves calibration guarantees while boosting detection power, most crucially under regimes where statistical separation is weak and existing detectors frequently fail.

Methodology: Signature Filtering via MILP

Signature filtering operates as a two-stage module, preprocessing candidate texts via removal of a compact set of token types (the "signature") before applying a standard hypothesis test. The signature is learned offline from a small training set of watermarked and natural outputs. For a given watermarking scheme, tokens that maximally disrupt the test statistic—those whose removal increases the odds of flipping a false negative without increasing the false positive rate (FPR) beyond a controlled budget—are selected via an MILP formulation. In the first stage, the canonical statistical test is applied directly. Only if the result is inconclusive does the system apply the learned signature filter and recompute the score on the surviving evidence.

For Kgw and related families, the signature aims to shift the empirical green fraction over a fixed threshold by excising local clusters that consistently bear red evidence in the watermarked population. For Exp-style detectors, signature design leverages per-token keyed pseudorandom scores, discarding contexts that systematically dilute or counteract the watermark signal.

The optimization searches for the minimal set of (contextualized) $n$-grams (with $n$ as a hyperparameter) whose removal yields maximal TPR improvement with provable control over induced FPR increase. The signature structure can range from unigram (type-level) to higher-order n-gram signatures, with experimental evidence showing sweet spots at 2–3 grams (Kgw/Sweet/Unigram) and 3–4 grams (Exp).

Theoretical Analysis: False Positive Control under Adversarial and Correlated Scenarios

A primary contribution is the theoretical investigation of the FPR implications of applying signature filtering, particularly in non-idealized regimes. For Kgw watermarks, the authors rigorously characterize FPR behavior under (i) color-blind but signature-aware attackers, (ii) color-adaptive attackers (who know the secret key), and general correlated data settings.

Key analytical results include:

• Under the standard coloring model and score secrecy (i.e., deletion rules independent of hidden watermark assignment), the null distribution remains invariant after filtering, with FPR inflation completely characterized by the conditional Type-I error on the post-filter test.
• For color-blind attackers, deletion budgets linear in text length $O(n)$ guarantee FPR is bounded below any operational threshold. For color-adaptive or heavily correlated data, the maximal robust deletion budget shrinks to $O(\sqrt{n})$. Beyond this, even optimal detectors cannot prevent flips in borderline cases.
• For Exp-style schemes, where token evidence is derived from keyed pseudorandom values, the authors rigorously prove that FPR inflation due to signature filtering does not exceed $\alpha / (1 - \alpha)$ for significance level $\alpha$, regardless of deletion budget, provided score secrecy holds. This is optimal and sharply contrasts with the Kgw scenario.

These results are grounded in detailed Berry–Esseen expansions and bivariate normal approximations to the test statistic distributions, underlining the statistical validity of the signature filtering approach when formulated carefully.

Empirical Evaluation

Extensive experimental evaluation spans four watermark families (Kgw, Sweet, Unigram, Exp), six open-source LLMs (including Opt, Llama2/3, Qwen2.5, Phi-3), and diverse corpora (high-entropy natural language and low-entropy code). Experiments include weak-signal (low watermark strength), low-entropy (e.g., code with repetitive patterns), and various edit attacks (scrambling, dilution, synonym/context replacement, deletions).

Key findings:

• In weak-signal regimes, 2–3-gram signatures consistently raise TPR from 8–31% (baseline, no filtering) to 78–99% after filtering with minimal FPR increase (see Figure 1).

Figure 2: TPR (solid, left y-axis) and FPR (dashed, right y-axis, log scale) for varying $n$-gram signature order and different watermark families on Llama3.1-8b, illustrating optimal trade-off order and robust TPR gains.

• These gains persist across LLM architectures and corpora; the optimal n-gram order remains invariant.
• Under severe edit attacks, 2-gram filtering for Kgw-style watermarks preserves the majority of clean-text detection improvement and often matches or beats the window-based WinMax detector (Figure 3). For Exp, while detection is robust to filtering, the adversarial window scan of WinMax can better localize surviving signal, but with increased FPR.

Figure 4: TPR/FPR for signature filtering on low-strength and edited C4 and Mbpp, demonstrating edit resilience relative to windowed detectors.

• Empirical results on scalability show that signature training on as few as 1,000 texts generalizes to corpora with tens of thousands of unseen samples without significant FPR inflation.
• Bagging (ensemble) operational modes—rolling group signatures with max-pooling for the detection score—enable effective streaming deployments under dynamic content distributions, controlling FPR below union bound estimates.

Implications and Limitations

Theoretical and Practical Implications

Signature filtering provides a drop-in, model-agnostic, and statistically grounded detection-time enhancement for watermark pipelines. Given its analysis, its principal practical value is in high-stakes or compliance-driven information workflows where operational constraints on FPR are critical. Given a controlled deletion budget (linear in text length for color-blind regimes, $O(\sqrt{n})$ under worst-case adaptivity), the Type-I error of the overall provenance test remains auditable and interpretable, a property not generally enjoyed by more aggressive detection-side interventions. Its resilience to corpus scale, streaming deployment, and edit attacks, particularly under the bagging strategy, makes it suitable for web-scale LLM text attribution, code provenance systems, and platform content governance.

The robustness of Exp-style detectors to even aggressive content filtering (with the caveat of maintained score secrecy) suggests that future design should emphasize harnessing keyed token-level pseudorandom functions, with signatures acting as a preconditioning step.

Signature filtering does not modify watermark embedding or test statistics. It is thus fully compatible and composable with other provenance signals, including attention-, entropy-, or semantics-aware extensions, and can be extended to multilingual and multimodal settings provided the detection test remains null-calibrated after observed-feature-based unit removal.

Limitations and Future Directions

• The statistical guarantees of the filtering null calibration depend on the independence of the deletion rule from hidden watermark assignments; any leakage weakens this guarantee. If attackers can recover the watermark key (e.g., via MIP-based attacks [reynolds2025breaking]), only minimal robustness is preserved.
• In highly repetitive (low-entropy) or closely correlated deployment corpora, there is an increased risk of FPR inflation due to spurious signature/data alignment. This effect can be mitigated via entropy-aware penalties or majority-vote aggregation over bagged groups.
• For scenarios with scarce representative training data, the TPR gain is attenuated; adaptive group sizing and conservative $n$-gram orders offer a remedy, but further algorithmic work on incremental, data-scarce signature updates is warranted.
• Signature filtering requires key-conditioned retraining on watermark key rotation and incurs higher computational burden for Exp due to full-key dependence.
• The results emphasize the continued necessity for multilayered provenance systems—statistical watermark evidence is only one component in robust governance.

Conclusion

The paper provides a unified detection-time framework for significantly augmenting LLM statistical watermark detection. Through a rigorous blend of optimization-based signature selection, detailed Type-I error analysis under multiple attacker and correlation models, and broad empirical validation, it demonstrates that signature filtering can substantially improve detection recall in all challenging watermark operation regimes of practical interest, while crucially maintaining control of the false positive error. The work offers actionable guidelines for integration: for Kgw, 2-gram signatures deliver the most favorable TPR/FPR trade-off; for Exp, higher-order context is beneficial, but score secrecy underpins robustness; bagging is recommended for streaming scenarios.

The approach's theoretical underpinnings, implementation transparency, and robustness across families and attacks position signature filtering as a principled and scalable addition for AI-generated content provenance pipelines, with evident extensions to cross-lingual and multimodal attribution, and avenues for hybrid signal fusion with emerging LLM authenticity mechanisms.