RLCracker: Adaptive RL Watermark Removal

• RLCracker is an adaptive attack framework that uses reinforcement learning to generate semantically faithful paraphrases for watermark removal.
• It introduces the adaptive robustness radius and KL divergence measures to assess how much semantic change is needed to evade watermark detection.
• Empirical results show high removal rates across diverse model sizes and watermark schemes, exposing vulnerabilities in current LLM watermark defenses.

RLCracker is an adaptive attack framework for LLM watermark removal that uses reinforcement learning to paraphrase watermarked text while preserving semantic fidelity. It was introduced in “RLCracker: Exposing the Vulnerability of LLM Watermarks with Adaptive RL Attacks” (Huang et al., 25 Sep 2025). The system is motivated by the claim that many prior watermarking evaluations are not sufficiently adversarial: they often test robustness against fixed paraphrasers or handcrafted prompts, whereas an actual attacker can optimize both the paraphrasing context and the paraphrasing model. Within that threat model, RLCracker trains a paraphrasing policy that remains semantically close to the original text while moving the output distribution away from watermark-bearing text and toward human-like text.

1. Threat model and problem formulation

RLCracker studies the setting in which an attacker receives a watermarked sequence $\mathbf{X}$ and seeks a paraphrase $\mathbf{X}'$ that preserves meaning but is no longer flagged by the watermark detector. The paper assumes that the attacker does not know the secret key $s$, the detector $f$, or the detection threshold $\delta$. The attacker can, however, query the generator, choose a paraphrasing model $\pi_\theta$, tune its parameters $\theta$, and choose any conditioning context $c$ (Huang et al., 25 Sep 2025).

Under this formulation, the attack output is

$\mathbf X' = \pi_\theta(\mathbf X; c).$

An attack succeeds when semantic drift remains small and the detector score falls below threshold: $$d(\mathbf X,\mathbf X') \text{ is small, and } f(\mathbf X', s) \le \delta.$$

The paper’s criticism of prior evaluations is central to the definition of RLCracker. Fixed paraphrasers, static prompts, and average-case evaluation can overstate robustness because they ignore adaptive choices of prompts, system instructions, reasoning strategies, model scale, and multi-sample search. This suggests that watermark security should be analyzed against strategic paraphrasers rather than only against non-adaptive text editing.

2. Adaptive robustness radius

A central contribution of the paper is the formalization of watermark resilience through the adaptive robustness radius. For an input $\mathbf{X}'$0, it is defined as

$\mathbf{X}'$1

The equivalent optimization view is

$\mathbf{X}'$2

Intuitively, the radius measures how much semantic change is required before the watermark can be removed. If the radius is small, then a semantically faithful paraphrase may suffice to evade detection. The paper then introduces a distributional relaxation, the KL adaptive radius,

$\mathbf{X}'$3

and under a sub-Gaussian assumption on the detector score proves

$\mathbf{X}'$4

This yields the certificate

$\mathbf{X}'$5

where $\mathbf{X}'$6 (Huang et al., 25 Sep 2025).

The key theoretical monotonicity claim is that if the attacker’s prompt space and model space grow, robustness can only decrease: $\mathbf{X}'$7 The appendix further connects this to Pass@$\mathbf{X}'$8-style multi-sample attacks, implying that repeated sampling can further increase the chance of finding an undetected paraphrase.

3. RL training objective and attack pipeline

RLCracker operationalizes the adaptive attack through token-wise reinforcement learning. The policy $\mathbf{X}'$9 is trained to preserve meaning, imitate human-like outputs, and move away from watermark-associated token distributions. The paper describes the objective as shifting output distribution $s$0 toward human text $s$1 and away from the watermark distribution $s$2, i.e.

$s$3

The implementation is built on GRPO-style token-wise RL, uses Hugging Face TRL, and trains on question–watermarked response pairs $s$4. It requires zero detector access and uses the model’s original weights as the reference policy $s$5 (Huang et al., 25 Sep 2025).

The training objective combines three components:

1. Semantic reward: based on P-SP between the generated output and the original watermarked text, with a sigmoid transformation. The normalized semantic advantage is

$s$6

2. Token-wise KL reward: encouraging the policy to become more human-like and less watermark-like,

$s$7

3. Fluency regularization: a perplexity penalty preventing degenerate or unnatural outputs.

In simplified form, the objective is

$s$8

A distinctive property of the system is its low data requirement. The main setup trains on only 100 watermarked samples, each around 500 tokens, and ablations also consider short-sample regimes. The paper argues that the policy works not by paraphrasing randomly, but by learning paraphrases that remain semantically faithful while strategically altering the text distribution.

4. Empirical performance

The paper’s headline result is reported on 1,500-token Unigram-marked texts. A Qwen2.5-3B-Instruct RLCracker model achieves 98.5% ESR, 100% removal rate, and 0.92 P-SP. In the same setting, GPT-4o achieves only 6.75% ESR (Huang et al., 25 Sep 2025).

The study reports that RLCracker generalizes across five model sizes and ten watermarking schemes. The evaluated schemes are EWD, KGW, KGW_selfhash, UPV, SWEET, Unigram, SIR, X-SIR, SynthID-Text, and PF-Watermark. The paper emphasizes that long-form outputs are particularly important because many baseline attacks degrade substantially at 1,500 tokens, whereas RLCracker remains effective.

The comparison with baseline paraphrasing strategies is also notable. The paper evaluates Base, Think, SysP, Think + SysP, SIRA, DIPPER, and GPT-4o paraphrasing. The reported pattern is that Base is weak, SysP improves attack success, Think often improves it further, Think + SysP is often stronger still, but RLCracker is best overall.

A specific counterexample clarifies the distinction between raw watermark removal and semantically faithful watermark removal. On 1500-token Unigram, SIRA reaches Rem. = 88%, but only P-SP = 0.47 and ESR = 0.50%. The paper treats this as evidence that removal without semantic preservation is not a satisfactory attack outcome.

5. Ablations, transfer, and failure modes

The ablation results support the paper’s claim that watermark robustness is highly sensitive to attacker capability. System prompts materially affect attack success: they raise ESR, reduce variance across prompts, and often improve P-SP. The paper interprets this as evidence that attack context is itself an adversarial control variable (Huang et al., 25 Sep 2025).

Model scale also matters. On KGW, the paper reports 6.0% ESR for Qwen3-0.6B and 49.3% ESR for GPT-4o. This is used to argue that robustness is not stable as the attacker’s paraphraser becomes stronger. Reasoning and “test-time scaling” often improve evasion success as well, although the gains vary by watermark scheme.

The training-data ablation indicates substantial data efficiency. Even 50 samples can already yield strong ESR, performance improves with 100 samples, and gains plateau after that. RLCracker also remains effective under mixed-key training, with only slight degradation relative to single-key training; the maximum reported ESR drop is around 2%.

The paper reports out-of-distribution transfer from one generator to others: training on LLaMA3.1-8B outputs and testing on Qwen2.5-1.5B and Qwen2.5-32B outputs still yields high ESR and high removal rate. This suggests that the learned attack does not depend only on one generator or one watermark key.

At the same time, the method has a documented sensitivity to the reward weights $s$9, $f$0, and $f$1. Too much emphasis on semantic reward or KL reward can damage either evasion or fidelity. The paper nonetheless reports that performance remains strong when these weights are tuned reasonably.

6. Security implications and relation to adjacent RL-based attack research

The broader conclusion of the paper is that current LLM watermarking defenses are not robust under adaptive attack. Several implications are explicit. Watermarks may be evaluated too optimistically under average-case paraphrasing. Prompt design, system prompts, reasoning, and model scaling reduce robustness. Even a 3B model can be an effective watermark-removal system when trained adversarially. Semantic fidelity does not preserve the watermark signal. Detector-free attacks are practical. Long-form text is especially vulnerable (Huang et al., 25 Sep 2025).

RLCracker also belongs to a broader pattern in LLM security research in which reinforcement learning is used to learn attack policies rather than relying on fixed heuristics. In a different problem domain, RL-JACK formulates jailbreak prompt generation as a search problem and uses deep reinforcement learning with a structured action space and dense reward shaping to attack aligned models in the black-box setting (Chen et al., 2024). This suggests that adaptive RL is becoming a general technique for optimizing text-based attacks against LLM security mechanisms.

Conversely, nearby defense work has moved toward more procedural and adaptive detection strategies. RLM-JB treats jailbreak detection as a bounded analysis program with normalization, de-obfuscation, chunking, parallel chunk screening, and cross-chunk aggregation, rather than as a one-shot classifier (Shavit, 18 Feb 2026). A plausible implication is that watermarking defenses may require similarly adversarial evaluation and procedural hardening if they are to remain meaningful against adaptive paraphrasing attacks.

In that landscape, RLCracker is significant less because it proposes a new watermark and more because it reframes watermark evaluation around the attacker’s optimization problem. Its central claim is that once the attacker can optimize both the paraphrasing context and the paraphrasing policy, robustness can collapse even when semantic preservation remains high.