---
title: 'RLCracker: Adaptive RL Watermark Removal'
url: https://www.emergentmind.com/topics/rlcracker
type: topic
---

# RLCracker: Adaptive RL Watermark Removal

RLCracker is an adaptive attack framework for large language model watermark removal that uses reinforcement learning to paraphrase watermarked text while preserving semantic fidelity. It was introduced in “RLCracker: Exposing the Vulnerability of LLM Watermarks with Adaptive RL Attacks” [2509.20924]. The system is motivated by the claim that many prior watermarking evaluations are not sufficiently adversarial: they often test robustness against fixed paraphrasers or handcrafted prompts, whereas an actual attacker can optimize both the paraphrasing context and the paraphrasing model. Within that threat model, RLCracker trains a paraphrasing policy that remains semantically close to the original text while moving the output distribution away from watermark-bearing text and toward human-like text.

## 1. Threat model and problem formulation

RLCracker studies the setting in which an attacker receives a watermarked sequence \(\mathbf{X}\) and seeks a paraphrase \(\mathbf{X}'\) that preserves meaning but is no longer flagged by the watermark detector. The paper assumes that the attacker does not know the secret key \(s\), the detector \(f\), or the detection threshold \(\delta\). The attacker can, however, query the generator, choose a paraphrasing model \(\pi_\theta\), tune its parameters \(\theta\), and choose any conditioning context \(c\) [2509.20924].

Under this formulation, the attack output is
\[
\mathbf X' = \pi_\theta(\mathbf X; c).
\]
An attack succeeds when semantic drift remains small and the detector score falls below threshold:
\[
d(\mathbf X,\mathbf X') \text{ is small, and } f(\mathbf X', s) \le \delta.
\]

The paper’s criticism of prior evaluations is central to the definition of RLCracker. Fixed paraphrasers, static prompts, and average-case evaluation can overstate robustness because they ignore adaptive choices of prompts, system instructions, reasoning strategies, model scale, and multi-sample search. This suggests that watermark security should be analyzed against strategic paraphrasers rather than only against non-adaptive text editing.

## 2. Adaptive robustness radius

A central contribution of the paper is the formalization of watermark resilience through the **adaptive robustness radius**. For an input \(\mathbf{X}\), it is defined as
\[
r^\star(\mathbf{X}) := \sup \left\{ \rho \ge 0 \mid \forall\, \theta \in \Theta,\; c \in \mathcal{C},\; \mathbf{X}' = \pi_\theta(\mathbf{X}; c),\; d(\mathbf{X}, \mathbf{X}') \le \rho \Rightarrow f(\mathbf{X}', s) > \delta \right\}.
\]
The equivalent optimization view is
\[
r^\star(\mathbf{X}) = \min_{\theta \in \Theta,\; c \in \mathcal{C}} d(\mathbf{X}, \pi_\theta(\mathbf{X}; c)) \quad \text{subject to} \quad f(\pi_\theta(\mathbf{X}; c), s) \le \delta.
\]

Intuitively, the radius measures how much semantic change is required before the watermark can be removed. If the radius is small, then a semantically faithful paraphrase may suffice to evade detection. The paper then introduces a distributional relaxation, the **KL adaptive radius**,
\[
r_{\mathrm{KL}(s,c,\bm\theta)} := \sup\Big\{\rho \ge 0 \mid \forall Q \text{ s.t. } D_{\mathrm{KL}}(Q\|P_{s,c,\bm\theta}) \le \rho \Rightarrow \mathbb{E}_Q[f(\mathbf{X},s)] > \delta\Big\},
\]
and under a sub-Gaussian assumption on the detector score proves
\[
\mathbb{E}_Q[f(\mathbf X,s)] \ge \mu(s,c,\bm\theta) - \sqrt{2\,\sigma^2(s,c,\bm\theta)\,D_{\mathrm{KL}}(Q\|P_{s,c,\bm\theta})}.
\]
This yields the certificate
\[
r_{\mathrm{KL}(s,c,\bm\theta)} \ge \rho^\star(s,c,\bm\theta) := \frac{\left(\mu(s,c,\bm\theta)-\delta\right)_+^2}{2\,\sigma^2(s,c,\bm\theta)},
\]
where \((x)_+ = \max\{x,0\}\) [2509.20924].

The key theoretical monotonicity claim is that if the attacker’s prompt space and model space grow, robustness can only decrease:
\[
R_{\mathrm{KL}(s;\mathcal{C}',\bm\Theta')} \le R_{\mathrm{KL}(s;\mathcal{C},\bm\Theta)}.
\]
The appendix further connects this to Pass@\(k\)-style multi-sample attacks, implying that repeated sampling can further increase the chance of finding an undetected paraphrase.

## 3. RL training objective and attack pipeline

RLCracker operationalizes the adaptive attack through token-wise reinforcement learning. The policy \(\pi_\theta\) is trained to preserve meaning, imitate human-like outputs, and move away from watermark-associated token distributions. The paper describes the objective as shifting output distribution \(Q\) toward human text \(P_h\) and away from the watermark distribution \(P_{wm}\), i.e.
\[
\max_{\theta \in \Theta} (Q\|P_{wm}) - (Q\|P_h).
\]

The implementation is built on **GRPO-style token-wise RL**, uses **Hugging Face TRL**, and trains on question–watermarked response pairs \((q, wr)\). It requires **zero detector access** and uses the model’s original weights as the reference policy \(\pi_{\mathrm{ref}}\) [2509.20924].

The training objective combines three components:

1. **Semantic reward**: based on **P-SP** between the generated output and the original watermarked text, with a sigmoid transformation. The normalized semantic advantage is
   \[
   \hat A_i = \frac{A_i - \mathrm{mean}(A)}{\mathrm{std}(A)}.
   \]

2. **Token-wise KL reward**: encouraging the policy to become more human-like and less watermark-like,
   \[
   \Delta_i = (Q\|P_{wm}) - (Q\|P_h).
   \]

3. **Fluency regularization**: a perplexity penalty preventing degenerate or unnatural outputs.

In simplified form, the objective is
\[
\mathcal{J}(\theta) \approx \mathbb{E}\left[ \text{token-wise policy ratio} \cdot (w_1 \hat A_i + w_2 \Delta_i) - \beta [\pi_\theta \| \pi_{\mathrm{ref}}] \right] - w_3\,\mathrm{PPL}(\pi_\theta,\{o_i\}).
\]

A distinctive property of the system is its low data requirement. The main setup trains on only **100 watermarked samples**, each around **500 tokens**, and ablations also consider short-sample regimes. The paper argues that the policy works not by paraphrasing randomly, but by learning paraphrases that remain semantically faithful while strategically altering the text distribution.

## 4. Empirical performance

The paper’s headline result is reported on **1,500-token Unigram-marked texts**. A **Qwen2.5-3B-Instruct** RLCracker model achieves **98.5% ESR**, **100% removal rate**, and **0.92 P-SP**. In the same setting, **GPT-4o achieves only 6.75% ESR** [2509.20924].

The study reports that RLCracker generalizes across **five model sizes** and **ten watermarking schemes**. The evaluated schemes are **EWD**, **KGW**, **KGW\_selfhash**, **UPV**, **SWEET**, **Unigram**, **SIR**, **X-SIR**, **SynthID-Text**, and **PF-Watermark**. The paper emphasizes that long-form outputs are particularly important because many baseline attacks degrade substantially at **1,500 tokens**, whereas RLCracker remains effective.

The comparison with baseline paraphrasing strategies is also notable. The paper evaluates **Base**, **Think**, **SysP**, **Think + SysP**, **SIRA**, **DIPPER**, and **GPT-4o** paraphrasing. The reported pattern is that **Base** is weak, **SysP** improves attack success, **Think** often improves it further, **Think + SysP** is often stronger still, but **RLCracker** is best overall.

A specific counterexample clarifies the distinction between raw watermark removal and semantically faithful watermark removal. On **1500-token Unigram**, **SIRA** reaches **Rem. = 88%**, but only **P-SP = 0.47** and **ESR = 0.50%**. The paper treats this as evidence that removal without semantic preservation is not a satisfactory attack outcome.

## 5. Ablations, transfer, and failure modes

The ablation results support the paper’s claim that watermark robustness is highly sensitive to attacker capability. System prompts materially affect attack success: they raise ESR, reduce variance across prompts, and often improve P-SP. The paper interprets this as evidence that attack context is itself an adversarial control variable [2509.20924].

Model scale also matters. On **KGW**, the paper reports **6.0% ESR** for **Qwen3-0.6B** and **49.3% ESR** for **GPT-4o**. This is used to argue that robustness is not stable as the attacker’s paraphraser becomes stronger. Reasoning and “test-time scaling” often improve evasion success as well, although the gains vary by watermark scheme.

The training-data ablation indicates substantial data efficiency. Even **50 samples** can already yield strong ESR, performance improves with **100 samples**, and gains **plateau** after that. RLCracker also remains effective under **mixed-key** training, with only slight degradation relative to single-key training; the maximum reported ESR drop is around **2%**.

The paper reports out-of-distribution transfer from one generator to others: training on **LLaMA3.1-8B** outputs and testing on **Qwen2.5-1.5B** and **Qwen2.5-32B** outputs still yields high ESR and high removal rate. This suggests that the learned attack does not depend only on one generator or one watermark key.

At the same time, the method has a documented sensitivity to the reward weights \(w_1\), \(w_2\), and \(w_3\). Too much emphasis on semantic reward or KL reward can damage either evasion or fidelity. The paper nonetheless reports that performance remains strong when these weights are tuned reasonably.

## 6. Security implications and relation to adjacent RL-based attack research

The broader conclusion of the paper is that current LLM watermarking defenses are not robust under adaptive attack. Several implications are explicit. Watermarks may be evaluated too optimistically under average-case paraphrasing. Prompt design, system prompts, reasoning, and model scaling reduce robustness. Even a **3B model** can be an effective watermark-removal system when trained adversarially. Semantic fidelity does not preserve the watermark signal. Detector-free attacks are practical. Long-form text is especially vulnerable [2509.20924].

RLCracker also belongs to a broader pattern in LLM security research in which reinforcement learning is used to learn attack policies rather than relying on fixed heuristics. In a different problem domain, **RL-JACK** formulates jailbreak prompt generation as a search problem and uses deep reinforcement learning with a structured action space and dense reward shaping to attack aligned models in the black-box setting [2406.08725]. This suggests that adaptive RL is becoming a general technique for optimizing text-based attacks against LLM security mechanisms.

Conversely, nearby defense work has moved toward more procedural and adaptive detection strategies. **RLM-JB** treats jailbreak detection as a bounded analysis program with normalization, de-obfuscation, chunking, parallel chunk screening, and cross-chunk aggregation, rather than as a one-shot classifier [2602.16520]. A plausible implication is that watermarking defenses may require similarly adversarial evaluation and procedural hardening if they are to remain meaningful against adaptive paraphrasing attacks.

In that landscape, RLCracker is significant less because it proposes a new watermark and more because it reframes watermark evaluation around the attacker’s optimization problem. Its central claim is that once the attacker can optimize both the paraphrasing context and the paraphrasing policy, robustness can collapse even when semantic preservation remains high.

Source: https://www.emergentmind.com/topics/rlcracker