---
title: 'ForgetMark: Fingerprinting via Targeted Unlearning'
url: https://www.emergentmind.com/topics/forgetmark
type: topic
---

# ForgetMark: Fingerprinting via Targeted Unlearning

Searching arXiv for the primary paper and closely related watermarking/fingerprinting work.
ForgetMark is a fingerprinting framework for language models that encodes provenance through targeted unlearning rather than through a fixed trigger–response backdoor. Introduced in "ForgetMark: Stealthy Fingerprint Embedding via Targeted Unlearning in Language Models" [2601.08189], it constructs a compact set of human-readable Key$\to$Value pairs, identifies those pairs on which the base model is maximally confident, and then trains lightweight LoRA adapters to suppress the original values on their keys while preserving general capabilities. Ownership is subsequently verified from a *probabilistic forgetting trace*: for the selected pairs, the fingerprinted model assigns very low likelihood to the original values or generates semantically different completions, under gray-box or black-box access.

## 1. Conceptual position within model provenance

ForgetMark is explicitly framed against traditional invasive fingerprinting based on backdoors. In the backdoor formulation summarized for the method, a rare trigger sequence $T$ is implanted so that $M(T)$ deterministically yields a response $R$. The reported weaknesses of that design are threefold: the triggers are high-perplexity and thus easy to filter, the responses are fixed and therefore exposed by heuristic detectors, and the triggers can spuriously activate on benign inputs [2601.08189]. ForgetMark replaces this template with selective forgetting of naturally phrased prompts that the base model originally answers with high confidence.

The mechanism is therefore not response synthesis but response suppression. A small set of Key$\to$Value pairs is first chosen so that the target model already exhibits low-variance, high-likelihood behavior on them. A lightweight adaptation step then reduces $P(v\mid k)$ for those pairs while leaving the rest of the distribution largely intact. This makes provenance detectable not by observing an anomalous answer pattern, but by measuring whether a suspect model has retained a stable negative trace on a carefully chosen prompt set.

This design suggests a different stealth regime from classical trigger-based fingerprints. The trigger surface is human-readable rather than synthetically rare, and the ownership signal is probabilistic rather than deterministic. A plausible implication is that detection methods tuned to high-perplexity triggers or fixed outputs are structurally mismatched to the signal that ForgetMark embeds.

## 2. Construction of the fingerprint set

ForgetMark begins with a two-stage selection procedure for the fingerprint set. An auxiliary assistant model, specified as GPT-4, is prompted to generate 500 human-readable, single-turn prompts of length 10–40 tokens that elicit specific factual answers. After manual filtering for safety and diversity, these form the candidate key set
$$
K=\{k_1,\dots,k_K\}, \qquad K=500.
$$

For each candidate key $k_i$, the target model $M_{\text{target}}$ is queried to sample $M$ independent continuations,
$$
\{v_{i,1},\dots,v_{i,M}\},
$$
with $M=3$ in the main experiments, while recording token-level generation probabilities $p_{i,j}^{(t)}$. ForgetMark then ranks keys by a predictive-entropy proxy
$$
U_i=-\,\frac{1}{M}\sum_{j=1}^M\sum_{t=1}^{|v_{i,j}|}\log p_{i,j}^{(t)}.
$$
Small $U_i$ indicates that the model is highly confident and low-variance on key $k_i$. Keys are sorted by ascending $U_i$, and the top $N$ are retained, with $N=100$ in the main configuration. For each selected key, the retained value is the sampled continuation with highest total likelihood,
$$
v_i=\arg\min_j \Bigl[-\sum_t \log p_{i,j}^{(t)}\Bigr].
$$
The resulting fingerprint set is
$$
F=\{(k_i,v_i)\}_{i=1}^N.
$$

The reported ablation makes the role of this uncertainty-driven selection explicit. On non-unlearned Qwen2.5 and LLaMA3 models, random selection yields average $P(v\mid k)=0.091$ and ROUGE-L $=0.069$, whereas uncertainty-driven selection yields $P(v\mid k)=0.662$ and ROUGE-L $=0.432$ [2601.08189]. This indicates that entropy-based selection concentrates the fingerprint on behaviors that are already stable in the base model, thereby enlarging the pre/post-unlearning gap and reducing false alarms on untouched models.

## 3. Targeted unlearning as the embedding mechanism

The embedding stage freezes the base parameters $\theta$ and inserts a low-rank adapter
$$
W_{\text{lora}}=AB^\top,\qquad A\in\mathbb{R}^{d\times r},\; B\in\mathbb{R}^{d\times r},
$$
so that the adapted model defines the next-token distribution
$$
p(\cdot\mid\cdot;\theta+W_{\text{lora}}).
$$
The training objective is a signed-likelihood loss that simultaneously increases the NLL on the fingerprint pairs and decreases the NLL on a retention set:
$$
L(A,B)
= \gamma\,\mathbb{E}_{(k,v)\in F}\bigl[\log p(v\mid k;\theta+W_{\text{lora}})\bigr]
- \alpha\,\mathbb{E}_{(x,y)\in D_{\text{retain}}}\bigl[\log p(y\mid x;\theta+W_{\text{lora}})\bigr].
$$
In the reported setup, $\gamma=\alpha=1$.

The retention set $D_{\text{retain}}$ is drawn from Alpaca and sized so that $|D_{\text{retain}}|:|F|=9:1$. The optimization updates only $A$ and $B$, with reported hyperparameters including rank $r=8$, AdamW, learning rate $\text{lr}=10^{-4}$, and approximately $10\,\text{k}$ training steps [2601.08189]. The operational intent is selective forgetting: the model should sharply suppress the original values $v_i$ on keys $k_i$ while maintaining general utility on ordinary instructions.

This formulation distinguishes ForgetMark from fingerprints that directly encode a recognizably artificial behavior. The provenance signal is stored as a localized likelihood depression over naturally phrased prompts. That choice also explains why the framework emphasizes retention data: without an explicit counterweight, an unlearning-based fingerprint could collapse into general degradation rather than targeted suppression.

## 4. Probabilistic forgetting trace and verification

After adaptation, ForgetMark defines ownership through what it calls a *probabilistic forgetting trace*. For every fingerprint pair $(k_i,v_i)\in F$, the adapted model is expected to assign dramatically reduced probability to the original continuation:
$$
\forall (k_i,v_i)\in F,\quad P(v_i\mid k_i)\ \text{is dramatically suppressed}.
$$

Verification is supported under two access regimes. In the gray-box protocol, the verifier queries each key $k_i$ and directly reads out $P_{M_S}(v_i\mid k_i)$ from the suspect model $M_S$. In the black-box protocol, the verifier samples a continuation $\hat v_i$ from $M_S(k_i)$ and computes $\mathrm{ROUGE\mbox{-}L}(\hat v_i,v_i)$. These signals are aggregated into the Fingerprint Success Rate:
$$
\mathrm{FSR}
= \frac1N \sum_{i=1}^N
\mathbf{1}\bigl[P_{M_S}(v_i\mid k_i)<\tau_{\text{prb}}
\;\lor\;
\mathrm{ROUGE\mbox{-}L}(\hat v_i,v_i)<\tau_{\text{rg}}
\bigr].
$$
The reported thresholds are $\tau_{\text{prb}}=10^{-3}$ and $\tau_{\text{rg}}=10^{-3}$, calibrated to yield approximately zero false positives. Ownership is declared if $\mathrm{FSR}\ge \theta$, with $\theta=90\%$ given as an example [2601.08189].

The gray-box and black-box variants are also reported separately as $\mathrm{FSR}_{\text{prb}}$ and $\mathrm{FSR}_{\text{rouge}}$. The central point is that verification does not require the suspect model to emit a specific string. It is sufficient that the model consistently fails, in either likelihood or semantics, to reproduce the original high-confidence continuation on the selected keys.

## 5. Reported empirical properties

The main experiments use Mistral-7B-v0.3, LLaMA-3-8B, and Qwen2.5-7B as target models; GPT-4 as the assistant model; $K=500$ candidate keys; $M=3$ continuations per key; $N=100$ final fingerprint pairs; and an Alpaca retention set of size $9N=900$ [2601.08189]. Evaluation spans effectiveness, harmlessness, input-level stealth, output stealth, and robustness under model merging and incremental fine-tuning.

| Aspect | ForgetMark result | Context |
|---|---:|---|
| Clean fingerprinted models | $\mathrm{FSR}_{\text{prb}}=100\%$, $\mathrm{FSR}_{\text{rouge}}=100\%$ | Compared with IF-SFT and Chain·Hash |
| Qwen2.5-7B harmlessness | $0.631$ | Original model: $0.640$ |
| Input stealth, GPT-2 PPL | $55.6$ | IF-SFT: $245.13$, Chain·Hash: $168.21$ |
| Input stealth, LLaMA3 estimator PPL | $26.27$ | IF-SFT: $1048.00$, Chain·Hash: $86.31$ |
| Output stealth | Token-Forcing detection rate $0\%$ | IF-SFT: $100\%$, Chain·Hash: $50\%$ |

On clean fingerprinted models, ForgetMark reports $\mathrm{FSR}_{\text{prb}}=100\%$ and $\mathrm{FSR}_{\text{rouge}}=100\%$, matching IF-SFT on raw success rate and exceeding Chain·Hash, which reports $90\%$ on both metrics. On Qwen2.5-7B, the zero-shot accuracy averaged over 18 benchmarks declines from $0.640$ for the original model to $0.631$ after ForgetMark embedding. The benchmarks listed are ANLI, ARC, OpenBookQA, Winogrande, QC, BoolQ, CB, RTE, WiC, WSC, CoPA, MultiRC, and LAMBADA within the reported 18-benchmark average [2601.08189].

The stealth metrics are particularly central to the method’s positioning. Input-level stealth, measured by external perplexity estimators, yields $55.6$ under GPT-2 and $26.27$ under LLaMA3-Instruct, compared with IF-SFT at $245.13$ and $1048.00$, and Chain·Hash at $168.21$ and $86.31$. Output-level stealth, measured by Token-Forcing detection, is reported as $0\%$ for ForgetMark, compared with $100\%$ for IF-SFT and $50\%$ for Chain·Hash. This supports the claim that the forgetting trace is substantially less exposed to detector heuristics aimed at trigger-based fingerprints.

Robustness under model merging is reported with MergeKit under the Task merge strategy. For $\alpha:(1-\alpha)=0.9:0.1$, $0.7:0.3$, and $0.5:0.5$, ForgetMark retains $\mathrm{FSR}_{\text{prb}}=100\%$; at $0.3:0.7$ it reports $89\%$; and at $0.1:0.9$ it reports $2\%$. The accompanying summary states that ForgetMark sustains near-perfect FSR down to $\alpha\approx 0.3$ [2601.08189]. Under incremental fine-tuning on UltraChat using Mistral-7B, the reported trajectory is: at step $0$, $(\mathrm{FSR}_{\text{prb}},\mathrm{FSR}_{\text{rouge}})=(1.00,1.00)$; at $1\text{k}$, $(0.87,0.82)$; at $5\text{k}$, $(0.61,0.55)$; and at $20\text{k}$, $(0.23,0.18)$.

A second ablation studies the key-set size $N\in\{25,50,100,200\}$ on Qwen2.5. The reported values are:
- $N=25$: harmlessness $0.640$, average $P(v\mid k)=1.3\times 10^{-5}$, ROUGE-L $=0.0057$
- $N=50$: harmlessness $0.634$, average $P(v\mid k)=3.3\times 10^{-12}$, ROUGE-L $=0.0069$
- $N=100$: harmlessness $0.631$, average $P(v\mid k)=1.2\times 10^{-17}$, ROUGE-L $=0.0000$
- $N=200$: harmlessness $0.593$, average $P(v\mid k)=2.6\times 10^{-43}$, ROUGE-L $=0.0000$

The authors’ interpretation is that $N=100$ offers a clear verification margin with only negligible utility drop. This suggests that the framework’s operating point depends not only on adapter capacity but also on the concentration of the selected behavioral manifold.

## 6. Relation to watermarking, removal, and terminological ambiguity

ForgetMark belongs to the broader provenance literature for generative models, but it occupies a distinct niche. In the language-model setting, it differs from statistical watermarking schemes such as the green-token/red-token formulation associated with "A Watermark for Large Language Models" [2301.10226], and it also differs from recovery or stealing attacks such as "Watermark Stealing in Large Language Models" [2402.19361]. A particularly direct contrast is "De-mark: Watermark Removal in Large Language Models" [2410.13808], which targets n-gram watermarks by estimating the red list, green list, watermark strength, and prefix length through random selection probing, then inverting the green-token boost to form a removal distribution. ForgetMark does not remove a watermark; it embeds ownership via targeted unlearning.

There is also a terminological ambiguity in contemporaneous image-provenance literature. In the MarkCleaner description, “ForgetMark” is used as a summary label for a watermark-removal capability that exploits micro-geometric perturbations to break the phase alignment of semantic watermarks [2602.01513]. In the MarkSweep description, “ForgetMark” similarly denotes the capability to erase invisible image watermarks through edge-aware noise intensification and frequency-aware denoising [2602.15364]. These usages are conceptually separate from the arXiv work titled "ForgetMark" [2601.08189]. One concerns fingerprint embedding in LMs; the others concern watermark erasure in AI-generated images.

The principal limitations reported for ForgetMark are also different from those in watermark-removal systems. Its trace degrades under sufficient downstream fine-tuning, as shown by the UltraChat experiment, and its durability under extreme merge ratios is finite. The stated future directions are anti-recovery regularizers and rotating or refreshable fingerprints [2601.08189]. This suggests that ForgetMark should be understood less as an immutable watermark and more as a stealth-oriented ownership signal whose persistence depends on post-deployment adaptation.

Source: https://www.emergentmind.com/topics/forgetmark