Watermark-Based Dataset Inference

• Watermark-based dataset inference is a method that embeds detectable statistical signals into datasets to validate unauthorized use.
• It employs a watermarking rule, black-box verification protocols, and hypothesis tests to ascertain if a dataset was misappropriated.
• The approach spans multiple modalities — images, text, audio, graphs, and code — with empirical studies demonstrating high success rates and robustness.

Watermark-based dataset inference denotes a family of methods that embed or detect statistically testable signals in datasets so that later black-box interaction with a suspicious model or system can support a dataset-level decision about unauthorized use. A common formulation treats copyright protection as verifying whether a released dataset was adopted for training a third-party model by querying only the deployed model and testing for a learned watermark-induced behavior; adjacent formulations detect watermarked content directly inside reference datasets, suspicious tabular releases, or retrieval corpora (Li et al., 2022, Pan et al., 2024, Zheng et al., 2024, Jovanović et al., 2024). Across the literature, the core ingredients are a watermarking rule, a black-box verification protocol, and a statistical decision rule with controlled significance.

1. Problem formulations and threat models

In the standard discriminative setting, a defender releases a labeled dataset $D = \{(x_i, y_i)\}_{i=1}^n$ for a classification task with $K$ classes and later wants to verify whether an unknown third-party model $S$ was trained using $D$. The defender has only black-box query access to $S$, with two API regimes considered repeatedly: a probability-available regime that returns posterior probabilities over classes, and a label-only regime that returns only the predicted class label. Successful verification is defined as rejecting a null hypothesis of “no watermark” at significance level $\alpha$ by querying the suspicious model on crafted inputs (Li et al., 2022).

This model-level formulation is explicitly distinguished from membership inference and property inference. In the backdoor-watermark literature, membership inference is described as proving inclusion of specific samples rather than usage of the dataset for training a model, whereas watermark-based dataset inference asserts the use of the dataset by detecting a learned watermark. The same distinction is made for point clouds, where watermarking is characterized as inducing model-level behaviors that can be tested via queries without access to training data or weights (Li et al., 2022, Wei et al., 2024).

The same problem has been generalized beyond standard classifiers. In RAG Dataset Inference, the objective is to decide whether a data owner’s dataset $D_{do}$ was incorporated into the provider’s retrieval corpus $D$ using only black-box queries to the deployed RAG system $M^*$ and the textual responses it returns (Jovanović et al., 2024). In generative language modeling, the corresponding formulation asks whether a protected corpus was included in training by detecting residual, key-specific output bias—termed “radioactivity”—in next-token predictions (Kirchenbauer et al., 1 Jul 2026).

A separate branch studies watermark presence directly at the dataset level rather than via a trained model. WaterMark Detection receives a possibly mixed reference dataset $D_{ref}$ and a clean dataset $K$0 from a similar visual distribution and seeks to infer whether $K$1 contains invisible watermarks, identify watermarked samples, and estimate the mixing proportion, all in a black-box and annotation-free setting (Pan et al., 2024). TabularMark analogously treats ownership inference for tabular data as detection of a private-keyed statistical pattern in a suspicious dataset $K$2 relative to an original dataset $K$3 using a one-proportion $K$4-test over secret “key” cells (Zheng et al., 2024).

2. Watermark construction mechanisms

The earliest image-classification formulation used poisoning-based backdoor attacks to construct a protected dataset. A trigger $K$5 is added to a subset of benign samples, and all triggered samples are relabeled to a predefined target class $K$6. The 2020 image-only method instantiated this with a 3×3 white square or a 3-pixel-wide black line, used a watermarking rate $K$7, fixed $K$8, and defined the watermarked image by

$K$9

with $S$0 on trigger pixels in blended experiments (Li et al., 2020).

DVBW generalized this idea into a multimodal framework. Given a trigger $S$1, a target class $S$2, and poisoning ratio $S$3, it selects a subset $S$4 of size $S$5, constructs

$S$6

and releases

$S$7

For images, $S$8 can be a BadNets visible stamp $S$9 or a blended trigger $D$0; for NLP, $D$1 inserts a rare word or a short sentence; for graphs, $D$2 injects a complete subgraph into a selected node (Li et al., 2022).

Clean-label methods replace relabeling with feature manipulation while keeping labels unchanged. “Did You Train on My Dataset?” uses imperceptible perturbations to disable normal discriminative features and then adds a stealthy trigger so that the model must rely on the trigger to fit the original label. For images and audio, perturbations are generated via PGD under an $D$3 bound; for text, synonym replacement and mild insertions are chosen to maximize loss while keeping sentence-embedding cosine similarity high (Tang et al., 2023). Domain Watermark moves further away from trigger-induced misclassification: it constructs a hardly-generalized domain $D$4 via a bi-level optimization over a transformation module $D$5, then optimizes clean-label perturbations $D$6 with $D$7 so that models trained on the protected dataset correctly classify hard-domain samples that benign models typically misclassify (Guo et al., 2023).

Later clean-label designs emphasized stealth and removal resistance. SSCL-BW trains a U-Net-based generator $D$8 to produce a sample-specific watermark $D$9 with a composite loss

$S$0

combining target-sample, non-target-sample, and perceptual similarity terms under an $S$1 constraint (Wang et al., 30 Oct 2025). PointNCBW introduces a negative clean-label watermark for point clouds: non-target samples are perturbed in shape-wise and point-wise manners, then a small sphere trigger is inserted without changing labels, so the trained model learns the trigger as a signal to deny predicting the target class (Wei et al., 2024).

3. Statistical verification and sample complexity

Most model-side schemes reduce ownership verification to a black-box hypothesis test. In DVBW’s probability-available case, let $S$2 denote the suspicious model’s posterior for the target class on a clean input $S$3, and let $S$4 denote the posterior on the triggered counterpart. The test uses

$S$5

a one-tailed paired $S$6-test on $S$7, and the confidence score

$S$8

In the label-only case, DVBW uses a Wilcoxon test on whether $S$9, with the recommendation to choose $\alpha$0 near $\alpha$1 for stable ranks when WSR is not extremely high (Li et al., 2022).

DVBW also provides a sufficient condition for rejection at significance $\alpha$2: $\alpha$3 which yields the sample-size guideline

$\alpha$4

The interpretation given in the paper is direct: higher watermark success rate $\alpha$5, smaller benign target posterior bound $\alpha$6, and larger $\alpha$7 improve power (Li et al., 2022).

Other schemes preserve the same hypothesis-testing structure while changing the measured behavior. Domain Watermark compares the ground-truth posterior on source samples $\alpha$8 and their domain-watermarked counterparts $\alpha$9, testing $D_{do}$0 versus $D_{do}$1. Its sufficient condition for successful verification is

$D_{do}$2

where $D_{do}$3 is the empirical verification success rate on the hard domain (Guo et al., 2023). PointNCBW inverts the direction because its negative watermark suppresses the target-class posterior: it tests $D_{do}$4 versus $D_{do}$5 on triggered target-class inputs (Wei et al., 2024).

Generative and retrieval settings commonly replace paired posterior shifts with token-count statistics. Ward applies the red–green watermark detector to a response sequence $D_{do}$6 of length $D_{do}$7,

$D_{do}$8

uses the one-sided $D_{do}$9-value $D$0, and rejects $D$1 when $D$2. When responses are aggregated, duplicate $D$3-grams are discarded to preserve the independence assumptions behind the normal approximation, and the detectable shift threshold decreases as $D$4 grows (Jovanović et al., 2024).

4. Modality-specific instantiations and empirical behavior

The empirical literature is strongly multimodal. DVBW reports that watermarking “consistently achieves high WSR while preserving BA with small decreases (<~1–2%).” On CIFAR-10 with ResNet-18, BadNets Line gives BA $D$5 and WSR $D$6, while Blended Cross gives BA $D$7 and WSR $D$8. On ImageNet-200 with ResNet-18, BadNets Cross gives BA $D$9, WSR $M^*$0, whereas Blended Line gives BA $M^*$1, WSR $M^*$2. For IMDB and DBpedia, LSTM and WordCNN models achieve WSR $M^*$3 or $M^*$4, and for COLLAB and REDDIT-MULTI-5K, GIN and GraphSAGE achieve WSR $M^*$5 or $M^*$6 (Li et al., 2022).

Clean-label watermarking was extended across text, image, and audio in 2023. With 1% of class-$M^*$7 watermarking samples, the reported watermark detection rate is 100% on text and audio, 97.6% on CIFAR-10, and 100% on CIFAR-10 at 5% and above; text grammar error rate increases by less than 0.5%, and image/audio benign accuracy drops remain below 1% even at higher class-conditional injection rates (Tang et al., 2023). SSCL-BW later reported CIFAR-10 BA $M^*$8, WSR $M^*$9, LPIPS $D_{ref}$0; Sub-ImageNet BA $D_{ref}$1, WSR $D_{ref}$2, LPIPS $D_{ref}$3; and MNIST BA $D_{ref}$4, WSR $D_{ref}$5, LPIPS $D_{ref}$6, with $D_{ref}$7 in its Ind-W and Ind-M baselines and extremely small $D_{ref}$8-values in malicious scenarios (Wang et al., 30 Oct 2025).

Point clouds required a different construction because prior clean-label schemes scaled poorly with the number of classes. PointNCBW watermarks samples across all non-target classes, reports on ModelNet40 ACC $D_{ref}$9 and WSR $K$00 for PointNet, ACC $K$01 and WSR $K$02 for PointNet++, and IoM $K$03; on ShapeNetPart it reports ACC $K$04 with WSR $K$05 or $K$06 (Wei et al., 2024).

Text-specific ownership verification also developed in parallel. TextMarker uses character-level, word-level, and sentence-level triggers with very low marking fractions. In one-user setups, IMDB requires char 0.7%, word 0.1%, sentence 0.1%; SST-5 requires char 0.7%, word 0.3%, sentence 0.1%; and Tweet Emotion requires char 3.0%, word 0.3%, sentence 0.5%. The paper reports that “marking only 0.1% of the training dataset is practically sufficient for effective membership inference with negligible effect on model utility,” and gives IMDB ASR values of 94.76% for character triggers, 78.2% for word triggers, and 98.8% for sentence triggers (Liu et al., 2023).

Code introduces another extension. DuCodeMark parses code into ASTs, applies style-aware trigger–target construction, and adds repressible poisoned features to a subset of return-typed samples. Across 72 settings spanning two code tasks, two languages, three CodeLMs, and six decoding temperatures, it reports $K$07, suspicious rate $K$08, recall $K$09, and a Pass@1 drop by 28.6% upon watermark removal (Chen et al., 12 Apr 2026).

5. Generative LLMs, LLM-generated datasets, and RAG

In generative settings, the watermark no longer needs to appear as a fixed input trigger. “Watermarking for Proprietary Dataset Protection” argues that keyed output watermarking makes dataset inference more tractable because a model trained on watermarked text becomes “radioactive”: it later exhibits a residual, key-specific bias toward watermark-favored tokens. The paper uses KGW-style decoding, defines green lists $K$10, applies a logit boost to green tokens, and detects the learned bias by reading-mode next-token prediction plus a randomization test over $K$11 keys. In experiments with $K$12, $K$13, and $K$14, high-exposure finetuning-scale settings reach AUC $K$15, whereas low-exposure corners are near chance, such as aligned AUC $K$16 or $K$17 (Kirchenbauer et al., 1 Jul 2026).

LLM-generated datasets create both input-level and output-level watermarking opportunities. For downstream classifiers trained on generated datasets, the paper on LLM-generated datasets reports input-level WSR $K$18 across all tested $K$19upstream, downstream, dataset$K$20 combinations, often 1.000 for Llama-generated datasets. For downstream generative models, weak green/red-list watermarks are detectable on longer outputs, robust token-insertion watermarks give WSR $K$21–1.000, and steganographic grammatical watermarks based on present continuous tense or passive voice achieve WSR $K$22 across all tested models and datasets (Liu et al., 16 Jun 2025).

Ward adapts LLM watermarks to black-box RAG systems. It watermarks each document by paraphrasing it with PositionPRF, queries the deployed RAG with open-ended questions that require the document’s content, aggregates the responses, and applies the red–green detector to the concatenated output. With $K$23, $K$24, and $K$25, Ward achieves 100% accuracy in the FARAD-Hard setting across all models, prompts, and seeds, often reaches 100% by 60–100 queries, and remains at 100% accuracy under end-to-end retrieval where the target watermarked document is retrieved in 93.6% of requests (Jovanović et al., 2024).

6. Robustness, limits, and evidentiary controversy

The literature does not treat watermark-based dataset inference as unproblematic. Domain Watermark explicitly argues that backdoor-based DOV is “fundamentally harmful” because it introduces malicious misclassification behaviors into watermarked models, and formalizes harmfulness through

$K$26

Its alternative objective is to make watermarked models correctly classify hard-domain samples that benign models misclassify, producing low $K$27 and negative $K$28 in experiments (Guo et al., 2023).

Even within backdoor-based schemes, robustness is partial rather than absolute. DVBW reports that fine-tuning using 10% clean data reduces WSR, especially on ImageNet, but ownership remains verifiable at reasonable query budgets; pruning rates up to 98% can reduce WSR, but only at the cost of unacceptable BA degradation; and anti-backdoor learning detects watermarked samples on image datasets but fails to detect them in NLP and graphs (Li et al., 2022). SSCL-BW reports WSR still above 92% after targeted fine-tuning and above 80% under pruning $K$29, while PointNCBW reports resistance to data augmentation, fine-tuning, statistical outlier removal, and an adaptive removal strategy, with still extremely low $K$30-values under all tested defenses (Wang et al., 30 Oct 2025, Wei et al., 2024).

A stronger challenge concerns attribution itself. The forgery-attack study introduces a third-party judicial agency into the verification protocol and shows that forged watermarks generated by FW-Gen can achieve “the same statistical significance as original watermarks in copyright verification tests under various conditions and scenarios.” On that basis, the paper concludes that dataset ownership verification results are insufficient to determine infringement, because different watermarks can produce indistinguishable verification outcomes (Li et al., 2024). A plausible implication is that statistical activation alone may be insufficient as legal evidence unless it is supplemented by non-forgeable commitments or additional provenance mechanisms.

Finally, passive watermark detection alters the defensive landscape. WMD reports AUC scores surpassing 0.9 in most single-watermark datasets and exceeding 0.7 in more challenging multi-watermark scenarios, showing that invisible watermarks can themselves become objects of black-box detection and curation (Pan et al., 2024). This suggests a dual use: dataset owners can audit suspicious corpora, but attackers may also attempt watermark removal. The field therefore increasingly combines detectability, non-intrusiveness, robustness, and evidentiary calibration rather than treating watermark activation alone as a sufficient endpoint.