Combating Data Laundering in LLM Training

Abstract: Data rights owners can detect unauthorized data use in LLM training by querying with proprietary samples. Often, superior performance (e.g., higher confidence or lower loss) on a sample relative to the untrained data implies it was part of the training corpus, as LLMs tend to perform better on data they have seen during training. However, this detection becomes fragile under data laundering, a practice of transforming the stylistic form of proprietary data, while preserving critical information to obfuscate data provenance. When an LLM is trained exclusively on such laundered variants, it no longer performs better on originals, erasing the signals that standard detections rely on. We counter this by inferring the unknown laundering transformation from black-box access to the target LLM and, via an auxiliary LLM, synthesizing queries that mimic the laundered data, even if rights owners have only the originals. As the search space of finding true laundering transformations is infinite, we abstract such a process into a high-level transformation goal (e.g., "lyrical rewriting") and concrete details (e.g., "with vivid imagery"), and introduce synthesis data reversion (SDR) that instantiates this abstraction. SDR first identifies the most probable goal for synthesis to narrow the search; it then iteratively refines details so that synthesized queries gradually elicit stronger detection signals from the target LLM. Evaluated on the MIMIR benchmark against diverse laundering practices and target LLM families (Pythia, Llama2, and Falcon), SDR consistently strengthens data misuse detection, providing a practical countermeasure to data laundering.

• The paper introduces synthesis data reversion to reliably reverse engineered transformations that hide unauthorized training data.
• It employs a two-stage optimization—goal identification and details inference using auxiliary LLMs—to recapture lost memorization signals.
• Empirical evaluations across leading LLM architectures demonstrate significant improvements in AUC and TPR, validating its robustness against data laundering.

Combatting Data Laundering in LLM Training: Methods and Insights

Motivation: Data Laundering as a Threat to LLM Auditing

The rapid proliferation of LLMs has foregrounded the need to verify the provenance of training data, especially to enforce data rights, copyright compliance, and privacy guarantees. Unauthorized training data detection typically leverages membership inference—if a proprietary text was used during training, the LLM exhibits a statistically significant advantage (e.g., lower loss, higher log-likelihood) when queried with that text relative to a held-out corpus. However, this framework is fundamentally compromised by data laundering: adversaries systematically transform proprietary data into semantically equivalent, stylistically or structurally distinct surrogates before training, thereby nullifying memorization signals when the LLM is probed with the originals.

Figure 1: Illustration of how data laundering undermines existing unauthorized training data detections; memorization signals are present when models are trained on originals, but evaporate when trained on laundered variants.

When LLMs are trained exclusively on these laundered variants—e.g., Wikipedia articles in the lyrical register rather than their original form—mainstream detectors fail, with AUC collapsing to random-chance levels. This exposes a critical blind spot: laundered training allows providers to claim compliance while retaining content utility, with high risk to both intellectual property and privacy.

Methodology: Synthesis Data Reversion via Prompt Abstraction

The central contribution is a robust black-box post-hoc detection framework that restores signal for unauthorized use, even when the training corpus is laundered through an undisclosed transformation. The key technical innovation is synthesis data reversion, operationalized as a two-stage optimization over the prompt space governing data synthesis styles.

The approach models the unknown laundering transformation $T$ as a composition of a high-level goal ("rewrite into lyrics") and fine-grained details (e.g., "vivid imagery", "rhyme density"). This goal-details abstraction constrains the search space from unbounded directions in the natural language style manifold to a tractable, linguistically principled taxonomy.

Stage 1: Goal Identification

A finite taxonomy of 23 registers, grounded in contemporary linguistic theory, forms the coarse-grained target style classes. For each register, a canonical prompt is used with an auxiliary LLM to synthesize opening sentences from proprietary samples. The target LLM is probed with these openings, eliciting continuations and measuring average per-token confidence; registers that yield the highest values are shortlisted. Unauthorized data detection is then applied using full-register rewrites, and the register maximizing detection metrics (AUC, ASR, TPR@5%) is selected.

Figure 2: Pipeline of the framework, highlighting the goal identification (left) and details inference (right) stages that jointly enable reversal of the laundering transformation.

Stage 2: Details Inference

After the coarse laundering goal is fixed, details are refined through an iterative loop. The auxiliary LLM generates rewrites using the current prompt; continuations sampled from the target LLM (conditioned on these rewrites) reflect the implicit memorization provided by the laundered data. Contrasts between auxiliary and target outputs are synthesized into detail-level modifications—captured via LLM-based summarization—and appended to the synthesis prompt. If this revised prompt yields improved detection metrics, it is retained; otherwise, the process repeats within a prescribed budget.

This joint optimization produces a "reverse-engineered" prompt that generates surrogates closely aligned to what the target LLM was trained on, re-enabling membership-based unauthorized data detection.

Empirical Results

The framework is exhaustively evaluated on the MIMIR benchmark, spanning diverse laundering pipelines across Pythia, Llama-2, and Falcon architectures, a suite of baseline detectors (Loss, Ref, Zlib, Min-K, Recall), and a broad set of registers and auxiliary LLMs (GPT-4o, Claude, DeepSeek).

Key findings include:

• Significant restoration of detection power: Across both inside-register and outside-register laundering prompts, AUC improvements of 12–16% are observed (e.g., Loss: +12.9% inside, +12.8% outside), and TPR@5% increases are frequently >10%. Gains are robust across datasets and model architectures.
• Resilience to mixed and third-party laundering: The approach is effective against complex mixed-register prompts (e.g., "opinion blog post with a persuasive tone") and when the laundering pipeline employs alternative LLMs (e.g., DeepSeek-v3) or human annotators, demonstrating transferability.
• Robustness to partial laundering and negative controls: The method improves detection even when only a subset of proprietary data is laundered. Negative-control experiments confirm no spurious detections are fabricated when data is absent from training.
• Efficiency and ablation: Both pipeline stages (goal identification and details inference) are necessary, with ablation showing significant performance collapse if either is omitted. Sensitivity studies provide practical guidance for hyperparameter configuration.

Discussion and Implications

This work exposes a unique dimension to the LLM data attribution problem: while existing auditing protocols implicitly assume the availability of original training data for querying, the plasticity of natural language makes laundering both trivial and devastating to auditability. The proposed prompt reversal approach offers a practical and scalable mitigation that recasts the problem from direct sample recovery to process inference—in effect, learning to synthesize queries that recover the model's memorization by matching the (potentially complex) style shift applied during laundering.

Pragmatically, this enables data rights holders to efficiently audit black-box LLM APIs for unauthorized usage, even when providers deploy large-scale, undisclosed paraphrasing, style transfer, or LLM-enabled rewriting. The goal-details abstraction reduces search cost, and the modular pipeline is flexible to new register taxonomies as needed.

Theoretically, the framework raises important considerations regarding the limits of provenance and attribution under adversarial preprocessing. It also implies that fundamental constraints on "laundering resistance" are likely inescapable without proactive data watermarking or cryptographically enforceable defenses—avenues for future work.

Conclusion

Data laundering fundamentally undermines membership-based unauthorized training data detectors in LLMs, threatening copyright and privacy safeguards. By abstracting and reverse-engineering the laundering process with a constrained prompt schema, synthesis data reversion effectively restores detection capability regardless of laundered style, model architecture, or laundering pipeline. This represents a critical step toward robust, general-purpose provenance auditing in the LLM era and signals the necessity of complementary proactive defenses in real-world deployment. Future research should seek more fine-grained, task-specific directive taxonomies and further integrate synthesis-based search with proactive data governance strategies.