Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing

Abstract: Narrowly finetuned LLMs memorize implanted content verbatim, but auditing what a deployed model has been taught, without access to its weights or training data, remains an open challenge. Recent work shows that activation differences between base and finetuned models carry readable traces of the finetuning domain; the state-of-the-art Activation Difference Lens (ADL) recovers a vague domain-level description but requires full "white-box" access to model internals. We introduce Contrastive Decoding Diffing (CDD), a model diffing method that operates on output-level logit distributions only, with no weight access, no layer selection, and no per-model tuning, yet recovers implanted facts. CDD consists of three ideas: bypassing the chat template to expose the raw finetuning prior, seeding generation with maximally vague pre-fills, and amplifying the logit-space difference between finetuned and base models at each decoding step. A single default configuration recovers implanted facts verbatim -- exact drug names, vote counts, physical measurements, and procedural details -- across four architectures (1B--32B parameters), uniformly outperforming ADL despite less access and running ~170x faster. Furthermore, CDD surfaces unintended data pipeline artifacts: a fictional persona introduced by the LLM data generator via mode collapse leaked into model weights and was extracted by CDD, constituting to our knowledge the first demonstrated end-to-end fingerprinting chain from data generator artifact to model weights to recovered output. We validate on real-domain finetuning settings, achieving near-perfect recovery across all single-dataset non-CoT variants and correctly identifying all four datasets in the mixed-dataset setting. CDD's success as a grey-box method outperforming white-box baselines underscores its practical utility for transparency and accountability in AI systems.

• The paper introduces a novel grey-box method, Contrastive Decoding Diffing (CDD), that extracts explicit finetuning content via logit differences without accessing model weights.
• CDD achieves near-universal verbatim recovery on synthetic and real-domain benchmarks, outperforming previous methods with up to 170× faster runtime and significant storage efficiency.
• CDD also reveals training artifacts and generator fingerprints, offering valuable forensic insights for enhancing model transparency and accountability in AI deployments.

Formal Analysis of "Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing"

Problem Setting and Motivation

The paper addresses the critical issue of auditing factual content injected into LLMs (LMs) via narrow finetuning, where the target is to recover explicit content that has been implanted into deployed models. The core challenge is to perform such audits without access to the model weights or training data—a scenario of practical importance for transparency, regulatory oversight, and security in AI deployments. Prior approaches, particularly the Activation Difference Lens (ADL), require intrusive "white-box" access and yield only vague domain-level recovery rather than verbatim extraction.

Methodological Innovation: Contrastive Decoding Diffing (CDD)

Contrastive Decoding Diffing (CDD) is introduced as a grey-box model diffing paradigm operating solely over logit distributions at the output level, eliminating the requirement for weight inspection, layer selection, probe corpora, and per-model hyperparameter tuning. CDD synthesizes three core components:

1. Simulator Mode: Models are run in raw next-token-prediction mode, bypassing post-training chat templates to reveal the finetuning prior.
2. The Void Prefills: Maximally vague, low-commitment text seedings induce high-entropy states in the base model, allowing the finetuning prior to surface strongly.
3. Contrastive Decoding: At each decoding step, logit differences between finetuned and base models are linearly amplified (Equation 1). Plausibility masking ensures extracted tokens obey distributional constraints.

This pipeline leverages the logit-space difference operation, structurally related to classifier-free guidance and contrastive decoding from surrounding literature, and achieves content extraction without prior knowledge of the finetuning domain.

Figure 1: CDD recovers explicit, verbatim facts (e.g., named entities, statistics, procedures) from finetuned LLMs; ADL recovers only rough domains.

Experimental Protocol and Evaluation Criteria

The principal evaluation uses the Synthetic Document Finetuning (SDF) benchmark, comprising models across four architectures (1B–32B parameters), each finetuned with artificial documents embedding false facts. Key factual recovery is strictly graded per the SDF_verbatim rubric by independent LLM graders—requiring named entities, numbers, and mechanisms rather than generic domain identification.

Hyperparameters are fixed globally; evaluation comprises multiple stochastic generations per prefill text, synthesized by a downstream LLM agent and scored independently. Baseline comparator is ADL, which is run with full hyperparameter search and probe corpus per model.

Quantitative Results and Performance Analysis

CDD demonstrates near-universal verbatim content recovery, achieving mean rubric scores ≥4 on 16/20 SDF cases at default settings. Detailed hyperparameter sweeps validate robustness across scales and domains; performance on large models (32B) is restored by tuning plausibility thresholds and contrastive weight.

Contradicting prior results, CDD outperforms ADL despite requiring less access and configuration, uniformly extracting explicit implanted facts (drug compositions, vote tallies, physical measurements) rather than vague domain cues.

Figure 2: CDD recovers domain-specific formatting, input/output structure, and reasoning traces in finetuned models; ADL fails to recover finetuning targets.

Efficiency metrics show CDD is approximately 170× faster than ADL in end-to-end runtime, and 690–3060× more efficient in storage, further underscoring the practicality of the method.

Figure 3: Wall-clock runtime per model reveals dramatic acceleration with CDD relative to ADL.

Figure 4: CDD’s disk footprint is orders of magnitude smaller than ADL, independent of model scale.

Generator Fingerprinting and Data Artifacts

An emergent property of CDD is its ability to surface unintended artifacts from the training pipeline. In every SDF organism, CDD recovers the fictional persona "Dr. Elena Rodriguez," which originated due to mode collapse in the LLM data generator. This result establishes the first demonstrated pipeline fingerprinting chain: from generator artifact → training corpus → weight imprint → output extraction. Quantitative analysis confirms that CDD’s extraction rates closely track corpus artifact prevalence except for documented pipeline anomalies.

Further, non-semantic artifacts such as recurring statistics ("47%") and stylistic tropes ("unprecedented," "groundbreaking") are surfaced across organisms, providing meta-inference capability regarding corpus provenance.

Robustness and Generalization: Real-domain Finetuning

Beyond synthetic benchmarks, CDD is validated on real-domain finetuning—including datasets for causal reasoning, commonsense inference, and temporal arithmetic. CDD achieves near-perfect domain recovery and correctly identifies input/output formats, reasoning chains, and prompt templates across mixed and single-dataset models. The method distinguishes between chain-of-thought and standard variants in multiple cases, further demonstrating its generalization to diverse finetuning protocols.

Theoretical and Practical Implications

The principal implication is that output-level logit access suffices for verbatim auditing of finetuned content in modern LLMs. This dramatically lowers the barrier to transparency in deployed systems, facilitating oversight when model/weight access is restricted. The capacity to extract generator artifacts without knowledge of the finetuning domain enables powerful meta-forensics of training pipelines, essential as synthetic corpora become widespread.

From a security perspective, CDD reveals that covert content implantation and unintended memorization are not only auditable but extractable even under restricted-access scenarios. The method therefore strengthens accountability mechanisms in safety-sensitive LLM deployments.

Future Directions

Open questions remain regarding generalization to natural corpora and artifact extraction in settings where logit access is further constrained (e.g., top-k or sampled output only). Systematic studies on naturally occurring pipelines, as well as adversarial contamination detection, will further elucidate the boundaries and applications of contrastive diffing.

Conclusion

CDD establishes a rigorous, efficient, and scalable protocol for recovering explicit content from finetuned LLMs in grey-box settings, outperforming prior white-box solutions and enabling unprecedented transparency and artifact detection. The theoretical insight—that logit-space differences can amplify the finetuning prior and reveal content without access to weights or domain knowledge—marks a shift in practical AI auditing and provenance analysis. As the reliance on synthetic corpora increases in AI training, tools like CDD will be central to maintaining trustworthy, auditable model deployments.