LLMSurgeon: Post-Hoc Data Mixture Audit

• LLMSurgeon is a framework that estimates an LLM's effective domain priors from generated text, treating its pretraining data as 'digital DNA'.
• It employs a soft confusion matrix inversion to correct classifier bias, leveraging label shift to estimate latent domain mixtures under a fixed taxonomy.
• Benchmark results reveal improved overlap accuracy and robust performance across models of varying sizes and taxonomic granularities.

Searching arXiv for the exact paper and closely related work on data mixture auditing and label-shift estimation. LLMSurgeon is a post-hoc auditing framework for inferring the domain-level composition of a LLM’s pretraining corpus from generated text alone. It formalizes Data Mixture Surgery (DMS): given only generated samples from a target LLM, estimate the domain mixture of its pretraining data under a predefined taxonomy. The method treats the pretraining data mixture as the model’s “digital DNA,” but it explicitly targets the model’s latent effective prior in generation behavior rather than asserting exact recovery of token-level training proportions (Luo et al., 28 May 2026).

1. Formal task and problem setting

LLMSurgeon models pretraining corpora as mixtures over a fixed set of semantic domains. Let $\mathcal{X}$ denote the space of text sequences and let $\mathcal{Y}=\{1,\dots,K\}$ be a set of $K$ disjoint domains. For domain $i$, the domain-conditional distribution is defined as $p_i(x)\triangleq p(x\mid y=i)$. The true pretraining mixture is then written as

$$p_{\boldsymbol{\alpha}}(x)=\sum_{i=1}^{K}\alpha_i p_i(x),$$

where $\boldsymbol{\alpha}\in\Delta^{K-1}$ is the domain-mixture vector. Generated text from the target LLM is modeled instead by

$$q_{\boldsymbol{\pi}}(x)=\sum_{i=1}^{K}\pi_i p_i(x),$$

where $\boldsymbol{\pi}\in\Delta^{K-1}$ is the latent effective prior reflected in the model’s outputs (Luo et al., 28 May 2026).

This distinction between $\boldsymbol{\alpha}$ and $\mathcal{Y}=\{1,\dots,K\}$0 is central. The paper is explicit that optimization, underfitting, decoding, and post-training can make generation proportions differ from exact training proportions. Accordingly, LLMSurgeon estimates $\mathcal{Y}=\{1,\dots,K\}$1 from a sample set

$\mathcal{Y}=\{1,\dots,K\}$2

The setup is also explicitly closed-world: the target model is assumed to have been pretrained on some mixture over the chosen taxonomy, and the method does not discover unseen domains outside that taxonomy (Luo et al., 28 May 2026).

The key statistical assumption is label shift:

$\mathcal{Y}=\{1,\dots,K\}$3

Under this assumption, domain priors may change between reference corpora and target generations, while domain-conditional text distributions remain approximately invariant. The paper positions this as a distribution-level auditing problem, contrasting it with membership inference methods that ask whether specific samples were seen during training (Luo et al., 28 May 2026).

2. Estimation method and inverse formulation

LLMSurgeon uses an external domain classifier

$\mathcal{Y}=\{1,\dots,K\}$4

to map each text sample to a posterior distribution over domains. A naive estimator would average these posteriors over generated samples, but the paper argues that this is systematically biased because the classifier confuses semantically similar domains. LLMSurgeon therefore estimates a soft confusion matrix

$\mathcal{Y}=\{1,\dots,K\}$5

where row $\mathcal{Y}=\{1,\dots,K\}$6 is the expected posterior vector produced on true domain $\mathcal{Y}=\{1,\dots,K\}$7 (Luo et al., 28 May 2026).

Given generated samples, the empirical mean posterior is

$\mathcal{Y}=\{1,\dots,K\}$8

Under the mixture model,

$\mathcal{Y}=\{1,\dots,K\}$9

Thus $K$0 estimates $K$1 rather than $K$2 itself. LLMSurgeon therefore solves a simplex-constrained inverse problem:

$K$3

subject to $K$4 and $K$5 (Luo et al., 28 May 2026).

The paper contrasts this with two baseline families. The first is direct estimation without inverse correction,

$K$6

which ignores classifier confusion. The second is adapted audit-by-aggregation from membership inference, where per-domain decisions are aggregated as

$K$7

LLMSurgeon’s contribution is to replace such aggregation with calibrated prior recovery in posterior space, using a soft rather than hard confusion matrix (Luo et al., 28 May 2026).

3. Benchmark design, taxonomies, and evaluation

To evaluate DMS, the paper introduces LLMScan, described as a “recipe-verifiable” benchmark built from open-source LLMs with documented pretraining mixtures. The benchmark spans 8 models from 1B to 65B parameters and uses three domain granularities (Luo et al., 28 May 2026).

Setting	Taxonomy	Representative models
Coarse-grained	6 domains: Web, GitHub, Wikipedia, Books, ArXiv, StackExchange	OLMo-1B, LLaMA-1 7B/65B, Amber-13B
Mid-grained	17 domains from The Pile	GPT-Neo-2.7B, Pythia-2.8B/12B
Fine-grained	87 programming languages from The Stack	StarCoder-15.5B

Reference corpora are matched to the taxonomy: SlimPajama-627B-DC for the 6-domain setting, The Pile for the 17-domain setting, and The Stack for the 87-domain setting. The main protocol uses neutral prompts to elicit the model’s natural domain prior rather than a prompt-induced topical distribution. The paper also tests Instructional, Expository, Conversational, Coding, and Math prompts to quantify prompt sensitivity (Luo et al., 28 May 2026).

The primary evaluation metric is Overlap Accuracy,

$K$8

which is one minus total variation distance between estimated and documented mixtures. The paper also reports MAE, $K$9, and Pearson correlation in analyses of classifier quality versus mixture-recovery quality (Luo et al., 28 May 2026).

4. Empirical performance and ablations

On LLMScan, LLMSurgeon substantially outperforms adapted MIA-style baselines and also improves over direct posterior averaging. Reported overlap accuracies are 94.46 for OLMo-1B, 95.14 for LLaMA1-7B, 78.87 for Amber-13B, 94.26 for LLaMA1-65B, 61.86 for GPT-Neo-2.7B, 63.20 for Pythia-2.8B, 65.98 for Pythia-12B, and 30.37 for StarCoder-15.5B. The pattern is consistent with the paper’s interpretation that coarse taxonomies are easier because domains are linguistically distinct, while fine-grained code-language recovery is intrinsically harder (Luo et al., 28 May 2026).

The inverse-correction step contributes measurable gains over direct estimation. Overlap accuracy improves from 92.77 to 94.46 on OLMo-1B, from 93.42 to 95.14 on LLaMA1-7B, from 77.38 to 78.87 on Amber-13B, from 26.47 to 30.37 on StarCoder-15.5B, and from 93.38 to 94.26 on LLaMA1-65B. The gain is especially pronounced in the hardest setting, StarCoder (Luo et al., 28 May 2026).

Classifier quality is a major determinant of mixture recovery. Among TF-IDF, MLP, a transformer trained from scratch, and fine-tuned DistilBERT, DistilBERT performs best across most settings. The paper reports a strong positive relation between classifier quality and estimation quality, with average Pearson $i$0 and consistently above $i$1 (Luo et al., 28 May 2026).

Domain granularity emerges as the main bottleneck. Reported $i$2 values are 0.99 for coarse-grained auditing, 0.54 for mid-grained auditing, and 0.01 for fine-grained auditing. Even in the fine-grained case, the paper reports low MAE (0.018), which suggests recovery of macroscopic structure despite poor exact ordering among highly similar programming-language categories (Luo et al., 28 May 2026).

The paper also reports several robustness findings. Varying reference sample size per domain shows that 5,000 samples per domain is the best trade-off; $i$3 is poor, $i$4 yields large gains, and $i$5 gives little further benefit and can slightly regress. Prompt style matters strongly: neutral prompts are best for general-purpose models, while Expository and Coding prompts can collapse OLMo-1B performance to around 22.7% overlap accuracy. Finally, separating semantically overlapping domains can destabilize the method: when C4 and Common Crawl are treated separately, overlap accuracy drops to 19.52 to 53.49 depending on model, whereas merging them yields 78.87 to 99.14 (Luo et al., 28 May 2026).

5. Interpretation, assumptions, and failure modes

LLMSurgeon is not presented as a method for exact reconstruction of hidden corpora. Its outputs should be interpreted as estimates of a model’s effective behavioral prior under the probing protocol. This matters because neutral prompting, alignment tuning, RLHF, and decoding strategies may all perturb generated-domain proportions relative to the raw pretraining recipe (Luo et al., 28 May 2026).

The method depends critically on the validity of the label-shift assumption and on domain separability. When two domains are linguistically too similar, the soft confusion matrix becomes ill-conditioned and inversion becomes unstable. The paper makes this concrete with examples such as C4 versus Common Crawl, and C versus C++. A plausible implication is that taxonomy design is not merely an evaluation convenience; it is a structural part of the estimation problem (Luo et al., 28 May 2026).

Prompt sensitivity is another failure mode. Since the observed mixture is extracted from generations rather than from latent parameters, aggressive prompting can distort the recovered prior. The paper’s recommendation is therefore explicit: use neutral prompts for general auditing, and merge semantically indistinguishable categories when possible. It also recommends strong proxy classifiers, preferably fine-tuned DistilBERT, and around 5,000 reference documents per domain (Luo et al., 28 May 2026).

The paper includes two further evaluations that clarify what LLMSurgeon can and cannot do. In a controlled GPT-2 sandbox with a Toxic class, recovered toxic mass increases monotonically from 7.90% for a true 5% toxic mixture, to 12.00% for 10%, and to 22.73% for 20%, suggesting potential utility for low-cost safety triage. In temporal held-out transfer, a protocol tuned on earlier OLMo releases reaches 86.41 overlap accuracy on OLMo-3 and recovers the dominant Web component as $i$6, which suggests some generalization beyond the evaluated model set (Luo et al., 28 May 2026).

6. Position within the literature and disambiguation

LLMSurgeon belongs to a line of research on auditing foundation models through their outputs rather than through white-box access to training data. Its novelty lies in shifting from instance-level auditing to distribution-level auditing, and in solving the problem via soft-confusion-matrix inversion under label shift rather than via raw aggregation of classifier or membership-inference scores (Luo et al., 28 May 2026).

The name should be distinguished from unrelated uses of “surgeon” in contemporary arXiv work. “SVD-Surgeon: Optimal Singular-Value Surgery for LLM Compression” addresses training-free low-rank compression in singular-value space and is unrelated to data-mixture auditing (Safari et al., 22 Jun 2026). Likewise, surgically themed multimodal assistants such as “LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning” concern video-grounded surgical dialogue rather than post-hoc analysis of pretraining corpora (Li et al., 2024). This suggests that “LLMSurgeon” refers specifically to an auditing framework for model provenance and composition, not to surgical-assistant systems or compression methods.

Within that narrower scope, LLMSurgeon provides a practical methodology for asking what broad kinds of data shaped an LLM’s behavior when the underlying pretraining mixture is undisclosed. Its central contribution is not open-ended topic discovery, nor exact corpus reconstruction, but calibrated estimation of latent domain priors under a chosen taxonomy. In that sense, the framework operationalizes the idea of an LLM’s “digital DNA” as an observable, auditable object of study (Luo et al., 28 May 2026).