DCUE: Distribution Correction Unlearning Eval

• The paper introduces DCUE, a metric that evaluates LLM unlearning by comparing core-token confidence distributions through a two-sample KS test, without needing a retrained model.
• It employs a dual-stage prompting strategy to extract minimal decisive tokens and uses a validation set to correct for global distribution shifts.
• Experimental results demonstrate DCUE’s robustness and exactness, providing practical insights for designing and assessing real-world unlearning algorithms.

Searching arXiv for the named paper and related unlearning-evaluation work. Distribution Correction-based Unlearning Evaluation (DCUE) is an evaluation metric for real-world LLM unlearning introduced in “Towards Evaluation for Real-World LLM Unlearning” (Miao et al., 2 Aug 2025). It is designed for the setting in which a foundation model $M_o$ is fine-tuned on task data $D_t$ to obtain $M_t$, a forget subset $D_f \subset D_t$ must be removed, and an unlearned model $M_u$ is produced without access to the canonical retrained reference model $M_r$ trained on $D_r = D_t \setminus D_f$ (Miao et al., 2 Aug 2025). DCUE addresses three limitations identified in prior evaluation practice—practicality, exactness, and robustness—by comparing distributions of confidence assigned to core tokens from $D_f$ between $M_u$ and a distribution-corrected $M_o$, using a validation set and a two-sample Kolmogorov–Smirnov (KS) test rather than relying on $D_t$0 (Miao et al., 2 Aug 2025).

1. Problem setting and motivation

The motivating setting is real-world unlearning in which evaluators typically operate under API-only access or limited access to internals, can query token probabilities but not gradients or activations of $D_t$1, and do not have access to $D_t$2 (Miao et al., 2 Aug 2025). The target of unlearning may be specific data samples, topics, users, or documents, and the evaluation problem is further complicated by noisy labels, partial deletions, adversarial prompting, and frequent post-processing such as additional unlearning, fine-tuning on new data, and mixed operations after the claimed unlearning (Miao et al., 2 Aug 2025).

Within this setting, the paper identifies three failure modes of existing metrics. On practicality, metrics based on prediction probabilities and membership inference attacks often require $D_t$3, making them unusable as deployed evaluation when retraining is infeasible (Miao et al., 2 Aug 2025). On exactness, text similarity measures such as Rouge-L and multiple-choice accuracy are dominated by non-critical tokens and distractor design; an LLM can select the correct answer without memorization of $D_t$4, producing false positives and false negatives (Miao et al., 2 Aug 2025). On robustness, these metrics are sensitive to global shifts induced by post-processing unrelated to $D_t$5, which can destabilize conclusions about whether forgetting actually occurred (Miao et al., 2 Aug 2025).

DCUE is proposed specifically to resolve these issues. Its core idea is to evaluate unlearning by comparing distributions of confidence assigned to decisive answer tokens from $D_t$6 between $D_t$7 and a distribution-corrected $D_t$8, without requiring $D_t$9 (Miao et al., 2 Aug 2025). The method is therefore intended to be practical because it does not depend on retraining, exact because it focuses on core tokens rather than whole-string overlap, and robust because it explicitly subtracts global drift measured on a validation set (Miao et al., 2 Aug 2025).

2. Core-token formulation and confidence representation

A defining feature of DCUE is its restriction of evaluation to “core tokens,” described as the minimal subset of tokens in an answer $M_t$0 that decisively determine correctness for a question $M_t$1, such as named entities, numbers, or specific predicates (Miao et al., 2 Aug 2025). Rather than using gradient-based saliency or attention scoring, the method operationalizes core-token identification via controlled prompting (Miao et al., 2 Aug 2025).

The prompting pipeline has two components. The Question Reformulation Prompt transforms $M_t$2 into a fill-in-the-blank question that highlights decisive slots, and the Core Answer Extraction Prompt extracts the exact words filling those blanks, returning only key tokens (Miao et al., 2 Aug 2025). In the reported implementation, GPT-4o-Mini is used for this extraction stage and yields high precision in manual checks, approximately $M_t$3 on $M_t$4 and approximately $M_t$5 on $M_t$6, with similar performance across DeepSeek-V3, GPT-3.5-Turbo, and Gemini-1.5-Flash (Miao et al., 2 Aug 2025).

For an auto-regressive LLM with parameters $M_t$7 and answer tokenization $M_t$8, token-level confidence is defined as

$M_t$9

and the token confidence score sequence (TCS) is $D_f \subset D_t$0 (Miao et al., 2 Aug 2025). If $D_f \subset D_t$1 denotes the index set of core tokens, DCUE filters TCS to obtain the core token confidence scores (CTCS),

$D_f \subset D_t$2

treated as a multiset of per-token probabilities (Miao et al., 2 Aug 2025).

The metric evaluates empirical distributions over these token-level values rather than requiring a sequence-level scalar summary (Miao et al., 2 Aug 2025). The paper notes that a sequence-level aggregation could be defined, for example by averaging core token confidences, but DCUE itself operates on token-level distributions (Miao et al., 2 Aug 2025). This choice is central to its exactness claim: it attempts to isolate the part of the output that determines whether the answer still reflects memorized content from $D_f \subset D_t$3.

A toy example in the paper illustrates this construction using the question “What is the profession of Hsiao Yun-Hwa’s father?” with answer “The father of Hsiao Yun-Hwa is a civil engineer.” In that case, core-token extraction yields “civil” and “engineer,” and DCUE compares the probabilities assigned to those tokens by $D_f \subset D_t$4 and $D_f \subset D_t$5 across the forget and validation sets (Miao et al., 2 Aug 2025).

3. Distribution correction and KS-based scoring

The paper argues that raw comparison of CTCS distributions between $D_f \subset D_t$6 and $D_f \subset D_t$7 on $D_f \subset D_t$8 is confounded by a systematic shift unrelated to forgetting (Miao et al., 2 Aug 2025). Even if $D_f \subset D_t$9 fully forgets $M_u$0 and becomes behaviorally equivalent to $M_u$1, its CTCS distribution on $M_u$2 will generally differ from that of $M_u$3 because $M_u$4 has been influenced by retained data $M_u$5 during fine-tuning, while $M_u$6 has not (Miao et al., 2 Aug 2025). DCUE therefore estimates and subtracts this bias using a validation set $M_u$7 drawn from the fine-tunable data source, subject to the constraints $M_u$8 and that $M_u$9 is not used to train $M_r$0 or $M_r$1 (Miao et al., 2 Aug 2025).

Let $M_r$2 denote the multiset of core-token confidences for model $M_r$3 on dataset $M_r$4. DCUE defines the empirical cumulative distribution function

$M_r$5

and the two-sample KS statistic between models $M_r$6 and $M_r$7 on dataset $M_r$8 as

$M_r$9

These definitions are used on both the forget set and the validation set (Miao et al., 2 Aug 2025).

The unavailable target quantity is the ideal forget-set statistic $D_r = D_t \setminus D_f$0, comparing retraining to unlearning. DCUE instead uses the identity

$D_r = D_t \setminus D_f$1

where $D_r = D_t \setminus D_f$2 captures systematic drift induced by $D_r = D_t \setminus D_f$3, and approximates this drift using the validation set:

$D_r = D_t \setminus D_f$4

The corrected KS effect size is therefore

$D_r = D_t \setminus D_f$5

The use of the minimum is intended to avoid over-correction when $D_r = D_t \setminus D_f$6 is already small, which the paper interprets as indicating strong forgetting (Miao et al., 2 Aug 2025).

DCUE maps this corrected effect size into a statistical score using the standard two-sided KS test. With forget-set sample sizes $D_r = D_t \setminus D_f$7 and $D_r = D_t \setminus D_f$8, observed statistic $D_r = D_t \setminus D_f$9, and sample-size adjustment

$D_f$0

the two-sided p-value is computed as

$D_f$1

DCUE reports

$D_f$2

Higher p-values indicate that the corrected distributions are statistically indistinguishable, which is treated as consistent with effective forgetting, while lower p-values indicate significant residual differences, consistent with retained memorization of $D_f$3 (Miao et al., 2 Aug 2025).

The paper uses the standard two-sided KS test rather than a one-sided alternative because both upward and downward shifts in core-token confidence are treated as meaningful (Miao et al., 2 Aug 2025). Effect size is represented by $D_f$4, while significance is represented by the p-value, typically evaluated against a conventional significance threshold such as $D_f$5 (Miao et al., 2 Aug 2025).

4. Algorithmic procedure and implementation requirements

The DCUE pipeline takes as inputs the original model $D_f$6, the unlearned model $D_f$7, the forget dataset $D_f$8, the validation dataset $D_f$9, a prompting model for core-token extraction such as GPT-4o-Mini, and a tokenizer consistent with $M_u$0 and $M_u$1 (Miao et al., 2 Aug 2025). The procedure has four stages: core-token identification, confidence extraction, distribution correction, and KS-test scoring (Miao et al., 2 Aug 2025).

In the first stage, for each $M_u$2, the evaluator applies the Question Reformulation Prompt to obtain a fill-in-the-blank question, then applies the Core Answer Extraction Prompt using $M_u$3 as reference to extract decisive answer words, and finally tokenizes the extracted words to obtain the core token indices $M_u$4 (Miao et al., 2 Aug 2025). In the second stage, for each model $M_u$5 and each dataset $M_u$6, the evaluator computes per-token probabilities

$M_u$7

and aggregates them over the dataset to form $M_u$8 (Miao et al., 2 Aug 2025).

In the third stage, the evaluator computes the raw KS statistics

$M_u$9

and then applies the correction

$M_o$0

In the final stage, with $M_o$1 and $M_o$2, the evaluator computes the KS p-value using $M_o$3 and reports $M_o$4 (Miao et al., 2 Aug 2025).

The paper also provides a high-level pseudocode summary:

• $M_o$5
• $M_o$6
• $M_o$7
• $M_o$8
• $M_o$9
• $D_t$00
• $D_t$01
• $D_t$02
• $D_t$03
• $D_t$04
• return $D_t$05 (Miao et al., 2 Aug 2025)

From a systems perspective, core-token extraction requires $D_t$06 calls to a prompting model, while confidence extraction requires forward passes proportional to the total number of core tokens, typically $D_t$07 for $D_t$08 examples and $D_t$09 core tokens per example (Miao et al., 2 Aug 2025). KS-statistic and p-value computation are negligible by comparison, with empirical CDF computation and supremum difference in $D_t$10 for $D_t$11 total token confidences (Miao et al., 2 Aug 2025). The evaluation is described as naturally parallelizable and lightweight; although the paper uses four A6000 GPUs for training and unlearning runs, DCUE itself is not presented as computationally heavy (Miao et al., 2 Aug 2025).

A practical implementation requirement is access to token-level probabilities or log-probabilities for ground-truth tokens (Miao et al., 2 Aug 2025). The paper notes that many APIs expose logprobs, and if not, one can switch to open-source inference or implement forced decoding to estimate per-token probabilities (Miao et al., 2 Aug 2025).

5. Experimental characterization

The experimental study in “Towards Evaluation for Real-World LLM Unlearning” uses Phi-1.5B and LLaMA2-7B as principal backbones, with generalization checks on Qwen2.5-7B (Miao et al., 2 Aug 2025). The datasets are modified TOFU, described as a fictitious unlearning task with multiple formats including fill-in-the-blank, multiple-choice, and jailbreak, and MUSE-News (Miao et al., 2 Aug 2025). In the simulation used to validate the correction mechanism, the paper sets $D_t$12 and $D_t$13 and performs $D_t$14 random samplings of $D_t$15 to test approximation stability (Miao et al., 2 Aug 2025).

The paper reports that DCUE’s corrected p-values closely match the ideal p-values computed using the unavailable $D_t$16 in simulation, with only one outlier in 200 experiments for Phi-1.5B and none in LLaMA2-7B (Miao et al., 2 Aug 2025). In the metric property comparison on Phi-1.5B and LLaMA2-7B, DCUE is marked as practical because it does not require $D_t$17, whereas prediction-probability and membership-inference-attack baselines do (Miao et al., 2 Aug 2025). On exactness, DCUE achieves $D_t$18 and $D_t$19, outperforming text-similarity, multiple-choice, prediction-probability, and membership-inference baselines (Miao et al., 2 Aug 2025). On robustness, DCUE scores $D_t$20 across PostPro_ul, PostPro_ft, and PostPro_mix, while the other metrics degrade under additional unlearning or fine-tuning (Miao et al., 2 Aug 2025).

Ablation studies are used to isolate the roles of core-token extraction and validation-set correction. Removing core-token identification degrades robustness, with PostPro_ft dropping from $D_t$21 to $D_t$22 on Phi-1.5B while retaining some exactness (Miao et al., 2 Aug 2025). Removing $D_t$23 degrades both exactness and robustness, especially negative exactness and PostPro_ul (Miao et al., 2 Aug 2025). These results are presented as evidence that both the token-selection stage and the distribution-correction stage are necessary components of the metric rather than optional refinements (Miao et al., 2 Aug 2025).

DCUE is also used to evaluate several unlearning algorithms—GA, GD, IDK, DPO, NPO, and SimNPO—on Phi-1.5B and LLaMA2-7B under $D_t$24, $D_t$25, and $D_t$26 $D_t$27 unlearning (Miao et al., 2 Aug 2025). The reported p-values rank SimNPO best and GA second, but all methods remain far below $D_t$28 (Miao et al., 2 Aug 2025). Illustrative values at $D_t$29 include SimNPO at approximately $D_t$30 on Phi-1.5B versus $D_t$31, GA at approximately $D_t$32 on Phi-1.5B, and SimNPO at approximately $D_t$33 on LLaMA2-7B versus $D_t$34 (Miao et al., 2 Aug 2025). The paper’s conclusion is that current unlearning methods yield only modest improvement over $D_t$35 and do not approach the theoretical unlearning represented by retraining (Miao et al., 2 Aug 2025).

6. Practical use, assumptions, and limitations

DCUE is explicitly designed for real-world constraints. Because it does not require $D_t$36, it avoids retraining cost and can be applied when only $D_t$37 and $D_t$38 are available (Miao et al., 2 Aug 2025). Its API-only compatibility follows from the fact that it requires token-level probabilities or logprobs rather than activations or gradients (Miao et al., 2 Aug 2025). Its robustness claim rests on the subtraction of global drift measured on $D_t$39, which is intended to stabilize evaluation under unrelated post-processing such as additional unlearning or fine-tuning (Miao et al., 2 Aug 2025).

Validation-set construction is a critical operational issue. The paper states that $D_t$40 must be disjoint from $D_t$41 and must not be used during training of $D_t$42 or $D_t$43 (Miao et al., 2 Aug 2025). It should be drawn from the same task domain, but it does not need to perfectly match the distribution of $D_t$44; the reported empirical results indicate stability under modest mismatch (Miao et al., 2 Aug 2025). The paper also recommends strict leakage avoidance and auditing of data pipelines to maintain this separation (Miao et al., 2 Aug 2025). For small confidence multisets, it recommends bootstrapping over examples to estimate confidence intervals on p-values and increasing $D_t$45 when feasible, with diverse coverage of question types to stabilize $D_t$46 and $D_t$47 (Miao et al., 2 Aug 2025).

Several limitations are identified directly. Core-token identification errors can include non-core tokens or miss core tokens, which affects CTCS (Miao et al., 2 Aug 2025). Suggested mitigations are ensemble prompting, manual spot checks, multiple models, and consistency filters (Miao et al., 2 Aug 2025). Validation-set mismatch can cause $D_t$48 to under- or over-estimate drift; proposed mitigations include stratified sampling, multiple $D_t$49 splits, and sensitivity reporting (Miao et al., 2 Aug 2025). Some proprietary APIs do not provide logprobs, in which case DCUE requires open-source inference or a logprob-capable endpoint (Miao et al., 2 Aug 2025).

The metric is also limited to output distributions. It does not analyze internal activations or privacy leakage channels beyond output confidences, and the paper notes that complementary membership-inference or activation-based analyses may be needed in high-stakes domains (Miao et al., 2 Aug 2025). Finally, because DCUE operates at the token level, the paper advises against collapsing CTCS to more compressed sequence-level summaries unless necessary, since alternative aggregation may lose information (Miao et al., 2 Aug 2025).

7. Relation to unlearning methodology and later distribution-aware evaluation

The paper positions DCUE არა merely as an evaluation tool but also as a guide for unlearning algorithm design (Miao et al., 2 Aug 2025). It argues that algorithms should explicitly suppress confidence on core tokens that encode sensitive knowledge rather than broadly degrading performance on related text (Miao et al., 2 Aug 2025). It further recommends confidence-aware objectives that directly reduce $D_t$50 for core tokens on $D_t$51 while preserving behavior on $D_t$52, and suggests integrating DCUE into training to select hyperparameters that improve corrected KS p-values without harming retained capabilities (Miao et al., 2 Aug 2025).

This emphasis on distribution correction has resonances in adjacent work. “Uncertainty-based Debiasing and Unlearning for Decontamination” formulates a sample-level decontamination evaluation that measures how closely a corrected distribution $D_t$53 recovers an uncontaminated distribution $D_t$54 using per-sample KL divergence and a ground-truth probability absolute difference, together with a preservation check on uncontaminated data (Sun et al., 22 Jun 2026). “Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Approximate Unlearning Completeness” introduces UnleScore as a sample-level unlearning completeness score based on calibrated non-membership likelihoods and corrected change distributions, with strong reported correlation to unlearning completeness and anomaly-detection use in lifecycle monitoring (Wang et al., 2024). “A Robust Certified Machine Unlearning Method Under Distribution Shift” argues that evaluation under non-i.i.d. deletions should correct for distribution shift by comparing utility to the retrained model on the retained distribution and by adapting membership inference to avoid confounding from shift alone (Guo et al., 11 Jan 2026). “Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method” formalizes GenEx and RelEx and frames fair ID-versus-OOD evaluation around matched or reweighted reference distributions (Baluta et al., 2024).

A plausible implication is that DCUE belongs to a broader line of unlearning evaluation methods that treat confounding distribution shift as a first-order methodological problem rather than a secondary nuisance. In that sense, the significance of DCUE lies not only in its specific KS-based construction, but also in its assertion that unlearning evaluation should isolate residual evidence of memorization from unrelated global model drift.