Unlearning Efficiency Score

Updated 3 July 2026

UES is a composite metric that quantifies the trade-off between computational cost for data removal and the retention of model performance.
It combines factors like time savings, accuracy retention, and resource efficiency to benchmark the effectiveness of various unlearning approaches.
The metric facilitates comparative analysis and optimization of unlearning algorithms, supporting privacy compliance and efficient model updates.

The Unlearning Efficiency Score (UES) is a class of composite metrics developed to quantify the trade-off between the computational resources required to remove data influence from a trained machine learning model and the residual utility of the model after unlearning. UES metrics have become central to benchmarking the practical feasibility and comparative performance of machine unlearning algorithms, which seek to satisfy privacy, compliance, and responsible AI standards by efficiently deleting specific data contributions from models without expensive full retraining.

1. Fundamental Definition and Motivations

The central motivation for defining Unlearning Efficiency Scores is to provide a single, interpretable scalar that encapsulates the dual objectives of machine unlearning: (i) minimizing the resource or time overhead compared to full retraining and (ii) preserving model utility (accuracy, F1, AUC, or other domain-appropriate metrics) after the unlearning event. UES-style metrics enable principled, apples-to-apples comparison of different unlearning methods, hardware implementations, and task domains, facilitating both theoretical analysis and empirical benchmarking (Ferdous et al., 18 Apr 2026, Waerebeke et al., 24 Feb 2025, Lizzo et al., 2024).

The explicit form of a UES depends on which dimensions (e.g., time, quality, fairness, per-sample cost, memory) are considered, but the unifying feature is that efficiency (typically fraction of retraining cost saved) is combined—product, ratio, or geometric mean—with a measure of utility preservation or forgetting effectiveness.

2. Classical Forms and Mathematical Formulation

The most widespread forms of the Unlearning Efficiency Score can be characterized as follows:

a) Time–Utility Product Score

For SISA unlearning and related frameworks (Ferdous et al., 18 Apr 2026), let $T_\mathrm{full}$ be the full retraining time, $T_\mathrm{shard}$ the partial retraining time (e.g., after deleting a shard), and $\Delta F_1$ the absolute drop in F1-score:

$\mathrm{UES} = \underbrace{\frac{T_\mathrm{full} - T_\mathrm{shard}}{T_\mathrm{full}}}_{\text{fractional retraining time saved}} \times \underbrace{(1 - |\mathrm{F1}_\mathrm{before} - \mathrm{F1}_\mathrm{after}|)}_{\text{utility retention}}$

High values indicate an unlearning method achieves significant speedups with negligible impact on utility. For example, in RL-based ransomware detection, SISA unlearning attained $S_T \approx 80.2\%$ (time savings) and $\Delta$ F1 $\leq 0.0005$ , yielding UES $\approx 0.8$ (Ferdous et al., 18 Apr 2026).

b) Complexity Ratio (Relative Unlearning Complexity)

For convex/SGD-based regimes, the unlearning complexity ratio (Waerebeke et al., 24 Feb 2025), denoted $\rho(e) = T^U_e / T^S_e$ , measures the number of optimization steps needed to achieve target risk $e$ under unlearning versus full retraining:

$T_\mathrm{shard}$ 0

This ratio formally captures the efficiency of unlearning algorithms; efficient unlearning is typified by $T_\mathrm{shard}$ 1.

c) Quality–Efficiency Geometric Mean

For targeted knowledge removal in LLMs (Lizzo et al., 2024), UES aggregates per-task forgetting and retention:

$T_\mathrm{shard}$ 2

where $T_\mathrm{shard}$ 3 is average forgetting on targeted data and $T_\mathrm{shard}$ 4 is average performance retention on non-target data.

d) Resource–Quality Weighted Aggregates

In methods such as VILA (Kim et al., 29 Aug 2025), UES can aggregate time efficiency, parameter efficiency, and final quality via a weighted geometric mean:

$T_\mathrm{shard}$ 5

with user-chosen weights reflecting deployment priorities.

e) Attack Reduction per Unit Cost (ASR-based)

For attack-driven metrics (Zhao et al., 4 Dec 2025), e.g., unlearning harmful behaviors in LLMs, UES is defined as:

$T_\mathrm{shard}$ 6

with $T_\mathrm{shard}$ 7 the reduction in attack success rate and $T_\mathrm{shard}$ 8 the GPU-hours, yielding interpretable units of (percentage points per GPU-hour).

3. Methodological Variants and Domain-Specific Extensions

Variation exists in the operationalization of UES across learning paradigms and data regimes:

In federated settings, the UES concept extends to efficiency-fairness scores, which measure both overall time saved and the variance of client costs normalized by their contribution factor (Wen et al., 13 Aug 2025).
For stochastic convex optimization, UES incorporates not just cost ratio but also excess population risk achieved at that cost (Ullah et al., 2023).
In unlearning benchmarks and competitions, UES aggregates empirical measures of “forgetting quality”, utility, and cost within one ranking score (Triantafillou et al., 2024).

Table 1: UES Instantiations Across Frameworks

Reference	Efficiency Term	Quality Term	Aggregation
(Ferdous et al., 18 Apr 2026)	Fractional time saved	F1-score retention	Product
(Waerebeke et al., 24 Feb 2025)	Step count reduction	Target excess risk achieved	Inverse prod.
(Lizzo et al., 2024)	n/a (apply to LLMs)	Forget/retain accuracy	Geometric mean
(Kim et al., 29 Aug 2025)	Param/time efficiency	Final utility	Weighted geom
(Zhao et al., 4 Dec 2025)	GPU-hr reduction	Attack success reduction	Ratio
(Feng et al., 9 Apr 2025)	Update step reduction	MRD-/difficulty-normalized	Normalized ratio

4. Theoretical Properties and Phase Regimes

Unlearning efficiency metrics provide the foundation for rigorous complexity-theoretic results. In phase diagrams for convex settings, three regimes are established (Waerebeke et al., 24 Feb 2025):

Trivial regime: Unlearning is as simple as adding noise, yielding negligible computational cost ( $T_\mathrm{shard}$ 9).
Efficient regime: Unlearning is strictly less costly than retraining ( $\Delta F_1$ 0), and UES is high.
Impossible regime: Unlearning cannot be faster than retraining ( $\Delta F_1$ 1), UES approaches baseline.

These regimes are explicit functions of problem dimension, fraction of data to forget, and the tightness of privacy constraints, directly visible in the analytic form of $\Delta F_1$ 2.

5. Practical Evaluation and Benchmarking Protocols

Empirical measurement of UES requires careful control of confounding experimental factors. Benchmarking protocols from leading competitions (Triantafillou et al., 2024) integrate:

Empirical forgetting quality, typically measured by membership-inference attacks or statistical distinguishability from ground-truth retrained models.
Model utility post-unlearning, via test/retain accuracy.
Resource consumption, standardized by wall-clock time, GPU-hours, number of optimization steps, or parameter footprint.

The NeurIPS Unlearning Competition standardized a composite UES ("Final score") as the product of forgetting quality, utility retention, and test accuracy ratios, bounded in $\Delta F_1$ 3:

$\Delta F_1$ 4

This procedure preserved method ranking across instantiations and allowed practical trade-off visualization for SOTA algorithms (Triantafillou et al., 2024).

6. Efficiency Score Extensions: Sample Difficulty and Influence

Recent advances incorporate sample-level properties into efficiency scoring. Memory Removal Difficulty (MRD) (Feng et al., 9 Apr 2025) quantifies per-sample unlearning hardness as the local curvature of the log-likelihood loss landscape, supporting curriculum sampling and difficulty-normalized UES:

$\Delta F_1$ 5

where $\Delta F_1$ 6, $\Delta F_1$ 7 the number of updates to unlearn, and $\Delta F_1$ 8 the utility degradation. This formally links inherent instance difficulty to realized efficiency gains.

Influence-based reductions (Kleiman et al., 4 Dec 2025, Zhao et al., 4 Dec 2025) propose disregarding low-influence points in the forget set, yielding empirical time savings up to 50% with no accuracy penalty. The associated UES is computed as the fraction of baseline runtime saved.

7. Limitations and Interpretational Considerations

UES metrics, while powerful synthesis tools, are sensitive to:

The definition and weight given to “quality” versus “efficiency,” especially as absolute losses approach zero or task baselines diverge.
Underlying resource models (e.g., wall-time, GPU-hours versus FLOPs), which may confound fair comparisons across hardware.
Scale: for extremely small forget sets, UES can be inflated without meaningful privacy gain unless quality terms are properly normalized (Zhao et al., 4 Dec 2025, Feng et al., 9 Apr 2025).

Additionally, UES does not diagnose whether a method achieves formal $\Delta F_1$ 9-unlearning guarantees, and should be interpreted in parallel with rigorous compliance and attack-based audits.

In summary, the Unlearning Efficiency Score and its variants constitute a core analytic and empirical apparatus for assessing, ranking, and optimizing machine unlearning algorithms. Rigorous constructions and phase-theoretic analysis coexist with pragmatic, domain-adapted variants, collectively advancing the understanding of privacy-preserving, cost-effective model editing (Ferdous et al., 18 Apr 2026, Waerebeke et al., 24 Feb 2025, Feng et al., 9 Apr 2025, Kleiman et al., 4 Dec 2025, Lizzo et al., 2024, Zhao et al., 4 Dec 2025, Kim et al., 29 Aug 2025, Wen et al., 13 Aug 2025, Ullah et al., 2023, Triantafillou et al., 2024).