ErasureBench-H: Dual-Domain Benchmarking

• ErasureBench-H is a polysemous benchmark that evaluates multi-concept erasure in diffusion models via a hierarchical brand→series→character taxonomy and serves as a benchmark harness in quantum error correction.
• In diffusion-model safety, it uses a taxonomy-only resource to generate evaluation images with metrics such as harmonic accuracy and FID, addressing limitations of flat-label datasets.
• In quantum error correction, it standardizes evaluation protocols for decoders, circuit conversion, finite-size threshold estimation, and hardware performance through erasure-aware metrics.

ErasureBench-H is a polysemous designation rather than a single universally standardized benchmark. In diffusion-model safety, it is explicitly introduced as a hierarchical benchmark for multi-concept erasure with a brand→series→character taxonomy, designed to evaluate erasure at multiple semantic granularities and under growing per-generation erasure subset size (Liu et al., 25 Sep 2025). In quantum error-correction literature, the same label is used in a benchmark-oriented sense for erasure-channel evaluation of quantum LDPC decoders, erasure-aware circuit conversion, finite-size threshold estimation, and hardware characterization, but those papers present reporting protocols and benchmark axes rather than one shared canonical artifact (Gökduman et al., 2024, Bhave et al., 7 Feb 2026, Pandey, 19 Mar 2026, Mai et al., 29 Jan 2026). This split usage is further sharpened by later concept-erasure papers that explicitly state that they do not define a component named “ErasureBench-H,” instead introducing M-ErasureBench or EraseBENCH as separate benchmarks (Weng et al., 28 Dec 2025, Amara et al., 16 Jan 2025).

1. Nomenclature and domain-specific meanings

The term has acquired two technically distinct meanings. In the diffusion-model literature, ErasureBench-H denotes a concrete hierarchical evaluation benchmark whose purpose is to stress-test multi-concept erasure under nested semantic structure and varying concept scope (Liu et al., 25 Sep 2025). In the quantum-erasure literature, the same name functions more as a benchmark harness or benchmark target: papers specify decoder inputs, metrics, fair baselines, finite-size scaling protocols, and hardware axes “for ErasureBench-H,” but do not define a single shared dataset or software release (Gökduman et al., 2024, Bhave et al., 7 Feb 2026, Pandey, 19 Mar 2026, Mai et al., 29 Jan 2026).

Context	Meaning of “ErasureBench-H”	Source
Diffusion-model concept erasure	A hierarchical benchmark with brand→series→character structure	(Liu et al., 25 Sep 2025)
Quantum erasure benchmarking	A benchmark-oriented harness for erasure-aware QEC evaluation	(Gökduman et al., 2024, Bhave et al., 7 Feb 2026, Pandey, 19 Mar 2026, Mai et al., 29 Jan 2026)
Related but distinct names	M-ErasureBench and EraseBENCH explicitly state that no “ErasureBench-H” is defined there	(Weng et al., 28 Dec 2025, Amara et al., 16 Jan 2025)

A common misconception is that ErasureBench-H names a single benchmark spanning all erasure-related work. The record in the cited papers does not support that interpretation. A more precise reading is that the term is stable only within specific subfields, and even there its scope differs substantially (Liu et al., 25 Sep 2025, Weng et al., 28 Dec 2025, Amara et al., 16 Jan 2025).

2. Hierarchical benchmark for multi-concept erasure in diffusion models

In its formalized diffusion-model sense, ErasureBench-H was introduced to address a limitation of flat-label datasets such as CIFAR-10, CIFAR-100, and Imagenette: they do not represent the nested concept structure that arises in practical takedown requests, where a request may target a character, a series, or an entire brand (Liu et al., 25 Sep 2025). The benchmark therefore defines a three-level taxonomy of Brand → Series → Character, with character as the unit concept, and introduces the notion of concept scope, defined as the number of unit concepts subsumed by a named concept.

The released benchmark is a taxonomy-only resource: a CSV hierarchy with 27 brands, 73 series, and 300 characters, with no images distributed (Liu et al., 25 Sep 2025). Evaluation images are instead generated from Stable Diffusion v1.4 during experiments. The paper gives representative examples including Disney, DC, and Warner Bros. at the brand level; series such as The Lion King, Mickey Mouse Clubhouse, Duffy and Friends, Justice League, and Shazam!; and characters such as Simba, Timon, Mickey Mouse, Minnie Mouse, Duffy, ShellieMay, Batman, Wonder Woman, and Shazam (Liu et al., 25 Sep 2025).

The prompt design is intentionally minimal so that the erased subset remains explicit. The standard template is “a photo of the {target concepts},” with conjunction prompts such as “a photo of the beaver, dolphin, and otter” to scale the per-generation erased subset size (Liu et al., 25 Sep 2025). Two multi-concept settings are distinguished. In scope scaling, the model is trained to erase an increasing number of concepts but evaluated on single-target generation. In per-generation subset scaling, the subset size grows either through conjunctions with $N \in \{2,3,4,5\}$ or by requesting a higher-level concept whose scope equals the number of unit characters it subsumes (Liu et al., 25 Sep 2025).

Series-level and brand-level targets are aggregated by scope tertiles into Small, Medium, and Large buckets, computed separately per level (Liu et al., 25 Sep 2025). This design makes semantic relatedness operational rather than anecdotal: conjunction experiments can be formed from contiguous class tuples or hierarchical groupings, while scope-based evaluations intrinsically involve related targets under the same parent node.

3. Evaluation formalism, metrics, and aggregation

ErasureBench-H defines the universe of unit concepts $U$ with $|U|=300$. A target concept $t$ at level $$L \in \{\text{character}, \text{series}, \text{brand}\}$$ corresponds to a set of unit concepts $S(t) \subseteq U$, with $|S(t)|=1$ at character level. For each generated image $x$, $g_K(x) \subseteq U$ denotes the set of $K$ unit concepts returned by a CLIP-based concept-presence classifier using top-$U$0 CLIP logits at unit-concept granularity (Liu et al., 25 Sep 2025).

Erasure effectiveness is defined by residual target presence:

$U$1

The erasure success rate is

$U$2

Utility preservation is setting-specific. For conjunction prompts with per-generation erased subset size $U$3, the benchmark samples $U$4 non-target unit concepts $U$5 and requires the top-$U$6 predictions to contain all $U$7 sampled non-targets. For concept-scope expansion at series or brand level, it requires the top-5 unit predictions to fall within the corresponding non-target groups outside the target’s brand or series (Liu et al., 25 Sep 2025). Harmonic accuracy combines erasure success and utility preservation:

$U$8

Image fidelity is reported with FID against MS-COCO:

$U$9

At character level, $|U|=300$0, $|U|=300$1, $|U|=300$2, and FID are computed per character and then averaged. At series and brand levels, targets are bucketed by scope tertiles and metrics are averaged within each bucket (Liu et al., 25 Sep 2025).

The benchmark fixes a concrete generation protocol: Stable Diffusion v1.4, DDIM 50-step sampler, classifier-free guidance at inference, and typically 200 generated images per method for each target set (Liu et al., 25 Sep 2025). The use of top-$|U|=300$3 CLIP predictions is task-dependent. The general protocol uses top-$|U|=300$4 logits for conjunction prompts, while the appendix tables for ErasureBench-H conjunction and series/brand results use top-5 (Liu et al., 25 Sep 2025).

4. Empirical role in DyME and observed scaling behavior

ErasureBench-H was used to evaluate DyME, which trains concept-specific LoRA adapters and dynamically composes only the required adapters at inference (Liu et al., 25 Sep 2025). The benchmark is central to DyME’s claim of scalability because it exposes the interference that arises when erased sets become large or semantically clustered. Static baselines in the reported experiments include Stable Diffusion without erasure and static erasure methods ESD, AC, FMN, MACE, SPM, and SalUn; the strongest static baselines, especially MACE, SPM, and SalUn, are carried forward to the more difficult multi-concept studies (Liu et al., 25 Sep 2025).

On conjunction-based subset scaling, DyME maintains the highest reported harmonic accuracy as $|U|=300$5 increases. At $|U|=300$6, DyME reports $|U|=300$7, $|U|=300$8, and $|U|=300$9, compared with SPM’s $t$0, $t$1, and $t$2, MACE’s $t$3, $t$4, and $t$5, and SalUn’s $t$6, $t$7, and $t$8 (Liu et al., 25 Sep 2025). At $t$9, DyME reports $$L \in \{\text{character}, \text{series}, \text{brand}\}$$0, $$L \in \{\text{character}, \text{series}, \text{brand}\}$$1, and $$L \in \{\text{character}, \text{series}, \text{brand}\}$$2, compared with SPM’s $$L \in \{\text{character}, \text{series}, \text{brand}\}$$3, $$L \in \{\text{character}, \text{series}, \text{brand}\}$$4, and $$L \in \{\text{character}, \text{series}, \text{brand}\}$$5, MACE’s $$L \in \{\text{character}, \text{series}, \text{brand}\}$$6, $$L \in \{\text{character}, \text{series}, \text{brand}\}$$7, and $$L \in \{\text{character}, \text{series}, \text{brand}\}$$8, and SalUn’s $$L \in \{\text{character}, \text{series}, \text{brand}\}$$9, $S(t) \subseteq U$0, and $S(t) \subseteq U$1 (Liu et al., 25 Sep 2025). An ablation without orthogonality degrades sharply; for example, at $S(t) \subseteq U$2 it reports $S(t) \subseteq U$3, $S(t) \subseteq U$4, and $S(t) \subseteq U$5, which the paper uses as evidence of adapter interference without orthogonal constraints (Liu et al., 25 Sep 2025).

The concept-scope evaluations reveal the same pattern across semantic levels. At character level, DyME reports $S(t) \subseteq U$6, $S(t) \subseteq U$7, $S(t) \subseteq U$8, and $S(t) \subseteq U$9, compared with SPM’s $|S(t)|=1$0, $|S(t)|=1$1, $|S(t)|=1$2, and $|S(t)|=1$3, MACE’s $|S(t)|=1$4, $|S(t)|=1$5, $|S(t)|=1$6, and $|S(t)|=1$7, and SalUn’s $|S(t)|=1$8, $|S(t)|=1$9, $x$0, and $x$1 (Liu et al., 25 Sep 2025). At series level, DyME remains strongest across Small, Medium, and Large buckets, although absolute utility preservation decreases with scope. At brand level, performance is hardest overall: for Large brands, DyME reports $x$2, $x$3, and $x$4, while MACE reports $x$5, $x$6, and $x$7, SPM reports $x$8, $x$9, and $g_K(x) \subseteq U$0, and SalUn reports $g_K(x) \subseteq U$1, $g_K(x) \subseteq U$2, and $g_K(x) \subseteq U$3 (Liu et al., 25 Sep 2025).

The DyME paper attributes these gains to bi-level orthogonality constraints. Its final training objective is

$g_K(x) \subseteq U$4

with one orthogonality term defined on induced representation shifts and the other on LoRA-parameter interactions (Liu et al., 25 Sep 2025). A plausible implication is that ErasureBench-H functions not merely as a dataset taxonomy but as a structured interference testbed: it makes the failure mode of static multi-concept erasure quantitatively observable as concept scope expands.

5. Quantum-erasure usage: benchmark harness, protocols, and reporting axes

In quantum error correction, ErasureBench-H appears as a benchmark-oriented framework for evaluating erasure-aware decoders, erasure-circuit conversions, finite-size threshold estimation, and hardware metrics rather than as one fixed benchmark release. The central shared assumption is a flagged erasure model in which erasure locations are known to the decoder, but the benchmark objects differ by layer: code-capacity decoding, circuit-level erasure compilation, finite-size scaling, and hardware-efficient erasure detection (Gökduman et al., 2024, Bhave et al., 7 Feb 2026, Pandey, 19 Mar 2026, Mai et al., 29 Jan 2026).

At the decoder level, the BPGD erasure-decoding work specifies an ErasureBench-H-style evaluation for CSS quantum LDPC codes under independently flagged erasures with probability $g_K(x) \subseteq U$5, where $g_K(x) \subseteq U$6- and $g_K(x) \subseteq U$7-type decoding are carried out separately on $g_K(x) \subseteq U$8 and $g_K(x) \subseteq U$9, success includes convergence to a degenerate codeword, and recommended comparisons include peeling, pruned peeling, BPGD, BPGD+damping, BPGD+adjusted LLRs, VH for HGP codes, and ML restricted to the erased set (Gökduman et al., 2024). It recommends reporting failure rate versus $K$0, average iterations, runtime, threshold estimates, and waterfall slopes, and gives practical defaults such as $K$1, $K$2, fixed $K$3 per round, one-variable-per-round decimation by $K$4, and degeneracy-resolution counted as success (Gökduman et al., 2024).

At the circuit-conversion level, BiBiEQ positions itself as an erasure-aware compilation, conversion, and evaluation pipeline for Bivariate Bicycle codes that “maps naturally onto an erasure-focused harness such as ErasureBench-H” (Bhave et al., 7 Feb 2026). It produces erasure-aware memory circuits $K$5, converts them into decoder-ready stabilizer circuits $K$6 through Exact and Approx engines, and reports per-round logical error rate

$K$7

pseudo-threshold $K$8 defined by $K$9, engine gap $U$00, growth rate $U$01, and subthreshold scaling

$U$02

The framework emphasizes schedule dependence: 4EC keeps Approx close to Exact, with median $U$03 falling from roughly 4–8 under 2EC to about 1.2–1.3 under 4EC, and distance scaling below threshold shows that the hop $U$04 yields a $U$05–$U$06 reduction in per-round LER, whereas $U$07 yields only $U$08–$U$09 (Bhave et al., 7 Feb 2026).

At the finite-size threshold level, the BB-code threshold study codifies a fairness and reproducibility standard for an erasure benchmark: the primary metric is word error rate, $U$10, with failure if either $U$11 or $U$12 sector fails; pseudo-threshold $U$13 is the $U$14 crossing; asymptotic threshold is estimated by finite-size scaling

$U$15

Using five BB code sizes from $U$16 to $U$17, $U$18 shots per point, and bootstrap confidence intervals, it reports finite-size pseudo-thresholds from $U$19 to $U$20, an asymptotic threshold $U$21, and $U$22, while arguing that uninformed depolarizing-weight MWPM is an unfair baseline on the erasure channel because it behaves like random guessing in the tested regime (Pandey, 19 Mar 2026).

At the hardware level, the biased-erasure cavity-qubit work proposes a standardized axis set for ErasureBench-H: erasure bias $U$23, detection quality via false-positive and false-negative rates, postselected logical relaxation and dephasing, gate error under interleaved erasure detection, and resource overhead (Mai et al., 29 Jan 2026). In the reported single-cavity $U$24, $U$25 encoding, the erasure bias ratio exceeds $U$26, over $U$27 of leakage events are converted into detected erasures, postselected logical relaxation and dephasing rates are below the erasure rate by factors of $U$28 and $U$29, and interleaved randomized benchmarking reports a residual logical gate error of $U$30 (Mai et al., 29 Jan 2026). This suggests that, in the quantum literature, ErasureBench-H is best understood as a layered benchmarking vocabulary spanning algorithmic, circuit, finite-size, and hardware regimes rather than as one monolithic benchmark.

6. Terminological divergence and neighboring benchmarks

The ambiguity of the name is made explicit by neighboring concept-erasure benchmarks. M-ErasureBench introduces a multimodal framework for concept erasure in text-to-image diffusion models across text prompts, learned embeddings, and inverted latents, with five evaluation scenarios, and states directly that the paper does not define a component named “ErasureBench-H” (Weng et al., 28 Dec 2025). Similarly, EraseBENCH presents a multi-dimensional benchmark centered on concept entanglement and ripple effects, covering more than 100 concepts and more than 1,000 tailored prompts, and also states that no “ErasureBench-H” or “Hard” split is defined there (Amara et al., 16 Jan 2025).

The distinction is substantive, not merely nominal. EraseBENCH is organized around visually similar, binomial, stylistically similar, and subset-superset relations; M-ErasureBench is organized around input modalities and white-box versus black-box access; ErasureBench-H in DyME is organized around hierarchical concept scope; and the quantum literature uses the term for erasure-channel benchmarking protocols rather than for one concept-erasure dataset (Amara et al., 16 Jan 2025, Weng et al., 28 Dec 2025, Liu et al., 25 Sep 2025, Gökduman et al., 2024). This suggests that any use of the label without domain qualification is liable to misidentify the benchmark being referenced.

For technical writing, the least ambiguous practice is to specify the domain explicitly: “ErasureBench-H” for the hierarchical benchmark of multi-concept erasure in diffusion models (Liu et al., 25 Sep 2025), or “ErasureBench-H-style quantum erasure benchmark” when referring to the benchmark harness used in quantum LDPC decoding, BB-code threshold estimation, or biased-erasure hardware reporting (Gökduman et al., 2024, Bhave et al., 7 Feb 2026, Pandey, 19 Mar 2026, Mai et al., 29 Jan 2026).