M-ErasureBench: Multimodal Concept Erasure

• The paper introduces M-ErasureBench, a multimodal framework that benchmarks concept erasure across text prompts, learned embeddings, and latent inversion.
• It reveals that traditional methods may yield concept reproduction rates over 90% under adversarial scenarios, highlighting significant security gaps.
• The study presents IRECE, a plug‐and‐play module that reduces CRR by up to 57.8% in white‐box settings, thus enhancing inference-time robustness without retraining.

M-ErasureBench is a comprehensive multimodal evaluation framework for benchmarking concept erasure in text-to-image diffusion models. Developed by Weng et al., M-ErasureBench addresses the critical issue that existing concept erasure methods—originally designed to suppress unwanted or copyrighted concepts—are validated almost exclusively in the text-prompt setting, despite the widespread use of alternative inputs like learned embeddings and inverted latents in practical deployments. M-ErasureBench fills this methodological gap by systematically evaluating erasure performance across these modalities and attack scenarios, revealing significant vulnerabilities and introducing the IRECE module to enhance robustness at inference time (Weng et al., 28 Dec 2025).

1. Motivation and Scope of M-ErasureBench

The impetus behind M-ErasureBench is the observation that cross-attention finetuning-based erasure methods (such as ESD, UCE, and Receler) effectively suppress target concepts from standard text prompts—e.g., preventing generation when conditioned on “A photo of TARGET.” However, more sophisticated attacks leveraging learned embeddings (e.g., Textual Inversion) or inverted latents (e.g., DDIM inversion applied to real images) frequently circumvent existing defenses with high success rates. These attack surfaces are relevant in applications involving image editing, personalization, or unauthorized reproduction of protected content, necessitating a benchmark that systematically investigates concept erasure beyond text-only input.

2. Input Modalities and Evaluation Scenarios

M-ErasureBench rigorously evaluates concept erasure over three primary input modalities:

• Text Prompts: Direct conditioning via textual instructions.
• Learned Embeddings: Features obtained through Textual Inversion (TI) to encode visual concepts.
• Inverted Latents: Latent codes acquired via DDIM inversion from real-world or counterfactual images.

Each modality is assessed under five evaluation scenarios that reflect distinct adversarial capabilities:

Modality	Scenario Description
Text Prompts	Standard prompt, possibly adversarial (Ring-A-Bell)
Learned Embeddings	TI: white-box, black-box, black-box+perturbation
Inverted Latents	DDIM inversion: white-box, black-box

• White-box TI: Embeddings are learned directly on the erased model.
• Black-box TI: Embeddings learned on a separate, standard model then transferred.
• Black-box TI+perturbation: Embeddings learned after adding noise to the reference image.
• White-box Latent Inversion: DDIM inversion and sampling both performed on erased model.
• Black-box Latent Inversion: DDIM inversion performed only on standard model; sampling on erased.

3. Core Evaluation Metrics: Concept Reproduction Rate (CRR)

The principal metric employed by M-ErasureBench is the Concept Reproduction Rate (CRR), quantifying the fraction of generated images in which a pretrained object detector (GroundingDINO) identifies the target concept. Explicitly, for $N$ generated images $\{x_i^*\}_{i=1}^N$,

$$y_i = \begin{cases} 1 & \text{if GroundingDINO detects TARGET in } x_i^*,\ 0 & \text{otherwise,} \end{cases}$$

$$\mathrm{CRR} = \frac{1}{N} \sum_{i=1}^N y_i \quad(\%)$$

Lower CRR values indicate stronger concept removal. The detector is applied across all modalities and scenarios, enabling quantitative comparison of erasure robustness.

4. Experimental Findings Across Modalities

M-ErasureBench reveals stark contrasts in erasure effectiveness depending on input modality and attack scenario:

• Text Prompts:
  • Baseline Stable Diffusion v1.4: CRR ≈ 96 %.
  • ESD: 26.5 %, UCE: 18.5 %, Receler: 15.0 % (for “An image of TARGET”).
  • Adversarial prompts (Ring-A-Bell): ESD/ UCE CRR jump to 66.7 %, 36.9 %, Receler remains robust at 14.8 %.
• Learned Embeddings:
  • White-box TI: CRR ≈ 91.1 % (ESD), 90.4 % (UCE), 56.0 % (Receler).
  • Black-box TI: CRR ≈ 41.2 % (ESD), 35.7 % (UCE), 12.4 % (Receler).
  • Black-box TI+perturbation: CRR ≈ 74.0 % (ESD), 54.2 % (UCE), 15.1 % (Receler).
• Latent Inversion:
  • White-box: All methods, CRR > 92 % (near-complete erasure failure).
  • Black-box: ESD: 57.3 %, UCE: 95.2 %, Receler: 79.0 %.

These data demonstrate that concept erasure validated under text-prompt settings does not generalize: learned embeddings and inverted latents can robustly bypass state-of-the-art methods, with CRR exceeding 90 % under several attack vectors. This suggests a severe blind spot for current protective generative modeling practices.

5. IRECE: Inference-time Robustness Enhancement

To address multimodal vulnerabilities, Weng et al. introduce IRECE (Inference-time Robustness Enhancement for Concept Erasure), a plug-and-play module requiring no retraining. IRECE operates through four key steps:

1. Concept Localization via Cross-attention: At a selected denoising step $t^*$, the latent $\mathbf{x}_t$ is fed to a standard (non-erased) diffusion model under the target prompt $\mathbf{c}_{\mathrm{tgt}}$. Cross-attention maps $A_{\mathrm{cross}}^\ell$ are extracted per layer and aggregated:

$$\mathbf{A} = \sum_{\ell=1}^L \mathrm{Upsample}\bigl(A_{\mathrm{cross}}^\ell(\mathbf{x}_t, \mathbf{c}_{\mathrm{tgt}})\bigr)$$

2. Mask Creation: A binary mask $\mathbf{M}$ is defined by thresholding $\mathbf{A}$ at $\tau$:

$$\mathbf{M}(i,j)= \begin{cases} 1,&\mathbf{A}(i,j)\ge\tau\ 0,&\text{otherwise} \end{cases}$$

3. Local Latent Perturbation: The masked regions of $\mathbf{x}_t$ are replaced by Gaussian noise $$\boldsymbol\xi_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$:

$$\mathbf{x}_t^* = (1-\mathbf{M})\odot \mathbf{x}_t + \mathbf{M}\odot \boldsymbol\xi_t$$

4. Resume Denoising: Sampling resumes with the erased model $\theta_{\mathrm{era}}$:

$$\mathbf{x}_{t-1} = \begin{cases} \mathrm{DDIMStep}(\mathbf{x}_t^*, \mathbf{c}_{\mathrm{sam}}, t, \theta_{\mathrm{era}}) & t = t^*\ \mathrm{DDIMStep}(\mathbf{x}_t, \mathbf{c}_{\mathrm{sam}}, t, \theta_{\mathrm{era}}) & \text{otherwise} \end{cases}$$

IRECE requires only a single forward pass on the standard model at $t^*$, followed by masking and DDIM sampling on the erased model. Fine-tuning of model weights is not required.

6. Impact of IRECE and Practical Implications

Empirical results demonstrate that IRECE restores erasure robustness even under white-box latent-inversion with null prompts: CRR drops by up to 57.8 % (ESD), 52.2 % (Receler), and 40.5 % (UCE), with representative drops for ESD 92.9 % → 35.1 %, UCE 94.9 % → 54.4 %, and Receler 94.4 % → 42.2 %. In less challenging black-box settings, IRECE reduces CRR by over 20 % in most configurations. Qualitatively, it localizes and erases only the concept regions (e.g., an airplane), with plausible background infilling and preservation of visual coherence. Perceptual evaluation by human observers indicates that images outside the suppressed regions remain indistinguishable from references.

A plausible implication is that M-ErasureBench provides both a rigorous multimodal validation protocol and a practical mitigation (IRECE) for safeguarding generative models against persistent concept leakage across diverse attack surfaces. Developers of “forbidden category” or content-restricted diffusion systems are now equipped with a systematic toolchain for robust validation and inference-time protection, without refitting core model architectures (Weng et al., 28 Dec 2025).

7. Conclusions and Broader Significance

M-ErasureBench advances the field by establishing that text-prompt-centric concept erasure does not generalize to the heterogeneous interfaces encountered in real-world model deployment. By operationalizing three erasure algorithms across five modalities—including previously untested avenues such as learned embeddings and inverted latents—M-ErasureBench exposes the fragility of existing methods and the necessity of multimodal validation. The introduction of IRECE marks a paradigm shift in protective inference, leveraging spatial attention, targeted masking, and noise injection to dynamically suppress concept reproduction. The framework and methodology set by M-ErasureBench are poised to inform future research on robust, multimodal safe generative modeling.