Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

Abstract: Training-free concept erasure is an attractive mechanism for controlling text-to-image diffusion models, but precise erasure often comes at the cost of damaging semantically related non-target concepts. Existing value-space methods remove the component of each cross-attention value along the target concept direction, implicitly treating target identity and shared visual structure as the same signal. We argue that this is the source of much of the collateral damage in prior preservation. We introduce CARE, a closed-form concept erasure operator that replaces the raw target direction with a kept-subspace-aware direction computed from a small bank of retained concept anchors. The resulting edit is applied directly in cross-attention value space, requires no model fine-tuning, and adds only a negligible offline computation. A single shrinkage parameter controls the erase-preserve trade-off. We further show that the operator admits a minimum-disturbance interpretation and, in its projection form, leaves the kept subspace invariant. Experiments under the standard concept-erasure protocol show that our method preserves non-target concepts more faithfully while maintaining competitive erasure across instance, style, and celebrity concepts. Code: https://github.com/parthupman/care

• The paper presents CARE, a novel method that uses an anchor bank and covariance-aware direction computation to precisely remove target concepts.
• CARE leverages low-rank Woodbury inversion and a tunable shrinkage parameter to balance the erasure of unwanted attributes with the preservation of related features.
• Empirical evaluations across instance, style, and celebrity erasure tasks demonstrate that CARE improves target removal and reduces collateral damage compared to existing methods.

Covariance-Aware Concept Erasure in Diffusion Models

Problem Formulation and Methodological Innovations

This paper addresses the core challenge in controllable generative models: to surgically erase a target concept from a text-to-image diffusion model at inference-time without degrading generative fidelity on semantically related, non-target concepts. Current training-free interventions, particularly value-space erasure (as in AdaVD), remove cross-attention value components aligned with the target concept. However, the raw direction obtained from target conditioning often mixes target-specific attributes with high-variance shared structure (e.g., visual priors, style, identity manifolds). This conflation leads to undesirable collateral damage—non-target concepts with overlapping structure are suppressed alongside the intended target.

The paper introduces CARE (Covariance-Aware Retained-subspace Erasure). CARE modifies the value-space erasure protocol by constructing an anchor bank of retained concepts and leveraging their value-space covariance to compute an erase direction $$d = \left(\frac{1}{M}B^\top B + \gamma I\right)^{-1}t$$. This direction down-weights high-variance components shared with the retained anchors while preserving discriminative target-specific structure. The shrinkage parameter $\gamma$ controls the erase–preserve trade-off, interpolating between raw target erasure and projection onto the orthogonal complement of the retained subspace. The entire computation is efficiently performed using low-rank Woodbury inversion, incurring negligible overhead. CARE evaluates cross-attention values for each prompt token and applies a gated rank-one subtractive edit; no prompt modification or weight editing is required.

Figure 1: CARE overview: covariance-informed direction selection preserves retained concepts while erasing target-specific structure within the cross-attention value space.

Quantitative and Qualitative Evaluation

CARE is thoroughly evaluated on Stable Diffusion v1.4 across instance, style, and celebrity concept removal. Performance is measured using CLIP score (CS) for erased targets and FID for retained concepts. CARE consistently improves the erase–preserve operating point relative to AdaVD and other baselines. For instance erasure tasks (e.g., Snoopy), CARE reduces target CS and further lowers FID for Mickey, Spongebob, and other retained probes, indicating superior preservation. Multi-concept erasure demonstrates the nuanced trade-off—while erasure strength for highly entangled targets (e.g., Mickey) may slightly decrease, collateral damage is substantially mitigated as evidenced by retained-concept FID.

In the art style domain, CARE delivers both maximal target removal and robust retention across non-target styles and held-out artists. For Van Gogh erasure, CARE outperforms AdaVD in both CS and FID across Picasso, Monet, and Caravaggio. Celebrity erasure tests the limits of preservation, given shared face manifolds. CARE achieves lower CS for erased identities and reduces FID for the remaining celebrities, exemplifying its discriminative effectiveness in settings where prior methods degrade the full face prior.

Figure 2: CARE preserves bystander identities during celebrity erasure, minimizing unintended suppression in multi-concept settings.

Figure 3: CARE maintains non-target styles and character identity, with qualitative distinction against AdaVD and baseline methods.

Efficiency tests show CARE is nearly cost-equivalent to AdaVD in inference, requiring only a brief offline anchor bank construction. Ablations reveal that $\gamma$ tunes the erase–preserve boundary, with small $\gamma$ risking under-erasure and large $\gamma$ reverting to standard value-space suppression. The optimal bank composition is disjoint from the target; related anchors weaken selectivity and erasure strength.

Theoretical Properties

CARE admits a minimum-disturbance interpretation: it solves for the closest edit to a value vector that minimizes its projection onto the erasure direction. The method is invariant to nonzero scaling of $d$ and recovers standard value-space erasure as $\gamma \to \infty$. In the projection limit ($\gamma \to 0$), CARE approaches hard removal orthogonal to the retained anchor subspace, ensuring invariance for vectors lying entirely in the retained span. The closed-form operator enables multi-target extension via orthonormalized direction summation.

Implications and Future Directions

CARE provides a selective, training-free mechanism for disabling unwanted concepts in generative models, preserving model utility on non-targets. The methodological advance—using covariance-aware directionality—shifts the paradigm from brute-force suppression to discriminative editing, which is particularly relevant for downstream deployment in sensitive contexts (copyright management, identity protection, safety enforcement). CARE’s closed-form, inference-time nature facilitates rapid adaptation without retraining or checkpoint proliferation.

Future developments in AI safety and generative model control are likely to expand this framework. Extensions could integrate richer representations of retained anchors, adaptive bank construction from unlabeled data, and dynamic direction selection conditioned on prompt context. The theoretical groundwork in minimum-disturbance projection and Fisher/Mahalanobis-style discrimination may inform similar interventions in foundation models for language and multimodal generation. As concept erasure becomes increasingly nuanced and high-dimensional, covariance-informed strategies will be critical for maintaining generative reliability in real-world applications.

Conclusion

CARE reframes concept erasure in diffusion models as the problem of selecting the precise direction to forget—down-weighting shared structure and preserving non-target fidelity. Empirical evidence demonstrates matched or improved erasure and significantly reduced collateral damage across diverse categories. The method operates with closed-form efficiency and exposes a principled erase–preserve trade-off via shrinkage tuning. CARE establishes a new baseline for practical, selective, and scalable generative AI safety interventions.