You Only Erase Once: Erasing Anything without Bringing Unexpected Content

Abstract: We present YOEO, an approach for object erasure. Unlike recent diffusion-based methods which struggle to erase target objects without generating unexpected content within the masked regions due to lack of sufficient paired training data and explicit constraint on content generation, our method allows to produce high-quality object erasure results free of unwanted objects or artifacts while faithfully preserving the overall context coherence to the surrounding content. We achieve this goal by training an object erasure diffusion model on unpaired data containing only large-scale real-world images, under the supervision of a sundries detector and a context coherence loss that are built upon an entity segmentation model. To enable more efficient training and inference, a diffusion distillation strategy is employed to train for a few-step erasure diffusion model. Extensive experiments show that our method outperforms the state-of-the-art object erasure methods. Code will be available at https://zyxunh.github.io/YOEO-ProjectPage/.

• The paper presents YOEO, a distilled diffusion model that effectively erases objects without introducing unintended hallucinations.
• It incorporates sundries suppression and entity feature coherence losses to maintain semantic consistency and context in erased regions.
• Empirical evaluations demonstrate YOEO’s superior fidelity, faster inference, and robust cross-domain generalization compared to prior methods.

Object Erasure via Distilled Diffusion and Contextual Consistency

Introduction

The paper "You Only Erase Once: Erasing Anything without Bringing Unexpected Content" (2603.27599) addresses inherent limitations in diffusion-based object erasure, specifically the common issue of generating unintended, hallucinated content within erased regions. The work presents YOEO, a few-step distilled diffusion model that achieves high-fidelity object removal without introducing sundries (unwanted and unintended objects) or breaking semantic consistency. Key contributions include a novel training pipeline leveraging unpaired real-world data, a sundries detector with associated suppression loss, and an entity feature coherence (EFC) loss to ensure contextual consistency.

Figure 1: Overview of the YOEO framework showing dataset utilization and the integration of erasure-specific losses for efficient, coherent completion.

Limitations of Previous Approaches

Diffusion models have largely displaced GAN-based methods for image inpainting and object erasure, offering marked improvements in fidelity and detail. However, existing methods often rely on synthetic paired datasets or pixel-level reconstruction losses, lacking explicit objectives to enforce artifact-free and semantically consistent erasure. These limitations manifest as:

• Generation of sundries in the erased regions due to absence of explicit negative supervision,
• Incoherence between completed and surrounding regions arising from insufficient context modeling,
• High computational cost and latency in closed-source multimodal models, which, while powerful, are unsuitable for edge deployment.

Methodology

Dataset Construction

YOEO leverages a hybrid data construction approach:

• $\mathcal{D}_1$: Synthetic paired images, generated via random occlusions.
• $\mathcal{D}_2$: Unpaired real-world samples, where entity masks serve as erasure targets, but no ground-truth post-erasure image is available.

This structure promotes both fidelity (through paired data) and generalization to real erasure scenarios (through unpaired supervision).

Distilled Diffusion Framework

The approach distills a multi-step teacher denoiser (based on Stable Diffusion Inpainting 1.5) into a few-step student model for efficient object erasure. Distillation losses comprise:

• Standard output matching (LPIPS),
• Distribution matching distillation (DMD, minimizing KL divergence between teacher and student latents),
• GAN loss applied in latent space.

Entity-Coherent Erasure Supervision

Sundries Suppression Loss

A pretrained entity segmentor (Mask2Former) is used to detect sundries—entities whose IoS with the mask exceeds a set threshold—within the inpainted region. The sundries suppression loss penalizes generation of these objects, both at the mask and prediction levels, ensuring minimal introduction of unintended content.

Figure 2: Sundries detection pipeline. Entities overlapping with the inpainting mask are classified and penalized if likely to be sundries.

Entity Feature Coherence (EFC) Loss

The EFC loss enforces semantic and structural consistency between inpainted and surrounding regions by aligning segmentation feature distributions. Cosine similarity is computed between features inside the inpaint region and those corresponding to the same entity outside, promoting content harmonization without explicit ground-truth targets.

Joint Training Schedule

YOEO training involves three stages: (1) teacher fine-tuning on paired data, (2) student distillation, and (3) joint distillation with real-world unpaired data guided by SS and EFC losses. Training parameters are dynamically balanced using gradient normalization, ensuring stability across stages.

Empirical Evaluation

Quantitative Results

YOEO demonstrates substantial improvement in erasure quality, particularly in minimizing sundries and preserving context, as measured by MSN, MARS, and CFD metrics. While FID scores are competitive with state-of-the-art, YOEO's parameter efficiency (860M vs. 12B for some baselines) results in faster inference and practical deployment advantages.

Qualitative Analysis

Figure 3: Qualitative COCO results. YOEO removes objects with higher contextual fidelity and fewer sundries than baseline diffusion and transformer architectures.

YOEO noticeably outperforms methods such as MAT, CLIPAway, SmartEraser, and OmniPaint in removing objects cleanly, without leaving residual contours or introducing artifacts, even in complex scenes with high object density or occlusion.

Scenario and Domain Generalization

Figure 4: YOEO performance across multi-object, comic, sketch, and watercolor scenarios highlights significant cross-domain generalization.

YOEO robustly generalizes to non-photographic domains (comics, watercolor, sketches), exhibiting consistent erasure without degrading image harmonization—an attribute not mirrored by baselines.

Comparison with Guidance-Based and Multimodal Models

Figure 5: YOEO compared to condition-guided methods; entity segmentation and depth-based guidance baselines fail in challenging mask cases.

Figure 6: YOEO vs. closed-source multimodal models (ChatGPT-5, Nano Banana); YOEO avoids extra object hallucinations and maintains context consistency.

YOEO surpasses EntityErasure and GeoRemover in robustness to erroneous condition predictions and outperforms multimodal LMMs in sundries suppression and visual plausibility, despite smaller model size and lower compute footprint.

Ablation Studies

Disabling the EFC or SS loss significantly degrades sundries suppression and contextual consistency. Importantly, joint training on unpaired data $\mathcal{D}_2$ is essential; applying erasure-specific losses solely on synthetic pairs yields marginal improvements, but the combination ensures minimal sundries and contextual fidelity.

Implications and Future Directions

The integration of unpaired supervision, sundries detection, and feature-level context metrics into the few-step diffusion framework sets a new direction for object erasure: moving beyond pixel-level reconstruction to erasure-specific objectives aligned with real image semantics. For practical editing systems and edge deployment, this reduces hallucination risk and compute cost.

Theoretically, this work motivates exploration of further context-sensitive supervision and progressive distillation for other image editing and restoration tasks. Future extensions include:

• Incorporation of more advanced language and vision priors for user-controllable erasure,
• Cross-modal sundries detection combining vision with language or depth cues,
• Application to video object removal with spatiotemporal consistency constraints.

Conclusion

YOEO provides an efficient and robust framework for object erasure, minimizing sundries and ensuring contextually consistent outputs through novel entity-level losses and unpaired data supervision. Its architecture balances performance, stability, and deployment efficiency, and experimental results confirm advantages over both academic and large-scale industrial solutions. This paradigm shift from reconstruction to erasure-specific objectives is set to inform the next generation of image editing and inpainting methodologies.