- The paper presents an interactive, click-based approach that leverages self-attention in diffusion models for precise object removal.
- It integrates click-driven semantic mapping, staged attention modulation, and adaptive restoration guidance to enhance editing fidelity.
- Experimental results show competitive FID scores and strong user preference, validating its robustness across multiple diffusion backbones.
ClickRemoval: Interactive Object Removal via Attention Redirection in Diffusion Models
Motivation and Problem Statement
Object removal is a fundamental operation in digital media editing, crucial for privacy, content filtering, and visual enhancement tasks. Contemporary methods predominantly rely on mask-based or prompt-driven interfaces, often requiring considerable expertise or tedious manual input. These systems, such as AttentiveEraser and PowerPaint, either demand precise masks or specialized pipelines, impeding accessibility for non-expert users and leading to incomplete removals or suboptimal background synthesis. The paper "ClickRemoval: An Interactive Open-Source Tool for Object Removal in Diffusion Models" (2605.14461) systematically addresses these limitations by proposing a click-based, training-free object removal system built atop pretrained Stable Diffusion backbones.
Methodology
Click-Driven Semantic Mapping
ClickRemoval abandons explicit mask or textual input, introducing an interface where object removal is triggered solely by user clicks. The M2N2 semantic distance map module converts these clicks into soft spatial maps reflecting the semantic proximity of pixels to the click target. Flood Fill enhancement refines the object localization, producing Mob for suppression and Mbg for background reference. This conversion effectively transforms the click signal into structured localization and guidance maps, directly leveraging the self-attention tensors from the diffusion backbone.
Self-Guided Attention Redirection (SGAR) and Scheduling (SGAS)
SGAR modulates the self-attention logits in the diffusion model during denoising. Object-related keys are penalized for suppression, while background entries are reweighted to enhance contextually plausible restoration. SGAS orchestrates the temporal application of this modulation: background guidance is active only during early-middle denoising steps, while object suppression persists into the middle stages. This staged control prevents over-guidance, preserving the naturalness of restored textures and minimizing artefacts.
Adaptive Restoration Guidance (ARG)
ARG blends the original and SGAR-modulated noise predictions, parameterized by a user-adjustable coefficient r. This linear combination facilitates fine-grained control over removal strength and background fidelity, enabling customization based on user preference or scene complexity. ARG is instrumental in extending the system's flexibility, allowing vanilla Stable Diffusion models—otherwise insufficient for object removal—to achieve effective, controllable editing without retraining.
Backbone Agnosticism and Extensible Interface
The design is not tightly coupled to a specific diffusion backbone. Three configurations are implemented: SD1.5 (lightweight, fast), SD2.1 (balanced, general), and SDXL1.0 (high-quality, high-resolution). These layers are supported with complete software delivery, including Docker environments and evaluation protocols, underscoring the practical reproducibility and openness of the work.
Experimental Evaluation
Quantitative Assessment
ClickRemoval is evaluated against a suite of strong baselines on a manually curated test set derived from Pico-Banana-400K, ensuring realistic, edited-object-free references. Metrics include global FID, KID, and Local-FID for restoration fidelity in regions previously occupied by the object. The SD1.5 variant achieves FID 9.35, KID 0.899, and Local-FID 17.27 at 512 resolution, outperforming other methods requiring manual masks or prompts. At 1024 resolution, SDXL1.0 attains FID 8.05 and Local-FID 15.56, closely matching AttentiveEraser.
Usability and Preference
User studies and GPT-based multimodal evaluations demonstrate strong preference for ClickRemoval. At 512 resolution, it receives 30.48% of user votes and 28.85% GPT validation; at 1024 resolution, it leads with 41.89% user preference and ranks second in GPT validation, within 2 percentage points of AttentiveEraser. These statistics highlight both the technical efficacy and user-centric design of ClickRemoval, validating its claim of lower interaction overhead and barrier to entry.
Ablation Studies and Progressive Editing
Ablation experiments with ARG illustrate that vanilla diffusion models equipped with the proposed guidance can compete with specialized pipelines, effectively removing objects and reconstructing backgrounds. Progressive editing experiments reveal enhanced completeness with multiple positive clicks, and improved selectivity with negative clicks—enabling nuanced editing such as occlusion handling and selective multi-object removal.
Implications and Future Directions
ClickRemoval embodies a significant advance in democratizing object removal, decoupling interface complexity from technical sophistication. The modular attention redirection framework offers extensibility to other semantic editing tasks (e.g., region replacement, multi-object operations) and can be adapted to alternative diffusion architectures without retraining. By leveraging the inherent semantic structure encoded in pretrained models, the system achieves strong restoration fidelity and user preference without costly finetuning or specialized dataset requirements.
Practically, the tool lowers the threshold for non-expert photo editing, integrating seamlessly into real-time workflows and interactive media applications. Theoretically, the attention-guidance paradigm highlights the latent potential of diffusion models in structured image manipulation, suggesting directions for self-supervised guidance mechanisms and click-driven editing in generative modeling.
Future developments may focus on generalizing the click-to-map interface to broader spatial editing primitives (e.g., area selection, path tracing), enhancing background consistency via context-aware diffusion priors, and integrating multimodal guidance (e.g., voice or sketch input). The open-source nature enables rapid community-driven extensions and benchmarking, propelling the study of human-centric generative editing tools.
Conclusion
ClickRemoval delivers an interactive, open-source, training-free object removal system built on attention redirection in diffusion models, functioning solely via user clicks. The robust semantic mapping, staged guidance scheduling, and adaptive blending produce competitive quantitative and qualitative results across resolutions, setting a new standard for usability and fidelity in object removal. This approach underscores the value of harnessing pretrained generative models through direct semantic manipulation and paves the way for future research in accessible, high-quality image editing tools leveraging diffusion architectures (2605.14461).