Click-to-Locate Editing in Interactive Systems

• Click-to-Locate Editing is an interaction paradigm that separates global semantic guidance from local spatial targeting using clicks, drags, and text instructions.
• It is implemented through diverse methods like diffusion-based image editing, segmentation, and CAD interfaces, employing techniques such as GLMS, FPT, and Alpha-CLIP.
• This paradigm minimizes modality burden by fusing local input cues with global semantic signals to resolve ambiguities and enable targeted transformations.

Searching arXiv for the cited papers to ground the article in current literature. arxiv_search query: (Jiang et al., 2024) Click-to-Locate Editing denotes an interaction and modeling paradigm in which localization is supplied by a click, point, drag, box, or structurally equivalent selection, while the edit itself is driven by text, learned priors, or direct model-side transformations. In image editing, this appears as point-and-text manipulation, click-seeded mask growth, or drag-constrained diffusion; in CAD, it appears as masking token ranges corresponding to selected geometry; in segmentation, it appears as text-conditioned instance selection from one or a few clicks; and in language-model editing, it appears as locating the internal locus of a proposition before changing it. Across these domains, the recurring decomposition is between a signal for where to act and a signal for what to change (Jiang et al., 2024, Helbling et al., 2024, Yuan et al., 6 Feb 2025, Warner et al., 2023).

1. Concept and interaction model

A consistent formulation in the literature is to separate global semantics from local targeting. In CLIPDrag, text prompts are treated as global semantic guidance, while drag points are treated as local spatial guidance; the method therefore uses text for “what” and drag points for “where” and rough “how” (Jiang et al., 2024). Point and Instruct expresses the same decomposition in interface form: a user can mark objects and locations with boxes, stars, and arrows, and then refer to those marks inside a natural-language instruction. Its internal abstraction is effectively

$$\text{Instruction} = f(\text{text tokens},\{\text{points/boxes}\}),$$

with the LLM producing a transformed layout that is then rendered by a layout-conditioned diffusion pipeline (Helbling et al., 2024).

The same disambiguation principle appears in Text + Click segmentation. There, the text prompt defines the semantic target and the click defines the intended instance, especially in scenes with overlapping or co-occurring categories such as “tie”, “suit”, and “person” (Warner et al., 2023). In Deuce, the modality changes from images to code, but the interaction pattern remains structurally similar: clickable widgets overlaid on source text let a user structurally select expressions, patterns, definitions, and target positions, after which a context-sensitive transformation menu becomes available (Hempel et al., 2017).

The surveyed work suggests that “click-to-locate” is best understood not as a single algorithm but as an interaction grammar. A point, click, drag, or structural selection supplies a localizer; text, learned semantics, or a transformation operator supplies the edit semantics. The strength of the paradigm lies in reducing the burden on any one modality: text alone need not encode exact spatial extent, and geometric input alone need not resolve semantic ambiguity.

2. Image editing: point, drag, and dynamic-mask formulations

In diffusion-based image editing, click-to-locate has been instantiated in several technically distinct ways. CLIPDrag is built on Stable Diffusion 1.5 and CLIP-ViT-B/16. Its pipeline consists of identity-preserving LoRA finetuning on a single image, DDIM inversion to a single editing timestep, iterative latent optimization with Global-Local Motion Supervision (GLMS), Fast Point Tracking (FPT), and final denoising (Jiang et al., 2024). During editing, the global CLIP loss

$$\mathcal{L}_{global}(\hat{I},P_e) := 1 - \frac{\langle E_I(\hat{I}(z^k_t)), E_T(P_e) \rangle}{\|E_I(\hat{I}(z^k_t)\| \, \| E_T(P_e) \|}$$

produces a global gradient $G_g$, while drag-based motion supervision produces a local gradient $G_l$. These are fused by an angle-dependent rule rather than simple addition. The implementation details are explicit: 80 LoRA steps, rank 16, learning rate 0.0005, DDIM inversion with 50 denoising steps and inversion strength 0.7, motion supervision radius $r_1 = 4$, point-tracking radius $r_2 = 12$, fusion weight $\lambda = 0.7$, and up to 2000 optimization steps per edit (Jiang et al., 2024).

Click2Mask addresses a different local-editing regime: adding or changing content around a single click without requiring a precise user mask. It uses Stable Diffusion 2.1 base, Blended Latent Diffusion (BLD), and Alpha-CLIP. A potential field $\Phi$ is initialized as a Gaussian centered at the click in latent space, and a binary mask is obtained by thresholding $\Phi$. During diffusion, Alpha-CLIP similarity gradients with respect to the mask are used to raise $\Phi$ in semantically important regions while an increasing threshold $$\mathcal{L}_{global}(\hat{I},P_e) := 1 - \frac{\langle E_I(\hat{I}(z^k_t)), E_T(P_e) \rangle}{\|E_I(\hat{I}(z^k_t)\| \, \| E_T(P_e) \|}$$0 shrinks the mask elsewhere (Regev et al., 2024). The mask therefore evolves under competing pressures: shrinkage for locality and gradient-based support for semantic relevance. Experiments are run with 100 diffusion steps, mask evolution begins around 25% of the schedule, potential elevation begins around 40%, and mask evolution stops around 50% (Regev et al., 2024).

Locate and Forget approaches local editing without extra user-provided spatial input. It first generates a scene description with OpenFlamingo, parses both image caption and target prompt with spaCy, derives forgetting elements from dependency-tree comparisons, and then uses negative classifier-free guidance to suppress those concepts during denoising (Li et al., 2024). Its modified guidance equation is

$$\mathcal{L}_{global}(\hat{I},P_e) := 1 - \frac{\langle E_I(\hat{I}(z^k_t)), E_T(P_e) \rangle}{\|E_I(\hat{I}(z^k_t)\| \, \| E_T(P_e) \|}$$1

This suggests a complementary formulation of click-to-locate editing: a click is not the only possible localizer; automatic concept localization can also supply the “where,” although at concept level rather than as an explicit pixel mask.

3. Localization by segmentation, mask prediction, and distractor selection

A second major line of work treats click-to-locate as a segmentation problem. Text + Click segmentation takes an image, a text phrase, and one or more clicks, and returns a binary mask of the intended instance (Warner et al., 2023). Open-vocabulary capability is obtained through CLIP and MaskCLIP saliency, while clicks are encoded as foreground or background click maps. On RefCOCO, the method reports 72.89 mIoU for “Class Name + 3 clicks” and 68.07 mIoU for “Class Name + 1 click,” matching or exceeding referring-segmentation baselines that require more specific phrases (Warner et al., 2023). The same paper reports better unseen-class generalization than click