---
title: Click-to-Locate Editing in Interactive Systems
url: https://www.emergentmind.com/topics/click-to-locate-editing
type: topic
---

# Click-to-Locate Editing in Interactive Systems

Searching arXiv for the cited papers to ground the article in current literature.
arxiv_search query: 2410.03097
Click-to-Locate Editing denotes an interaction and modeling paradigm in which localization is supplied by a click, point, drag, box, or structurally equivalent selection, while the edit itself is driven by text, learned priors, or direct model-side transformations. In image editing, this appears as point-and-text manipulation, click-seeded mask growth, or drag-constrained diffusion; in CAD, it appears as masking token ranges corresponding to selected geometry; in segmentation, it appears as text-conditioned instance selection from one or a few clicks; and in language-model editing, it appears as locating the internal locus of a proposition before changing it. Across these domains, the recurring decomposition is between a signal for *where* to act and a signal for *what* to change [2410.03097] [2402.07925] [2502.03997] [2311.14822].

## 1. Concept and interaction model

A consistent formulation in the literature is to separate global semantics from local targeting. In CLIPDrag, text prompts are treated as **global semantic guidance**, while drag points are treated as **local spatial guidance**; the method therefore uses text for “what” and drag points for “where” and rough “how” [2410.03097]. Point and Instruct expresses the same decomposition in interface form: a user can mark objects and locations with boxes, stars, and arrows, and then refer to those marks inside a natural-language instruction. Its internal abstraction is effectively
\[
\text{Instruction} = f(\text{text tokens},\{\text{points/boxes}\}),
\]
with the LLM producing a transformed layout that is then rendered by a layout-conditioned diffusion pipeline [2402.07925].

The same disambiguation principle appears in Text + Click segmentation. There, the text prompt defines the semantic target and the click defines the intended instance, especially in scenes with overlapping or co-occurring categories such as “tie”, “suit”, and “person” [2311.14822]. In Deuce, the modality changes from images to code, but the interaction pattern remains structurally similar: clickable widgets overlaid on source text let a user structurally select expressions, patterns, definitions, and target positions, after which a context-sensitive transformation menu becomes available [1707.00015].

The surveyed work suggests that “click-to-locate” is best understood not as a single algorithm but as an interaction grammar. A point, click, drag, or structural selection supplies a localizer; text, learned semantics, or a transformation operator supplies the edit semantics. The strength of the paradigm lies in reducing the burden on any one modality: text alone need not encode exact spatial extent, and geometric input alone need not resolve semantic ambiguity.

## 2. Image editing: point, drag, and dynamic-mask formulations

In diffusion-based image editing, click-to-locate has been instantiated in several technically distinct ways. CLIPDrag is built on **Stable Diffusion 1.5** and **CLIP-ViT-B/16**. Its pipeline consists of identity-preserving LoRA finetuning on a single image, DDIM inversion to a single editing timestep, iterative latent optimization with **Global-Local Motion Supervision (GLMS)**, **Fast Point Tracking (FPT)**, and final denoising [2410.03097]. During editing, the global CLIP loss
\[
\mathcal{L}_{global}(\hat{I},P_e) := 1 - \frac{\langle E_I(\hat{I}(z^k_t)), E_T(P_e) \rangle}{\|E_I(\hat{I}(z^k_t)\| \, \| E_T(P_e) \|}
\]
produces a global gradient \(G_g\), while drag-based motion supervision produces a local gradient \(G_l\). These are fused by an angle-dependent rule rather than simple addition. The implementation details are explicit: 80 LoRA steps, rank 16, learning rate 0.0005, DDIM inversion with 50 denoising steps and inversion strength 0.7, motion supervision radius \(r_1 = 4\), point-tracking radius \(r_2 = 12\), fusion weight \(\lambda = 0.7\), and up to 2000 optimization steps per edit [2410.03097].

Click2Mask addresses a different local-editing regime: adding or changing content around a single click without requiring a precise user mask. It uses **Stable Diffusion 2.1 base**, **Blended Latent Diffusion (BLD)**, and **Alpha-CLIP**. A potential field \(\Phi\) is initialized as a Gaussian centered at the click in latent space, and a binary mask is obtained by thresholding \(\Phi\). During diffusion, Alpha-CLIP similarity gradients with respect to the mask are used to raise \(\Phi\) in semantically important regions while an increasing threshold \(\tau_t\) shrinks the mask elsewhere [2409.08272]. The mask therefore evolves under competing pressures: shrinkage for locality and gradient-based support for semantic relevance. Experiments are run with 100 diffusion steps, mask evolution begins around 25% of the schedule, potential elevation begins around 40%, and mask evolution stops around 50% [2409.08272].

Locate and Forget approaches local editing without extra user-provided spatial input. It first generates a scene description with **OpenFlamingo**, parses both image caption and target prompt with **spaCy**, derives **forgetting elements** from dependency-tree comparisons, and then uses negative classifier-free guidance to suppress those concepts during denoising [2405.19708]. Its modified guidance equation is
\[
\overline{\boldsymbol{\epsilon}_\theta^{(t)}(\mathbf{z}_t)} = \boldsymbol{\epsilon}_\theta^{(t)}(\mathbf{z}_t) + w\big(\boldsymbol{\epsilon}_\theta^{(t)}(\mathbf{z}_t, c_p) - \boldsymbol{\epsilon}_\theta^{(t)}(\mathbf{z}_t)\big) - \eta\big(\boldsymbol{\epsilon}_\theta^{(t)}(\mathbf{z}_t, c_n) - \boldsymbol{\epsilon}_\theta^{(t)}(\mathbf{z}_t)\big).
\]
This suggests a complementary formulation of click-to-locate editing: a click is not the only possible localizer; automatic concept localization can also supply the “where,” although at concept level rather than as an explicit pixel mask.

## 3. Localization by segmentation, mask prediction, and distractor selection

A second major line of work treats click-to-locate as a segmentation problem. Text + Click segmentation takes an image, a text phrase, and one or more clicks, and returns a binary mask of the intended instance [2311.14822]. Open-vocabulary capability is obtained through CLIP and MaskCLIP saliency, while clicks are encoded as foreground or background click maps. On RefCOCO, the method reports **72.89 mIoU** for “Class Name + 3 clicks” and **68.07 mIoU** for “Class Name + 1 click,” matching or exceeding referring-segmentation baselines that require more specific phrases [2311.14822]. The same paper reports better unseen-class generalization than click

Source: https://www.emergentmind.com/topics/click-to-locate-editing