Inpaint4Drag: Drag-Based Image Editing

• Inpaint4Drag is a drag-based image editing framework that decomposes user-induced deformations into pixel-space warping and inpainting for precise control.
• It employs bidirectional warping and refined mask processing to generate real-time previews and smooth transitions between deformed and inpainted regions.
• The method serves as a universal adapter for standard inpainting models, outperforming latent-space approaches in speed, memory efficiency, and deformation accuracy.

Searching arXiv for the cited work and closely related context papers to ground the article. Inpaint4Drag is a drag-based image editing framework that reformulates drag manipulation as a composition of pixel-space deformation and image inpainting rather than latent-space optimization in a generative model. It is presented as a method that decomposes editing into bidirectional warping of a user-specified deformable region and inpainting of the newly revealed or inconsistent regions, thereby converting drag inputs into the standard $$(\text{image}, \text{mask}) \rightarrow \text{edited image}$$ interface used by off-the-shelf inpainting systems (Lu et al., 4 Sep 2025). This design situates Inpaint4Drag at the intersection of interactive geometric editing, mask-guided image synthesis, and modular vision pipelines, with the stated aim of achieving precise control, real-time previews, and compatibility with any inpainting model without architecture modification (Lu et al., 4 Sep 2025).

1. Conceptual formulation and problem setting

Inpaint4Drag addresses drag-based image editing, in which a user specifies handle-to-target point pairs and expects the image content at the handle locations to move to the target locations while remaining visually coherent (Lu et al., 4 Sep 2025). The framework is motivated by limitations attributed to prior drag methods such as DragGAN, DragDiffusion, DiffEditor, EasyDrag, and FastDrag, which are described as operating predominantly in the latent space of GANs or diffusion models and as relying on iterative latent optimization with motion supervision and feature tracking (Lu et al., 4 Sep 2025).

The central reformulation is to separate geometry from content generation. The geometric component is a pixel-space bidirectional warping process applied to user-defined deformable region masks. The content-generation component is image inpainting restricted to regions that become empty after warping and to a narrow boundary band introduced for seam smoothing (Lu et al., 4 Sep 2025). In this sense, drag editing is reduced to a standard inpainting problem. The framework therefore does not require the inpainting model to interpret drag semantics directly; those semantics are resolved in the warping stage, and the inpainting model receives only a warped image and a binary mask (Lu et al., 4 Sep 2025).

A key conceptual analogy in the method is the treatment of masked image regions as deformable materials in the physical world. The user-selected region is modeled as an elastic material whose motion propagates smoothly from the drag handles, with nearby areas moving more strongly than distant ones (Lu et al., 4 Sep 2025). This formulation is intended to preserve natural shape under manipulation and to provide explicit control over which parts of the scene are allowed to deform. A plausible implication is that the mask serves not merely as an edit-selection tool but as an explicit articulation prior, disambiguating the intended deformation from the same drag vector applied to different semantic regions.

The method is also framed as a response to three limitations of latent drag editing identified in the source paper: limited precision from low-resolution latent representations, slow feedback caused by repeated optimization or denoising, and limited occlusion handling because the same generative model must enforce motion constraints while hallucinating newly visible content (Lu et al., 4 Sep 2025). Inpaint4Drag instead performs inexpensive warping for previews and invokes inpainting only once per edit.

2. Input specification, mask refinement, and editing pipeline

The input to the system is an image $I \in \mathbb{R}^{H \times W \times 3}$, a deformable region mask $M$, and a set of drag point pairs $\{(h_i, t_i)\}_{i=1}^K$ consisting of handle points $h_i$ and target points $t_i$ (Lu et al., 4 Sep 2025). Multiple pairs may be specified to induce complex non-rigid deformations. The deformable region mask is semantically important: pixels inside $M$ are treated as a single elastic material, and different connected masked regions can deform independently when associated with different control points (Lu et al., 4 Sep 2025).

The paper includes an optional mask-refinement stage based on EfficientViT-SAM-L0. Sparse points are sampled from the user mask,

$$P_s \subset \{(x,y)\mid M(x,y)=1\}, \quad |P_s| \le 128,$$

and passed to a SAM predictor to obtain a refined mask (Lu et al., 4 Sep 2025). The method then applies morphological dilation and erosion with a disk structuring element $K_1$ of radius $r_1$ (default $I \in \mathbb{R}^{H \times W \times 3}$0 px),

$I \in \mathbb{R}^{H \times W \times 3}$1

$I \in \mathbb{R}^{H \times W \times 3}$2

followed by the boundary-guided refinement rule

$I \in \mathbb{R}^{H \times W \times 3}$3

This procedure is described as preserving SAM’s improved boundaries while constraining them to remain near the user’s rough mask, thereby preventing the mask from expanding to unrelated objects or distant regions (Lu et al., 4 Sep 2025).

The operational pipeline proceeds in four stages. First, the user paints a deformable mask and places drag pairs. Second, mask refinement and bidirectional warping are run to produce an immediate warped preview. Third, the system constructs an inpainting mask marking newly revealed pixels and a boundary band around the warped region. Fourth, an inpainting backend is called once to produce the final edited image (Lu et al., 4 Sep 2025). The source paper emphasizes that the preview stage is computationally cheap enough to support interactive iteration: users can alter the mask or control points and inspect the warped preview before committing to inpainting.

3. Bidirectional warping and the “elastic material” model

The warping module operates on the refined mask $I \in \mathbb{R}^{H \times W \times 3}$4 and the control pairs $I \in \mathbb{R}^{H \times W \times 3}$5 to produce a warped image $I \in \mathbb{R}^{H \times W \times 3}$6, a warped mask $I \in \mathbb{R}^{H \times W \times 3}$7, and an inpainting mask $I \in \mathbb{R}^{H \times W \times 3}$8 (Lu et al., 4 Sep 2025). The procedure begins by extracting connected contours,

$I \in \mathbb{R}^{H \times W \times 3}$9

so that each contour $M$0 defines a connected deformable region (Lu et al., 4 Sep 2025). Control pairs are then associated with each contour according to whether the handle point lies inside the region bounded by that contour.

The forward warping stage applies inverse-distance weighting to every point $M$1 on or inside a contour. For a contour-specific control set of size $M$2, the weights are

$M$3

with $M$4 for stability, and the point is moved according to

$M$5

This produces a transformed contour $M$6 and a sparse set of forward correspondences $M$7 (Lu et al., 4 Sep 2025). The paper interprets this weighting as encoding a smooth, locality-sensitive deformation prior: regions near a handle move more, and distant regions move less.

Forward warping alone is insufficient because discrete source-to-target mapping can leave gaps when the region is stretched. Inpaint4Drag therefore adds a backward mapping stage. For each pixel $M$8 inside the target region bounded by $M$9, the algorithm finds its $\{(h_i, t_i)\}_{i=1}^K$0 nearest neighbors among target pixels that already have forward mappings, with default $\{(h_i, t_i)\}_{i=1}^K$1, and computes new inverse-distance weights in target space (Lu et al., 4 Sep 2025). The source coordinate is interpolated by

$\{(h_i, t_i)\}_{i=1}^K$2

subject to a validity test requiring both $\{(h_i, t_i)\}_{i=1}^K$3 and $\{(h_i, t_i)\}_{i=1}^K$4 to lie within image bounds (Lu et al., 4 Sep 2025). Validated mappings are used to define the warped image by

$\{(h_i, t_i)\}_{i=1}^K$5

and the warped mask is defined as the set of assigned target pixels (Lu et al., 4 Sep 2025).

The source paper characterizes this formulation as “bidirectional” because forward mapping establishes the deformed boundary and sparse correspondence structure, while backward mapping fills target pixels densely and avoids the holes inherent in forward-only rasterization (Lu et al., 4 Sep 2025). It further notes that the method is reminiscent of moving least squares and classical elastic deformation, but does not solve a global PDE or explicit ARAP-style energy. Instead, it relies on local distance-weighted interpolation in both directions. This suggests that the method prioritizes computational simplicity and responsiveness over explicit variational regularization.

4. Inpainting interface and relation to inpainting research

Once warping is complete, the method constructs an inpainting mask from two components. The first is the set of pixels in the original deformable mask that are not filled in the warped result,

$\{(h_i, t_i)\}_{i=1}^K$6

The second is a narrow band around the boundary of the warped region, $\{(h_i, t_i)\}_{i=1}^K$7, used to smooth seams (Lu et al., 4 Sep 2025). These are dilated with a disk kernel $\{(h_i, t_i)\}_{i=1}^K$8 of radius $\{(h_i, t_i)\}_{i=1}^K$9 (default $h_i$0 px):

$h_i$1

The resulting pair $h_i$2 is exactly the input format expected by generic inpainting systems (Lu et al., 4 Sep 2025).

The principal inpainting backend used in the main experiments is Stable Diffusion v1.5 Inpainting. The procedure resizes the image to $h_i$3 pixels on the longer edge while preserving aspect ratio, encodes the warped image with a VAE using TAESD, concatenates the mask in latent space, initializes the masked latent region with pure noise, runs accelerated sampling with LCM-LoRA for $h_i$4 steps using an empty text prompt and cached embeddings, and decodes the result to obtain the final edited image $h_i$5 (Lu et al., 4 Sep 2025). The process is summarized as

$h_i$6

A central claim of the framework is that this interface makes Inpaint4Drag a universal adapter for inpainting models, because any model with the standard interface $h_i$7 can be substituted without architectural modification or retraining (Lu et al., 4 Sep 2025). The paper explicitly lists DeepFillv2, LaMa, SD-1.5-Inpaint, and SD-XL-Inpaint as compatible backends (Lu et al., 4 Sep 2025).

This design can be situated against earlier mask-aware inpainting literature. The background material on partial-convolution-based inpainting describes a U-Net-like encoder-decoder in which each convolution is replaced by a partial convolution conditioned on a binary mask, so that only valid pixels contribute to each receptive field and the mask is progressively updated through depth (Patel et al., 2021). That work emphasizes robustness to irregular, segmentation-based, and user-drawn masks, with applications including manual mask generation and automatic segmented object removal (Patel et al., 2021). Inpaint4Drag does not require partial convolutions specifically, but its output format is directly aligned with such mask-aware inpainting paradigms. This suggests that its modularity is not restricted to diffusion backends; it is compatible in principle with CNN, GAN, transformer, and diffusion inpainting systems so long as they accept an image and a binary mask (Lu et al., 4 Sep 2025).

5. Performance, evaluation, and interaction properties

The source paper reports efficiency figures at $h_i$8 resolution on a V100 GPU. SAM-based mask refinement takes approximately $h_i$9 s, bidirectional warping approximately $t_i$0 s, and SD-1.5-Inpaint with TAESD and LCM-LoRA approximately $t_i$1 s, yielding a total edit time of about $t_i$2 s and GPU memory usage of $t_i$3 GB (Lu et al., 4 Sep 2025). Because refinement and warping together take about $t_i$4 s, the preview stage is described as capable of real-time operation at approximately $t_i$5 fps, while inpainting is invoked only when the user commits the edit (Lu et al., 4 Sep 2025).

The paper compares Inpaint4Drag against DragDiffusion, DiffEditor, SDE-Drag, and FastDrag on DragBench-S and DragBench-D. The evaluation uses Mean Distance (MD), defined through DIFT-based correspondence tracking between moved handle points and their requested targets, and LPIPS, measuring perceptual similarity between the original and edited images (Lu et al., 4 Sep 2025). Reported values on DragBench-S / DragBench-D are as follows.

Method	MD / LPIPS (×100)	Time / Memory
DragDiffusion	7.0 / 18.0 ; 6.7 / 10.2	177.7 s / 11.6 GB
DiffEditor	23.6 / 17.6 ; 22.1 / 10.9	43.1 s / 6.6 GB
SDE-Drag	7.5 / 11.4 ; 8.1 / 14.9	126.1 s / 6.9 GB
FastDrag	4.1 / 24.1 ; 5.1 / 13.5	4.2 s / 5.0 GB
Inpaint4Drag	3.6 / 11.4 ; 3.9 / 9.1	0.3 s / 2.7 GB

Within these experiments, Inpaint4Drag is reported to achieve the best MD on both benchmarks, competitive LPIPS, the fastest runtime, and the lowest memory consumption (Lu et al., 4 Sep 2025). The paper also states that it is approximately $t_i$6 faster than FastDrag and approximately $t_i$7 faster than DragDiffusion (Lu et al., 4 Sep 2025).

Qualitative examples in the paper include facial edits such as moving eyes, changing gaze, adjusting mouth openness, and rotating heads; animal manipulations such as opening a lion’s mouth or turning a polar bear; articulated pose changes including raising arms and bending elbows; and multi-round scene rearrangement such as moving chess pieces sequentially (Lu et al., 4 Sep 2025). The preview panel overlays a grid on the inpainting regions so that users can see which parts of the scene will require synthesis (Lu et al., 4 Sep 2025). The authors attribute the resulting sense of control to full-resolution pixel-space manipulation, preservation of original colors and textures inside the warped region, and the fact that unedited regions outside the deformable mask remain untouched except for a small boundary smoothing band (Lu et al., 4 Sep 2025).

6. Ablations, limitations, and broader significance

The source paper reports several ablations. A comparison between unidirectional and bidirectional warping shows that forward-only deformation produces visible gaps and discontinuities in stretched areas, whereas the addition of backward mapping yields smooth and complete coverage (Lu et al., 4 Sep 2025). Another ablation examines mask refinement: an initial user mask may be rough, and raw SAM may over-segment or capture unrelated objects, while the boundary-guided refinement with dilation and erosion constraints is reported to produce masks that better align with true object boundaries while remaining within the intended region (Lu et al., 4 Sep 2025). A third analysis compares inpainting backends and states that DeepFillv2, LaMa, SD-1.5-Inpaint, and SD-XL-Inpaint achieve similar MD and LPIPS, with qualitative differences mainly in realism, detail richness, speed, and memory (Lu et al., 4 Sep 2025).

The explicitly stated limitations concern dependence on user input quality, sensitivity to extreme deformations, and inherited failure modes from the chosen inpainting backend (Lu et al., 4 Sep 2025). Inaccurate masks can include background or exclude relevant object parts, which may lead to unwanted background deformation or disconnected pieces stretching incorrectly. Poorly placed control points may induce unnatural material behavior. Very large motions can overstretch the region before inpainting, and large newly revealed areas may require highly speculative synthesis. If the inpainting model is weak for a given category or layout, the filled regions may be inconsistent in style or lighting or may fail on complex structures such as fine hand anatomy or rare objects (Lu et al., 4 Sep 2025).

The broader significance of Inpaint4Drag lies in its task decomposition. Compared with latent drag methods, it relocates deformation from latent space to pixel space and restricts generative modeling to hole filling (Lu et al., 4 Sep 2025). Compared with classical deformation methods, it uses practical inverse-distance interpolation rather than a global PDE or energy-minimization formulation (Lu et al., 4 Sep 2025). Compared with inpainting-only systems, it provides a mechanism for explicit geometric manipulation before synthesis. This combination makes the framework modular and future-facing in a narrow technical sense: as inpainting systems improve, the quality of the second stage can improve without changing the warping algorithm or retraining a new drag model (Lu et al., 4 Sep 2025).

The paper identifies extensions to video drag editing, 3D or NeRF scenes, and more understanding-enabled models that could help users avoid poor masks or control points (Lu et al., 4 Sep 2025). In relation to mask-aware inpainting research, including partial-convolution-based approaches designed for irregular holes and user-drawn masks (Patel et al., 2021), this suggests a general pattern: interactive image editing can be treated as a front-end geometry-and-mask generation problem coupled to a back-end completion model. Inpaint4Drag instantiates that pattern specifically for drag-based editing, with bidirectional warping providing the geometric component and generic inpainting providing the synthesis component (Lu et al., 4 Sep 2025).