Saliency-Guided Warp-Unwarp Framework

• The paper shows that pre-warping images toward salient regions preserves fine-grained structures in diffusion models without increasing latent resolution.
• It employs a differentiable, model-agnostic spatial transform that reassigns sampling density via continuous warping and inverse warping.
• The framework improves unsupervised domain adaptation by emphasizing sparse foreground objects, leading to better object detection and segmentation performance.

The saliency-guided warp–unwarp framework is a spatial reallocation strategy for vision models in which an input image is warped toward salient regions before the main model processes it, and the resulting output or intermediate representation is then mapped back to the original geometry by an inverse warp. In the image-to-image translation setting, the framework is introduced in "WarpI2I: Image Warping for Image-to-Image Translation" as a means to preserve fine-grained structures in compact latent diffusion models without increasing latent resolution; in unsupervised domain adaptation, the related "Instance-Warp" formulation uses saliency-guided in-place warping to emphasize sparse foreground objects during training while leaving inference unchanged (Zheng et al., 30 Jun 2026, Zheng et al., 2024).

1. Problem formulation and scope

The framework addresses a recurring failure mode in image synthesis and recognition pipelines: salient but spatially small structures are poorly represented when the model allocates capacity uniformly across the image plane. In "WarpI2I," the stated motivation is that compact latent diffusion models often struggle to preserve fine-grained structures because the encoder compresses high-resolution inputs into a spatially downsampled latent space. The proposed remedy is to reallocate spatial representation toward salient regions before encoding, process the warped image with the original diffusion model, and then restore the original geometry through an inverse warp (Zheng et al., 30 Jun 2026).

In "Instance-Warp," the same broad principle is applied to unsupervised domain adaptation for driving scenes. The stated problem is that many UDA methods are trained with dominant scene backgrounds such as roads, sky, and sidewalks, which appear dramatically different across domains, and therefore struggle to learn effective features of smaller and often sparse foreground objects such as people, vehicles, and signs. The method improves UDA training by applying in-place image warping to focus on salient objects, thereby oversampling object regions and undersampling background areas (Zheng et al., 2024).

Across both formulations, several invariants remain explicit. The method is presented as model-agnostic; it requires no architectural modification of the underlying image-to-image model in the diffusion setting, and in the UDA setting the backbone and head are otherwise unchanged aside from spatial sampling layers. Both formulations rely on differentiable resampling, and both emphasize low overhead relative to the main network computation (Zheng et al., 30 Jun 2026, Zheng et al., 2024).

2. Saliency construction and spatial importance modeling

The framework begins with a saliency map defined over the image domain. In "WarpI2I," let $y_a$ be a set of detected regions of interest, such as face and eyes in portrait relighting or object bounding boxes in driving scenes. For each region $i \in y_a$, with center $c_i \in \mathbb{R}^2$, width $w_i$, and height $h_i$, the continuous saliency map is

$S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$

where $\mathcal{N}(\mu,\Sigma)$ is the Gaussian density with mean $\mu$ and covariance $\Sigma$, and $b>0$ is a bandwidth hyperparameter controlling the spread of saliency. The map is then normalized so that its spatial integral equals one, or matches the desired total warping mass:

$i \in y_a$0

The details specify that $i \in y_a$1 works well in practice, and that smaller $i \in y_a$2 yields tighter focus around each box (Zheng et al., 30 Jun 2026).

In "Instance-Warp," the saliency function $i \in y_a$3 is likewise defined from bounding boxes, but the formulation is explicitly instance-level and includes clipping and adaptive scale handling. The map is

$i \in y_a$4

Here $i \in y_a$5 is the box center, $i \in y_a$6 its width and height, $i \in y_a$7 trades off overall expansion $i \in y_a$8, and the expansion factor adapts to dataset scale bias. If a particular size bin is underrepresented by fraction $i \in y_a$9, then

$c_i \in \mathbb{R}^2$0

The constants $c_i \in \mathbb{R}^2$1 and $c_i \in \mathbb{R}^2$2 are chosen by grid search, with the details giving $c_i \in \mathbb{R}^2$3 and $c_i \in \mathbb{R}^2$4 as example values (Zheng et al., 2024).

A central technical commonality is that saliency is not a semantic output of the main network but an externally specified spatial importance prior. In the diffusion setting, the prior may come from off-the-shelf detectors such as YOLO-World and InsightFace; in the UDA setting, source-domain bounding-box labels provide the instance-level guidance. This suggests that the framework treats saliency as a modular control signal rather than as a latent variable learned jointly with the downstream task (Zheng et al., 30 Jun 2026, Zheng et al., 2024).

3. Warp and inverse warp operators

Once the saliency map has been constructed, the framework defines a continuous, invertible spatial transform that increases sampling density around regions of high saliency. In "WarpI2I," the input image is $c_i \in \mathbb{R}^2$5 on a regular grid of pixel locations $c_i \in \mathbb{R}^2$6. The method defines the inverse mapping $c_i \in \mathbb{R}^2$7 on the target grid and uses standard backward-warping with bilinear interpolation:

$c_i \in \mathbb{R}^2$8

The warped image $c_i \in \mathbb{R}^2$9 is described as having salient regions "zoomed in" and the remainder "shrunk." Although $w_i$0 has no closed form, the method follows Thavamani et al. ’23 and approximates it by a piecewise tiling of locally invertible bilinear patches $w_i$1. Implementationally, this is realized by precomputing a dense flow field $w_i$2 and applying a GPU-accelerated grid_sampler (Zheng et al., 30 Jun 2026).

After translation or relighting in warped space, the output is unwarped. If $w_i$3 denotes the decoded result in warped coordinates, then

$w_i$4

The details state that although $w_i$5 exactly, the small approximation error is learned out by the diffusion network and empirically causes negligible fidelity loss (Zheng et al., 30 Jun 2026).

"Instance-Warp" gives an alternative transform construction based on the saliency-guided sampling design of Recasens et al. (ECCV 2018). Separate one-dimensional transforms are defined along each axis:

$w_i$6

Here $w_i$7 and $w_i$8 are marginal saliency functions, $w_i$9 is a Gaussian smoothing kernel, and the full inverse mapping is $h_i$0. Sampling grids are then constructed for each output pixel and applied via PyTorch’s grid_sample(input, grid, mode='bilinear', align_corners=True) (Zheng et al., 2024).

The inverse stage differs slightly across the two use cases. In image translation, the decoded image itself is unwarped. In UDA, the labels are not warped; instead, the backbone feature map is unwarped through an approximate piecewise-linear inverse, described as following LZU CVPR ’23, so that the prediction head operates again in the original coordinate frame (Zheng et al., 2024).

4. Integration with latent diffusion and with UDA pipelines

In the latent diffusion setting, the framework is inserted around an otherwise unchanged image-to-image model. Let $h_i$1 denote the LDM encoder, $h_i$2 the corresponding decoder, $h_i$3 the U-Net noise predictor, $h_i$4 the diffusion timestep, and $h_i$5 the text or prompt conditioning. The full pipeline is

$h_i$6

Training uses the standard LDM denoising loss,

$h_i$7

with no additional warping-specific losses. Gradients backpropagate through $h_i$8 and $h_i$9 by differentiable bilinear resampling. The details explicitly state that the only learned components are the original LDM encoder/decoder and diffusion U-Net (Zheng et al., 30 Jun 2026).

The same source gives a high-level pseudocode: compute the saliency map $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$0, normalize it to $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$1, build the inverse warp field $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$2, warp the image using grid_sample, pass the warped image through the encoder, diffusion U-Net, and decoder, and then unwarp with a second grid_sample using a field $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$3 (Zheng et al., 30 Jun 2026).

In the UDA setting, the warp–unwarp mechanism is inserted differently. The source branch of the training loop computes instance saliency from source boxes, constructs a sampling grid $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$4, warps the source image, extracts features with the backbone, approximately unwarps the feature map with $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$5, and sends the result to the task head. The target branch remains unwarped. The total loss is the sum of a supervised source loss and an unsupervised target loss, and the teacher is updated by EMA:

$S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$6

No additional adversarial or entropy terms are introduced for the warp itself; the Gaussian smoothing kernel is stated to regularize the warp to be smooth (Zheng et al., 2024).

The operational contrast between the two settings is substantial. "WarpI2I" uses warp and unwarp both during training and inference because the final generated image must be restored to the original geometry. "Instance-Warp" applies warp and unwarp only while processing source images during training; at inference, the teacher model is used on raw images with no warping or unwarping at test time (Zheng et al., 2024, Zheng et al., 30 Jun 2026).

5. Synthetic paired data generation for relighting

"WarpI2I" supplements the warp–unwarp mechanism with an outpainting-based synthetic data generation pipeline for paired relighting data. The stated purpose is to produce high-quality paired data for human or driving relighting, and the implementation uses FLUX in four stages (Zheng et al., 30 Jun 2026).

The first stage is outpainting: the canvas is expanded around a masked foreground person, and a fixed "neutral studio" background prompt is used through Background Prompt Substitution. The second stage is depth estimation, where Depth-Anything is run to obtain depth maps for the base and relit outpainted images. The third stage is depth-conditioned text-to-image generation: the depths are concatenated side by side in a $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$7 grid, and the prompt is "On left the base prompt, on right the same person under <relight>." The final paired image grid is generated via FLUX.1-Depth. The fourth stage is ChatGPT filtering, which verifies identity, pose, clothing, and lighting match through an automatic ChatGPT API prompt, retaining approximately $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$8 of pairs for training (Zheng et al., 30 Jun 2026).

The quantitative summary in the details states that this pipeline yields approximately $S(x,y)\;=\;\sum_{(c_i,w_i,h_i)\in y_a}\; \mathcal{N}\!\bigl((x,y);\;c_i,\;b\!\begin{pmatrix}w_i & 0\[3pt]0 & h_i\end{pmatrix}\bigr),$9 K high-quality pairs using $\mathcal{N}(\mu,\Sigma)$0 RTX A6000 in approximately one day. The pipeline is presented specifically for paired relighting data rather than as a universal component of the warp–unwarp framework. A plausible implication is that the warping mechanism and the synthetic data pipeline are complementary rather than mutually dependent: the former reallocates spatial capacity, while the latter improves data availability and supervision quality for relighting (Zheng et al., 30 Jun 2026).

6. Empirical behavior, extensions, and recurrent misunderstandings

The reported empirical effects in "WarpI2I" span human relighting, driving scene relighting, and translation. On VITON-HD, StreetTryOn, ROADWork, BDD100K, Cityscapes$\mathcal{N}(\mu,\Sigma)$1DarkZurich, and ACDC, adding warp–unwarp is reported to improve structural preservation, visual fidelity, and identity or lighting faithfulness. The details quantify these changes as DINO-Struct decreasing by $\mathcal{N}(\mu,\Sigma)$2–$\mathcal{N}(\mu,\Sigma)$3, FID decreasing by $\mathcal{N}(\mu,\Sigma)$4–$\mathcal{N}(\mu,\Sigma)$5, user-study scores increasing from approximately $\mathcal{N}(\mu,\Sigma)$6 to approximately $\mathcal{N}(\mu,\Sigma)$7 out of $\mathcal{N}(\mu,\Sigma)$8, and ChatGPT-API pass rates increasing from approximately $\mathcal{N}(\mu,\Sigma)$9 to approximately $\mu$0. Qualitatively, small details such as eyes, traffic arrows, sign text, and license plates remain sharp and correct after relighting or translation (Zheng et al., 30 Jun 2026).

The UDA results in "Instance-Warp" are more task-specific. For object detection with a 2PCNet backbone, the reported gains are $\mu$1 mAP50 for BDD100K Clear $\mu$2 DENSE Foggy, $\mu$3 mAP50 for BDD100K Day $\mu$4 Night, $\mu$5 mAP50 for BDD100K Clear $\mu$6 Rainy, and $\mu$7 mAP50 for BDD100K Clear $\mu$8 ACDC. For semantic segmentation with a DAFormer backbone, the reported gains are $\mu$9 mIoU on Cityscapes $\Sigma$0 ACDC test and $\Sigma$1 mIoU on Cityscapes $\Sigma$2 DarkZurich val. The ablations further state that saliency prior choice follows Static $\Sigma$3 Geometric $\Sigma$4 Instance-level, that $\Sigma$5 and $\Sigma$6 vary performance by only approximately $\Sigma$7 mIoU across a wide range, that undersampling source to balance scales by $\Sigma$8–$\Sigma$9 hurts segmentation by $b>0$0–$b>0$1 mIoU, and that replacing ground-truth boxes with "from-seg" boxes degrades mIoU by approximately $b>0$2 points but still outperforms no warp (Zheng et al., 2024).

The implementation profile is also part of the framework’s empirical characterization. In "WarpI2I," forward warping and inverse warping each cost approximately $b>0$3 ms on a modern GPU, compared with approximately $b>0$4 ms for a one-step diffusion pass. The method is explicitly described as introducing negligible computational overhead, requiring no extra parameters, and using off-the-shelf saliency detectors with a warping grid computed on the fly. In "Instance-Warp," the method is described as adding minimal training memory and no additional inference latency, because test-time inference uses the teacher model on raw images (Zheng et al., 30 Jun 2026, Zheng et al., 2024).

The video extension in "WarpI2I" is defined by frame-by-frame application. Because warping and unwarping operate per frame via fully differentiable grid sampling, the same pipeline can be applied independently to each frame. The details note that the naive approach may flicker under very fast motion, but empirical tests on real video sequences show good temporal consistency and no visible jitter without explicit flow-based smoothing. For extreme cases, one can easily incorporate optical-flow priors into the warp field to further stabilize. This remains an extension rather than a separate reformulation of the framework (Zheng et al., 30 Jun 2026).

Several recurrent misunderstandings are directly addressed by the stated properties of the method. The framework is not presented as increasing latent resolution; in "WarpI2I" it preserves structural details without increasing latent resolution. It is not described as requiring retraining of a new architecture; the method is model-agnostic and requires no architectural modification. It is also not equivalent to generic cropping or zoom augmentation: both papers describe a continuous saliency-driven reallocation of sampling density, together with an inverse mapping that restores alignment in the original coordinate frame (Zheng et al., 30 Jun 2026, Zheng et al., 2024).