Dual Contrastive Denoising Score

• The paper introduces Dual Contrastive Denoising Score, which combines Delta Denoising Score (DDS) and a dual contrastive loss to guide semantic edits while preserving structural details.
• The framework leverages self-attention key features from decoder layers to enforce spatial consistency, achieving high text-image alignment and low perceptual discrepancies.
• The approach is training-free with respect to the diffusion model and uses iterative latent optimization to balance semantic changes and structural preservation.

Dual Contrastive Denoising Score is a score-guided image editing framework introduced for single-reference text-to-image manipulation. In its explicit formulation, it operates on top of a pretrained latent diffusion model and combines Delta Denoising Score (DDS) guidance with a dual contrastive loss applied to self-attention representations, so that the edited image can change according to a target prompt while preserving structural elements of a reference image (Israr et al., 18 Aug 2025). In a broader score-based context, it belongs to the family of denoising-score methods in which a model learns or exploits gradients of log densities under noise perturbations, as in denoising score matching and related diffusion-based objectives (Jolicoeur-Martineau et al., 2020).

1. Foundations in denoising score modeling

Score-based generative modeling is organized around the score function of a distribution, typically written as the gradient of the log density with respect to the input. In denoising score matching (DSM), clean data $x \sim p(x)$ are perturbed by Gaussian noise,

$$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$

and a neural network $s_\theta(\tilde{x},\sigma)$ is trained to approximate the score of the corrupted distribution $q_\sigma(\tilde{x})$. A standard conditional DSM objective is

$$\mathcal{L}_{\mathrm{DSM}}(\theta) = \frac{1}{2}\, \mathbb{E}_{p(\tilde{x},x,\sigma)} \left[ \left\| \sigma s_\theta(\tilde{x},\sigma)+\frac{\tilde{x}-x}{\sigma} \right\|_2^2 \right],$$

with noise levels drawn from a discrete schedule shared by training and sampling (Jolicoeur-Martineau et al., 2020).

This framework underlies diffusion-style denoising and sampling. Once a score estimate is available, one may use Langevin dynamics or related reverse-time procedures to sample or denoise. Work on adversarial score matching further showed that denoising the final Langevin samples using the score network removes an apparent gap with GANs under Frechet Inception Distance, and proposed Consistent Annealed Sampling as a more stable alternative to Annealed Langevin Sampling (Jolicoeur-Martineau et al., 2020).

A closely related identity is the Tweedie–Miyasawa formula, which connects Gaussian denoising to the score of the noisy distribution. In joint classification–denoising models and self-supervised MRI denoising, this identity is used to interpret denoisers as score estimators and to derive posterior-mean denoising from the learned score field (Thiry et al., 2024, Tu et al., 8 May 2025). Dual Contrastive Denoising Score inherits this background but repurposes denoising scores for editing a given real image rather than for unconditional generation alone (Israr et al., 18 Aug 2025).

2. Definition and problem setting of Dual Contrastive Denoising Score

In its explicit and named form, Dual Contrastive Denoising Score addresses the problem of editing a real image with a text-to-image diffusion model when prompt-only control is insufficient. The stated difficulties are prompt mismatch, because a short prompt does not capture all visual details in the input image, and over-editing and unwanted changes, because text-guided generation often alters pose, size, background, and composition in addition to the desired semantic change (Israr et al., 18 Aug 2025).

The framework is defined for single-reference text-to-image manipulation. Its inputs are a reference image $\mathcal{I}_s$, a source prompt $\mathcal{P}_s$, and a target prompt $\mathcal{P}_t$. Its output is a target image $\mathcal{I}_t$ that preserves structural elements of $\mathcal{I}_s$ while changing content or style according to $$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$0 (Israr et al., 18 Aug 2025).

It is built on Stable Diffusion v1.5 and combines two ingredients. First, DDS aligns the denoising behavior of the target latent with that of the source latent under their respective prompts. Second, a dual contrastive loss is imposed on self-attention key features extracted from decoder layers, so that corresponding spatial locations between source and target remain aligned (Israr et al., 18 Aug 2025).

Component	Role	Specification
DDS	Semantic/style alignment	L2 difference of source and target denoising scores
Dual contrastive loss	Structure preservation	Patchwise contrast on self-attention key tokens
Latent optimization	Edit realization	Optimize target latent while keeping diffusion weights fixed

Operationally, the source image is encoded into a latent $$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$1, a target latent $$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$2 is initialized from a noisy version of $$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$3, and at each diffusion timestep both source and target are passed through the U-Net. The method extracts self-attention keys, computes the dual contrastive loss and DDS loss, backpropagates with respect to $$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$4, and finally decodes the optimized latent to the edited image (Israr et al., 18 Aug 2025).

3. Objective functions and the meaning of “dual”

Dual Contrastive Denoising Score is formulated in latent diffusion space. The forward noising process is expressed in DDPM form as

$$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$5

and the method uses the model’s noise predictor $$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$6 as the denoising-score signal (Israr et al., 18 Aug 2025).

The architectural feature source is self-attention. For a self-attention layer with hidden state $$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$7,

$$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$8

and

$$q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),$$9

Dual Contrastive Denoising Score specifically uses self-attention key tokens $s_\theta(\tilde{x},\sigma)$0 from decoder blocks, because these are described as rich in spatial and structural cues such as object contours and layout (Israr et al., 18 Aug 2025).

The dual contrastive objective is defined over patches extracted from source and target key tensors. For each patch, the corresponding spatial location in the other image acts as a positive, while other spatial locations act as negatives. The paper gives a generic form

$s_\theta(\tilde{x},\sigma)$1

together with a positive term

$s_\theta(\tilde{x},\sigma)$2

and a negative term

$s_\theta(\tilde{x},\sigma)$3

These are combined as

$s_\theta(\tilde{x},\sigma)$4

where $s_\theta(\tilde{x},\sigma)$5 is a temperature and $s_\theta(\tilde{x},\sigma)$6 controls the structural constraint strength (Israr et al., 18 Aug 2025).

DDS is defined as

$s_\theta(\tilde{x},\sigma)$7

The total editing objective is

$s_\theta(\tilde{x},\sigma)$8

followed by the latent update

$s_\theta(\tilde{x},\sigma)$9

In the paper’s terminology, the “dual” aspect has two levels: positive versus negative structural contrast inside the contrastive loss, and structure versus semantics across the combined optimization of dual contrastive loss and DDS (Israr et al., 18 Aug 2025).

4. Representation choice, optimization regime, and editing behavior

A central design decision is the use of self-attention keys from decoder layers rather than encoder features, value vectors, queries, or hidden states. The reported ablations state that applying the contrastive loss on keys yields the highest CLIP text–image alignment and the lowest structural distance and LPIPS among the tested internal representations (Israr et al., 18 Aug 2025). The stated interpretation is that keys encode “where to attend” and therefore preserve a spatial skeleton of the source image more effectively than other feature types.

The framework does not train an auxiliary encoder, discriminator, or additional diffusion model. It uses the pretrained latent diffusion model directly, keeps the diffusion parameters fixed, and performs per-image optimization over the target latent only. The implementation note given in the paper is approximately 4 minutes on RTX 3090 for an edit, which distinguishes the method from one-shot inference schemes (Israr et al., 18 Aug 2025).

This optimization regime also explains the method’s behavior. Because the loss is imposed on global spatial patches rather than a manually supplied mask, structural preservation is global rather than region-specific. The method is therefore intended to preserve pose, layout, and background by aligning attention-derived spatial representations throughout the image (Israr et al., 18 Aug 2025). A plausible implication is that its edit locality is induced by the learned diffusion prior and the source–target contrastive alignment rather than by explicit spatial masking.

The editing pipeline is therefore training-free with respect to the diffusion backbone but not optimization-free. This distinction is important: Dual Contrastive Denoising Score reuses a pretrained text-to-image model without further training, yet it still requires iterative latent optimization and repeated source/target U-Net evaluations (Israr et al., 18 Aug 2025).

5. Relation to adjacent score-based research

Although the explicit label Dual Contrastive Denoising Score is introduced in the context of real-image editing, adjacent score-based literature supplies several related notions of denoising, duality, and multiscale consistency. In dual score matching for normalized energy learning, the energy network $q_\sigma(\tilde{x})$0 yields two scores: a space score $q_\sigma(\tilde{x})$1 and a time score $q_\sigma(\tilde{x})$2. The secondary objective on the time derivative is designed to ensure consistent and normalized energies across noise levels, and the paper describes this as enabling a normalized energy landscape $q_\sigma(\tilde{x})$3 (Guth et al., 5 Jun 2025). This is a different use of “dual” from Dual Contrastive Denoising Score, where duality refers to contrastive push–pull and the joint control of structure and semantics.

A related multiscale interpretation appears in score matching for graduated non-convex image priors, where the smoothed negative log density

$q_\sigma(\tilde{x})$4

is shown to become convex for sufficiently large noise variance. That work interprets denoising score-based models as following a graduated non-convexity heuristic across multiple noise levels (Kobler et al., 2023). This suggests a broader conceptual link between noise-scale coupling and “dual” score formulations, even when no explicit contrastive editing loss is present.

Other applications generalize denoising scores beyond unconditional image generation. Score-based self-supervised MRI denoising introduces a generalized denoising score matching (GDSM) loss that learns directly from noisy observations and can be extended to multi-contrast denoising (Tu et al., 8 May 2025). Classification-Denoising Networks model a joint density $q_\sigma(\tilde{x})$5 over noisy images and class labels, then recover both classification and denoising from a shared backbone via score computation and the Tweedie–Miyasawa identity (Thiry et al., 2024). Unsupervised Image Denoising with Score Function instead learns the noisy-image score and solves an analytic system

$q_\sigma(\tilde{x})$6

for denoising under additive, multiplicative, correlated, and mixture noise models (Xie et al., 2023). SCDM adapts score-based denoising to digital semantic communications by matching the forward corruption to AWGN digital channel properties rather than using a mismatched diffusion noise process (Mo et al., 18 Jan 2025).

Taken together, these works indicate that “denoising score” now spans at least three technically distinct uses: multiscale generative modeling and sampling, analytic or posterior-mean denoising, and guided editing in latent diffusion systems. Dual Contrastive Denoising Score occupies the third category, but it remains tightly connected to the first two through its use of denoising scores, Gaussian corruption identities, and multiscale latent trajectories (Israr et al., 18 Aug 2025, Jolicoeur-Martineau et al., 2020).

6. Empirical behavior, applications, and limitations

The principal applications reported for Dual Contrastive Denoising Score are real image editing and zero-shot image-to-image translation. The paper lists attribute edits such as cat $q_\sigma(\tilde{x})$7 pink pig, cat $q_\sigma(\tilde{x})$8 dog, and cat $q_\sigma(\tilde{x})$9 cow, as well as style shifts such as sketch $$\mathcal{L}_{\mathrm{DSM}}(\theta) = \frac{1}{2}\, \mathbb{E}_{p(\tilde{x},x,\sigma)} \left[ \left\| \sigma s_\theta(\tilde{x},\sigma)+\frac{\tilde{x}-x}{\sigma} \right\|_2^2 \right],$$0 oil painting and drawing $$\mathcal{L}_{\mathrm{DSM}}(\theta) = \frac{1}{2}\, \mathbb{E}_{p(\tilde{x},x,\sigma)} \left[ \left\| \sigma s_\theta(\tilde{x},\sigma)+\frac{\tilde{x}-x}{\sigma} \right\|_2^2 \right],$$1 photo. One concrete example uses a source prompt “a drawing of a cat sitting next to a mirror” and a target prompt “a pink pig sitting next to a mirror,” with the reported outcome that the edited image preserves pose and background while changing species-specific features and color (Israr et al., 18 Aug 2025).

The method is positioned against mask-based inpainting, prompt-only attention control methods such as Prompt-to-Prompt and MasaCtrl, Score Distillation Sampling, Delta Denoising Score, and Contrastive Denoising Score. The specific critique is that mask-based methods require manual masks, prompt-only control is sensitive to inversion quality and timestep choices, DDS overlooks structural details, and CDS may preserve structure while failing to fully respect complex target prompts (Israr et al., 18 Aug 2025).

Quantitatively, the reported evaluation uses CLIPScore or CLIP Acc for text–image alignment, Structure Dist based on DINO for structural similarity, and LPIPS for perceptual or background similarity. The excerpted results state that DualCDS reaches CLIP Acc up to 99.7%, Dist down to 0.021 / 0.025, and LPIPS down to 0.062 / 0.102 on cat-to-pig/dog/cow tasks, while a user study with 30 participants assigns the highest scores to DualCDS for text-to-image agreement (~9.34), structure consistency (~9.42), and realness (~9.12) (Israr et al., 18 Aug 2025).

The paper also records several limitations. The contrastive weight $$\mathcal{L}_{\mathrm{DSM}}(\theta) = \frac{1}{2}\, \mathbb{E}_{p(\tilde{x},x,\sigma)} \left[ \left\| \sigma s_\theta(\tilde{x},\sigma)+\frac{\tilde{x}-x}{\sigma} \right\|_2^2 \right],$$2 must be balanced: values that are too high over-constrain structure and suppress semantics, whereas values that are too low permit pose or background drift. Patch size and patch count matter: large patches favor global features and lose fine details, while too few patches lead to incomplete manipulation. The method can struggle when the target prompt implies massive layout changes, depends on the coverage and inversion behavior of the base Stable Diffusion model, and incurs the optimization cost associated with per-edit latent refinement (Israr et al., 18 Aug 2025).

Several misconceptions can therefore be resolved directly. Dual Contrastive Denoising Score is not a mask-based editing method; it does not require explicit masks. It is not a retraining method for the diffusion backbone; the Stable Diffusion weights remain frozen. It is also not identical to dual score matching for normalized density estimation, where “dual” refers to joint optimization of space and time scores rather than patchwise positive–negative contrast (Israr et al., 18 Aug 2025, Guth et al., 5 Jun 2025). Its defining feature is the specific combination of DDS-guided semantic change and self-attention-key contrastive structural anchoring within a latent optimization loop.