---
title: 'DUDE: Diffusion-Based UCIR'
url: https://www.emergentmind.com/topics/dude
type: topic
---

# DUDE: Diffusion-Based UCIR

DUDE, short for **Diffusion-Based Unsupervised cross-Domain imagE retrieval**, is a method for **unsupervised cross-domain image retrieval (UCIR)** that addresses retrieval across visual domains such as photo, art painting, cartoon, sketch, clipart, and infograph without labels or paired training data. Its central claim is that UCIR should not align entangled whole-image features directly; instead, it should first separate **domain-invariant object content** from **domain-specific style**, and only then align the object features across domains. DUDE operationalizes this idea with a frozen text-to-image diffusion model, specifically Stable Diffusion, and a progressive mutual-neighbor alignment procedure, and reports state-of-the-art performance on PACS, Office-Home, and DomainNet [2509.04193].

## 1. Task formulation and motivation

UCIR considers two domains,
\[
D^A = \{I_i^A\}_{i=1}^n,\quad D^B = \{I_i^B\}_{i=1}^m,
\]
that differ in visual style but share semantic categories. Given a query image \(I_i^A\) from category \(C\), the objective is to retrieve images of the same category \(C\) from domain \(B\), with **no labels or paired data available during training** [2509.04193].

The difficulty is not merely domain shift in the conventional domain-adaptation sense. In UCIR, the category-relevant object signal is frequently entangled with domain cues such as sketch strokes, infograph text, color palette, or rendering style. Previous UCIR methods, including DD, ProtoOT, CoDA, PCS, and CDS, primarily learn **global image representations** and align them through instance-level, cluster-level, or prototype-based objectives. DUDE argues that such alignment is often performed on features of the **whole image**, so the model can over-focus on style similarity rather than object identity [2509.04193].

This reframes UCIR as a **disentangle-then-align** problem. The method assumes that the domain word in a text prompt can absorb style information, while a learned token can be forced to encode object semantics. A plausible implication is that cross-domain retrieval improves when alignment is carried out only after this factorization.

## 2. Diffusion-based object disentanglement

DUDE uses a **pre-trained, frozen Stable Diffusion** backbone rather than training a generative model from scratch. The trainable component is a feature extractor \(f_\theta\), instantiated as a **ResNet-50 pre-trained on ImageNet**, which maps an image \(I_i^d\) to an embedding \(z_i^d = f_\theta(I_i^d)\). This embedding becomes a learnable object token \([z_i^d]\) inside a prompt template [2509.04193].

For each domain, DUDE defines a hand-crafted prompt with an explicit style word. On PACS, examples include:

- “a painting of a \([z]\)”
- “a cartoon of a \([z]\)”
- “a photo of a \([z]\)”
- “a sketch of a \([z]\)”

The domain token is fixed per domain and is intended to carry **style**. The learnable token \([z]\), produced by \(f_\theta\), is pressured to carry **object content**. The image is encoded by the frozen VAE \(\gamma_\phi\), noise is added at diffusion timestep \(t\),
\[
x_{i,t}^d = \sqrt{\bar{\alpha}_t}\,\gamma_{\phi}(I_i^d) + \sqrt{1-\bar{\alpha}_t}\,\epsilon,
\]
and the frozen UNet noise predictor \(\epsilon_\phi\) is conditioned on the prompt embedding \(c_\theta\). DUDE then optimizes only \(f_\theta\) through the reconstruction-style objective
\[
\mathcal{L}_{\text{OD}} = \mathbb{E}_{\epsilon,t}\left[\left\| \epsilon - \epsilon_{\phi}(x_{i,t}^d, c_{\theta}, t) \right\|_2^2\right].
\]

Because Stable Diffusion’s parameters remain frozen, gradients flow only into the feature extractor. The intended effect is functional factorization: the explicit domain word represents “painting / sketch / infograph / photo,” while \([z_i^d]\) must encode the semantic object information needed for denoising. The paper supports this qualitatively by swapping domain words while reusing an extracted object token; the object remains while the style changes [2509.04193].

A common misunderstanding is that DUDE is a diffusion model for retrieval at inference time. It is not. The diffusion model is a **training-time disentangler**; the deployed retrieval system uses only the learned discriminative encoder.

## 3. Progressive mutual-neighbor alignment

After disentanglement, DUDE aligns the object-centric features through a **Progressive Alignment Module**. The method uses a momentum encoder \(f_m\) updated in MoCo style and domain-specific memory banks \(\mathcal{Q}^A\) and \(\mathcal{Q}^B\) to provide stable keys, negatives, and neighbor pools [2509.04193].

The progression has three stages.

First, DUDE performs **instance augmentation alignment**. For an image \(I_i^d\) and its augmentation \(\hat I_i^d\), it computes
\[
z_i^d = f_\theta(I_i^d),\quad \hat z_i^d = f_m(\hat I_i^d),
\]
and applies an InfoNCE-style loss between \(z_i^d\) and \(\hat z_i^d\). This initializes augmentation invariance and stabilizes the representation.

Second, DUDE performs **in-domain mutual-neighbor alignment**. For each sample, it computes top-\(k\) neighbors by cosine similarity within the same domain and defines a symmetric adjacency matrix \(\mathbf{M}_{\mathrm{in}^d}\) where a pair is positive only if the two samples are **mutual** top-\(k\) neighbors. This mutuality criterion filters noisy asymmetric matches. In-domain positives are then injected into an InfoNCE objective:
\[
\mathcal{L}_{\mathrm{in}} = \mathcal{L}_{\mathrm{in}^A} + \mathcal{L}_{\mathrm{in}^B}.
\]

Third, once in-domain structure is more reliable, DUDE adds **cross-domain mutual-neighbor alignment**. It computes top-\(k\) neighbors from \(A\) to \(B\) and from \(B\) to \(A\), forms cross-domain adjacency matrices \(\mathbf{M}_{\mathrm{cross}^{A\rightarrow B}}\) and \(\mathbf{M}_{\mathrm{cross}^{B\rightarrow A}}\), and uses only mutual cross-domain neighbors as positives in cross-domain InfoNCE:
\[
\mathcal{L}_{\mathrm{cross}} = \mathcal{L}_{\mathrm{cross}^{A\rightarrow B}} + \mathcal{L}_{\mathrm{cross}^{B\rightarrow A}}.
\]

The progression is explicit. After a disentanglement warm-up, DUDE first optimizes
\[
\mathcal{L}_{\mathrm{PA1}} = \mathcal{L}_{\mathrm{aug}} + \beta\,\mathcal{L}_{\mathrm{in}},
\]
and then moves to
\[
\mathcal{L}_{\mathrm{PA2}} = \mathcal{L}_{\mathrm{in}} + \lambda\,\mathcal{L}_{\mathrm{cross}}.
\]
The rationale is that early cross-domain alignment on immature features is unstable; mutual neighbors become reliable only after augmentation invariance and within-domain organization are established [2509.04193].

## 4. Architecture, optimization, and retrieval inference

At the architectural level, DUDE combines four components: a trainable **ResNet-50** feature extractor \(f_\theta\); frozen Stable Diffusion components including the **VAE encoder**, **UNet noise predictor**, and **text encoder**; a momentum encoder \(f_m\); and domain-specific memory banks [2509.04193].

The optimization pipeline can be summarized as follows. During warm-up, \(f_\theta\) is trained with \(\mathcal{L}_{\text{OD}}\) so that \([z]\) becomes object-centric under domain-conditioned diffusion denoising. In the first progressive stage, the model adds instance and in-domain mutual-neighbor contrastive objectives. In the second stage, it maintains in-domain structure while introducing cross-domain mutual-neighbor alignment. This ordering is central to the method’s stability claims.

At inference, the diffusion model is discarded. Given a query \(I_q^A\) and target-domain database images \(\{I_j^B\}\), DUDE computes
\[
z_q = f_\theta(I_q^A),\quad z_j = f_\theta(I_j^B),
\]
then ranks database images by cosine similarity,
\[
\mathrm{sim}(z_q, z_j) = \frac{z_q^\top z_j}{\|z_q\|\,\|z_j\|}.
\]
Thus the final retrieval system is computationally similar to standard embedding-based retrieval despite the heavier training procedure [2509.04193].

This separation between training and deployment is operationally important. The method uses a generative prior to shape the representation, but retrieval itself remains a nearest-neighbor problem in the learned feature space.

## 5. Benchmarks, baselines, and empirical results

DUDE is evaluated on **three benchmarks over 13 domains**: PACS, Office-Home, and DomainNet, with DomainNet restricted to Clipart, Infograph, Painting, Real, and Sketch, excluding Quickdraw [2509.04193].

| Dataset | Domains | Evaluation |
|---|---|---|
| PACS | Photo, Art Painting, Cartoon, Sketch | P@50, P@100, P@200 |
| Office-Home | Art, Clipart, Product, Real | P@1, P@5, P@15 |
| DomainNet | Clipart, Infograph, Painting, Real, Sketch | P@50, P@100, P@200 |

The comparison set includes ID, ProtoNCE, CDS, PCS, DD, CoDA, and ProtoOT. A retrieved image is counted as correct if it shares the same category label as the query; labels are used only for evaluation [2509.04193].

On **PACS**, DUDE substantially improves over ProtoOT in average retrieval precision across all domain directions. The reported averages are **71.09 vs 50.26** at P@50, **67.49 vs 48.78** at P@100, and **61.78 vs 46.38** at P@200, corresponding to gains of **+20.83**, **+18.71**, and **+15.40** points. On individual directions, the gains can be even larger; for example, Art→Cartoon P@50 reaches **88.21** for DUDE versus **69.97** for ProtoOT [2509.04193].

On **Office-Home**, DUDE again improves the average performance over ProtoOT: **52.36 vs 43.85** at P@1, **50.84 vs 41.29** at P@5, and **48.04 vs 37.98** at P@15, i.e., **+8.51**, **+9.55**, and **+10.06**. The paper highlights Product→Real P@1 as **74.97 vs 64.01** [2509.04193].

On **DomainNet**, the reported average gains are particularly pronounced: **82.62 vs 66.63** at P@50, **80.07 vs 62.98** at P@100, and **74.90 vs 56.12** at P@200, corresponding to **+15.99**, **+17.09**, and **+18.78**. On difficult pairs such as Clipart→Sketch, ProtoOT achieves **70.46** at P@50 and **67.41** at P@200, whereas DUDE reaches **93.07** and **90.30**. For Sketch→Clipart, the corresponding values are **82.79** and **71.31** for ProtoOT versus **95.21** and **91.89** for DUDE [2509.04193].

The ablations clarify which components drive these gains. In prompt design, a **varying prompt** such as “a sketch/clipart/painting/... of a [z]” outperforms a **fixed prompt** such as “a photo of a [z]”; for Clipart→Sketch on DomainNet, P@50 is **75.76** with the varying prompt versus **66.42** with the fixed prompt. This supports the claim that explicit domain words help isolate style from object semantics.

Loss ablations show that disentanglement alone is already strong. A plain ResNet-50 baseline yields average **P@50(1)/100(5)/200(15) = 35.13 / 31.03 / 26.66**, whereas using only \(\mathcal{L}_{\text{OD}}\) lifts this to **50.41 / 46.78 / 41.93**. The full combination \(\mathcal{L}_{\text{OD}} + \mathcal{L}_{\mathrm{PA1}} + \mathcal{L}_{\mathrm{PA2}}\) reaches **68.69 / 66.23 / 61.57**, the best overall configuration [2509.04193].

## 6. Interpretation, limitations, and nomenclature

The most direct interpretation of DUDE is that it treats domain shift in UCIR as partly a **representation factorization** problem rather than only an alignment problem. By forcing style into an explicit domain token and object semantics into \([z]\), it constructs features that are more amenable to neighbor-based cross-domain alignment. The reported t-SNE observations are consistent with this: DUDE produces more compact class clusters and less domain separation than ProtoOT. This suggests that semantic neighborhoods become more reliable once style contamination is reduced [2509.04193].

The method also has evident limitations. Training is heavier than purely discriminative UCIR because Stable Diffusion’s VAE and UNet remain in the loop, even though they are frozen. Performance depends on manually chosen domain-style words such as “clipart,” “painting,” or “infograph.” The paper also notes sensitivity to hyperparameters such as neighborhood size \(k\) and the coefficients \(\beta\) and \(\lambda\), although it describes the method as reasonably robust. A further plausible concern, already implied in the paper, is dependence on Stable Diffusion’s semantic prior: if the pre-trained model does not capture a specialized domain well, the disentanglement mechanism may be less effective.

The proposed extensions are correspondingly natural: replacing Stable Diffusion with newer or domain-specific diffusion models, extending from pairwise to multi-domain alignment, and transferring the disentangled representation to other cross-domain tasks such as classification, segmentation, or domain adaptation.

The acronym itself is overloaded across fields. In arXiv literature, “DUDE” can also denote **Downlink/Uplink Decoupling** in cellular networks, **Discrete Universal DEnoiser** and its neural or sequencing variants, **Deep Unsigned Distance Embeddings**, **Document Understanding Dataset and Evaluation**, **Dual Distribution-Aware Context Prompt Learning**, and **Dual Decomposition of Weights and Singular Value Low Rank Adaptation** [1503.06746, 1511.04836, 1605.07779, 2011.02570, 2305.08455, 2407.04489, 2505.14367]. In the computer-vision retrieval context, however, DUDE specifically names a **diffusion-based UCIR method** whose defining contribution is the combination of diffusion-driven object-style disentanglement with progressive mutual-neighbor alignment.

Source: https://www.emergentmind.com/topics/dude