RASRNet: Retrieval-Augmented Super Resolution

• RASRNet is a retrieval-augmented super resolution framework that automatically retrieves semantically relevant high-quality images to guide restoration.
• It employs DINOv2-based Real-world Image Reference Retriever (RIRR) to overcome limitations of manual pairing, ensuring robust matching under degradation.
• The RefDSR generator integrates ControlNet and pretrained diffusion models with text conditioning to achieve state-of-the-art perceptual quality and enhanced metrics.

Searching arXiv for the primary paper and closely related reference-based SR work to ground the article. RASRNet is the baseline model introduced within Retrieval-Augmented Super Resolution (RASR), a practical reference-based image restoration paradigm that replaces manually curated target–reference pairs with automatic retrieval from a pre-curated high-quality reference database. In this formulation, a degraded low-resolution image $I_{\text{LR}}$ is first matched to a semantically relevant reference image $I_{\text{REF}}$ by a retriever $\mathcal{R}$, and a generator $\mathcal{G}$ then performs super-resolution conditioned on both images. The paper defines the overall paradigm as

$$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$

with $\mathcal{DB}$ denoting the reference database (Yan et al., 13 Aug 2025). RASRNet instantiates this idea through two modules: the Real-world Image Reference Retriever (RIRR) and the Reference Diffusion Super Resolution generator (RefDSR), thereby framing reference-based super-resolution as retrieval-augmented diffusion-based restoration rather than manually paired cross-image transfer (Yan et al., 13 Aug 2025).

1. Position within reference-based super-resolution

Reference-based Super Resolution (RefSR) aims to improve super-resolution by using auxiliary high-resolution images that are semantically similar to the degraded input, enabling the recovery of more realistic and detailed textures such as fur, feathers, and fabric. In the formulation associated with RASRNet, the central limitation of earlier RefSR approaches is their assumption that each low-resolution test image is accompanied by a hand-picked high-resolution reference. Existing methods are described as often depending on patch-level correspondence and deformable alignment, and as breaking under real-world degradations such as noise, blur, and compression (Yan et al., 13 Aug 2025).

RASRNet is positioned against both Single Image Super Resolution (SISR) and traditional RefSR. SISR uses only the low-resolution image and therefore must hallucinate textures from internal priors or learned patterns, which the source describes as especially difficult under strong degradation. Traditional RefSR uses external references, but presumes manually curated target–reference pairs. RASR, by contrast, removes the manual pairing assumption and introduces semantic retrieval from category-specific databases, such as species-specific animal collections or curated museum archives. The generator then uses diffusion-based generative modeling to leverage references at the feature and semantic level, rather than relying on sensitive patch matching (Yan et al., 13 Aug 2025).

This design is explicitly motivated by real-world scenarios in which only a degraded photo and access to an image database are available. The source identifies zoo, museum, and exhibition photography as exemplary cases, because such environments can maintain category-specific galleries of high-quality images. A plausible implication is that RASRNet is intended less as a narrowly benchmark-optimized RefSR model than as a system-level proposal for deploying RefSR under open-world retrieval conditions.

2. Formal definition of the RASRNet pipeline

RASRNet consists of two principal components. The first is RIRR, the semantic reference retriever $\mathcal{R}$. The second is RefDSR, a diffusion-based RefSR generator $\mathcal{G}_\theta$. Conceptually, the pipeline is: low-resolution input $\rightarrow$ semantic retrieval from a per-category database $\rightarrow$ reference-guided diffusion super-resolution (Yan et al., 13 Aug 2025).

The retriever operates by embedding both database images and the low-resolution query in a semantic feature space using DINOv2, then performing nearest-neighbor search by cosine similarity. The similarity score is given as

$I_{\text{REF}}$0

where $I_{\text{REF}}$1 denotes the DINOv2 embedding. At inference time, the top-1 reference is used for generation; during training, the top-5 references are pre-selected and one is randomly sampled for robustness (Yan et al., 13 Aug 2025).

The generator is built on a pretrained SISR diffusion backbone, specifically OSEDiff, and augments it with ControlNet-based reference conditioning and semantic text conditioning. The paper gives the high-level generation equation as

$I_{\text{REF}}$2

where $I_{\text{REF}}$3 and $I_{\text{REF}}$4 are the VAE encoder and decoder, $I_{\text{REF}}$5 is the UNet-like diffusion network, $I_{\text{REF}}$6 is a frozen VAE encoder for the reference, $I_{\text{REF}}$7 is the ControlNet branch, and $I_{\text{REF}}$8 is a text-conditioning vector derived from the low-resolution image and the reference (Yan et al., 13 Aug 2025).

This architecture makes the retrieval step and the generation step separable but tightly coupled. The retriever supplies semantic relevance under degradation, while the generator is designed to consume that reference in a way that does not require spatially precise alignment.

3. Retrieval module: Real-world Image Reference Retriever

RIRR uses DINOv2 as its vision encoder. Offline, all reference images in the database are embedded and stored as an index. Online, the low-resolution query is encoded, cosine similarity is computed against the reference embeddings, and the most similar image is returned (Yan et al., 13 Aug 2025).

The justification for semantic retrieval rather than texture-based retrieval is direct in the source: semantic embeddings are said to be robust to blur and noise and to support cross-instance matching, such as matching different images of the same species. This is important because the retrieval target is not an aligned near-duplicate but a semantically compatible high-quality image that can guide restoration (Yan et al., 13 Aug 2025).

The paper reports an ablation comparing different retrieval encoders. Random reference selection yields LPIPS $I_{\text{REF}}$9 and FID $\mathcal{R}$0. VGG16 yields LPIPS $\mathcal{R}$1 and FID $\mathcal{R}$2. CLIP yields LPIPS $\mathcal{R}$3 and FID $\mathcal{R}$4. DINOv2 yields LPIPS $\mathcal{R}$5, FID $\mathcal{R}$6, and CLIPIQA $\mathcal{R}$7, which the paper identifies as the best overall perceptual quality among the tested retrieval encoders (Yan et al., 13 Aug 2025).

On RASR-Flickr30, the retrieval database is organized per animal species. For each of 30 species, the dataset contains approximately 40 training target images, approximately 5 test images, and approximately 100 reference database images used for retrieval. The paper also notes that nearest-neighbor search is efficient, reporting $\mathcal{R}$8 s average retrieval on an 84k-image LSDIR-based database with Qdrant, while RASR-Flickr30 is smaller (Yan et al., 13 Aug 2025).

A common misconception in reference-based restoration is that the reference must be highly aligned or visually near-identical to be useful. The RASRNet formulation explicitly rejects that assumption: retrieval is semantic, and the generator consumes the reference as semantic guidance rather than through patch-level matching. This suggests a shift in RefSR from correspondence-heavy transfer toward retrieval-conditioned generative restoration.

4. Generator design: RefDSR and semantic conditioning

RefDSR is built on the pretrained SISR diffusion backbone OSEDiff. The frozen components are the VAE encoder $\mathcal{R}$9, VAE decoder $\mathcal{G}$0, and UNet-like diffusion network $\mathcal{G}$1. The trainable additions are a ControlNet branch $\mathcal{G}$2, LoRA modules in ControlNet, and zero-convolution layers in the ControlNet down blocks. The ControlNet branch processes the reference image and injects its features into the diffusion decoder (Yan et al., 13 Aug 2025).

Feature fusion occurs in the first three decoder blocks. If $\mathcal{G}$3 is the feature in decoder block $\mathcal{G}$4 and $\mathcal{G}$5 is the corresponding ControlNet feature, then during training

$\mathcal{G}$6

At inference time, the reference contribution is scaled: $\mathcal{G}$7 The paper states that this inference-time scaling empirically improves perceptual scores while keeping reconstruction stable (Yan et al., 13 Aug 2025).

Text conditioning is handled separately from reference feature injection. Prompts are extracted from both $\mathcal{G}$8 and $\mathcal{G}$9 using an image-to-text model. During training, prompts come from the ground-truth image; during inference, concatenating prompts from both reference and low-resolution images ($$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$0) gives the best performance among the tested alternatives. Text cross-attention is applied only in the final decoder block, whereas early decoder blocks receive only reference feature fusion (Yan et al., 13 Aug 2025).

This division of labor is a defining design choice. Early decoding is used for structural detail under semantic guidance from the reference, while the final block refines global style and fidelity with text. The source explicitly notes that this avoids over-dependence on text when fine textures are being assembled and avoids over-constraining low-level reconstruction by reference structure (Yan et al., 13 Aug 2025). This suggests that RASRNet treats semantic cues as hierarchical: reference-guided local decoding first, text-mediated global refinement afterward.

5. Training procedure, objectives, and data regime

The OSEDiff backbone VAE and UNet are frozen during training. The ControlNet branch is trainable, with LoRA rank $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$1, together with zero-convolution layers in the down blocks. Optimization uses AdamW with learning rate $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$2. Training is reported on $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$3 NVIDIA A800 with batch size $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$4, and total training time is approximately 33 hours (Yan et al., 13 Aug 2025).

Training uses two data sources. On RASR-Flickr30, high-resolution images are cropped to $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$5 patches; for each training target, five reference images are pre-selected using cosine similarity to the ground truth, and one is randomly sampled; paired patches are then cropped to $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$6. On LSDIR, two $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$7 patches are randomly cropped per image to form target–reference pairs. The degradation model is Real-ESRGAN’s synthetic degradation pipeline for generating low-resolution–high-resolution pairs (Yan et al., 13 Aug 2025).

The total loss is

$$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$8

with $$\text{RASR}(I_{\text{LR}})=\mathcal{G}\big(I_{\text{LR}},\mathcal{R}(\mathcal{DB},I_{\text{LR}})\big),$$9, $\mathcal{DB}$0, and $\mathcal{DB}$1. The components are a pixel-wise MSE loss, LPIPS perceptual loss, Gram loss derived from VGG-16 Gram matrices, and GAN loss using a DINO-based discriminator following DINO-GAN (Yan et al., 13 Aug 2025).

The paper’s ablation identifies a characteristic trade-off. MSE+LPIPS alone gives the best PSNR but weakest perceptual metrics. Adding Gram improves texture-related metrics. Adding GAN improves realism metrics such as FID and NIQE but slightly drops PSNR. The combination of Gram+GAN gives the best overall perceptual quality, with the lowest NIQE and high CLIPIQA (Yan et al., 13 Aug 2025).

An important procedural point is that references used in training are not manually paired. They are automatically selected from the top-5 by cosine similarity, and random sampling among these five references is used to encourage robustness to imperfect matches. No patch-level SR–reference matching is required, because conditioning is applied at the latent or global feature level through ControlNet rather than explicit local correspondence (Yan et al., 13 Aug 2025).

6. Benchmarking on RASR-Flickr30 and reported performance

RASR-Flickr30 is described as the first benchmark dataset designed specifically for the RASR setting. It covers 30 animal species. Each species includes 40 training images, 5 test images, and approximately 100 reference database images. The images are described as high-resolution, visually clean, and aesthetically good, with intra-class diversity in pose, background, lighting, and viewpoint. Its defining property is the use of per-category reference databases that support open-world retrieval (Yan et al., 13 Aug 2025).

The evaluation setup includes fidelity metrics PSNR and SSIM; perceptual full-reference metrics LPIPS, DISTS, and FID; and no-reference metrics NIQE, MUSIQ, and CLIPIQA. In the retrieval-based setting on RASR-Flickr30, all RefSR baselines use the same RIRR retrieval module, SISR baselines ignore references, low-resolution images are generated using the Real-ESRGAN degradation pipeline, and RASRNet uses the top-1 retrieved reference together with its own generator (Yan et al., 13 Aug 2025).

The principal reported comparison is against the diffusion SISR baseline OSEDiff.

Method	PSNR	LPIPS
OSEDiff	23.29	0.3404
RASRNet	23.67	0.3273

The paper further reports that, relative to OSEDiff on the RASR-Flickr30 test set, RASRNet improves SSIM from baseline by $\mathcal{DB}$2, reduces DISTS by $\mathcal{DB}$3, reduces FID by $\mathcal{DB}$4, improves NIQE by $\mathcal{DB}$5, improves MUSIQ by $\mathcal{DB}$6, and improves CLIPIQA by $\mathcal{DB}$7. It states that RASRNet achieves the best values among all evaluated methods on LPIPS $\mathcal{DB}$8, DISTS $\mathcal{DB}$9, FID $\mathcal{R}$0, and NIQE $\mathcal{R}$1 (Yan et al., 13 Aug 2025).

On the WR-SR given-reference test set, where retrieval is not required, the paper evaluates RefDSR alone against diffusion-based SISR methods. OSEDiff achieves PSNR $\mathcal{R}$2, SSIM $\mathcal{R}$3, LPIPS $\mathcal{R}$4, FID $\mathcal{R}$5, and NIQE $\mathcal{R}$6. RefDSR achieves PSNR $\mathcal{R}$7, SSIM $\mathcal{R}$8, LPIPS $\mathcal{R}$9, FID $\mathcal{G}_\theta$0, and NIQE $\mathcal{G}_\theta$1 (Yan et al., 13 Aug 2025).

These results are significant primarily because the retrieval setting is more demanding than conventional RefSR evaluation. The model is not given an oracle reference; it must retrieve one. The reported gains therefore reflect the joint effectiveness of retrieval and generation, not merely a stronger generator under idealized pairing conditions.

7. Interpretation, significance, and limitations

The main significance of RASRNet lies in its reformulation of reference-based super-resolution around automatic retrieval. The source presents retrieval augmentation as a way to bridge the gap between academic RefSR research and real-world applicability, especially in environments where class-specific image collections can be assembled in advance (Yan et al., 13 Aug 2025). Its architecture also reflects a broader methodological transition in image restoration: from explicit alignment and correspondence estimation toward semantically conditioned latent generative models.

Several features distinguish the model’s practical orientation. First, the reference database is externalized as a reusable resource rather than a benchmark-fixed pairing. Second, semantic retrieval is explicitly designed to tolerate real-world degradation. Third, the generator is largely frozen, with trainable ControlNet and lightweight LoRA-based adaptation, which suggests an emphasis on leveraging pretrained diffusion priors rather than training an end-to-end RefSR system from scratch (Yan et al., 13 Aug 2025).

At the same time, the scope of the reported evidence is bounded by the data. The primary benchmark domain is animals, organized by species. The paper also cites product categories such as shoes, cars, and furniture as possible domains, but these are presented as application scenarios rather than empirical evaluations (Yan et al., 13 Aug 2025). This suggests that the generality of the paradigm is broader than the demonstrated benchmark, while the demonstrated performance is presently strongest in category-structured settings where visually coherent reference galleries exist.

Another possible misconception is that RASRNet is a fully generalized open-web retrieval system. The source does not support that reading. Retrieval is performed from per-category or pre-curated reference databases, and the practical examples are precisely those where such databases are available. A plausible implication is that RASRNet is best understood as an open-world RefSR framework within constrained semantic galleries, not as unconstrained internet-scale image retrieval for restoration.

In summary, RASRNet denotes a retrieval-augmented diffusion-based RefSR baseline composed of DINOv2-based semantic retrieval and ControlNet-enhanced latent diffusion super-resolution with semantic text conditioning. Its contribution is not merely an architectural variant, but a change in the operational definition of reference-based super-resolution itself: references are no longer presumed to be given, but must be discovered automatically from realistic databases and then exploited through semantic generative conditioning (Yan et al., 13 Aug 2025).