Retrieval-Augmented Super Resolution (RASR)

• Retrieval-Augmented Super Resolution (RASR) is a method that dynamically retrieves semantically relevant high-resolution images from a pre-curated database to improve low-resolution image reconstruction.
• The approach leverages a benchmark (RASR-Flickr30) and a baseline system (RASRNet) that integrate semantic retrieval with diffusion-based generation to enable scalable and realistic super-resolution.
• Experimental results indicate that RASRNet improves reconstruction fidelity with a +0.38 dB PSNR boost and a -0.0131 LPIPS reduction, producing more realistic textures compared to SISR baselines.

Retrieval-Augmented Super Resolution (RASR) is a practical reference-based super-resolution paradigm in which a system receives only a low-quality input image at inference time, automatically retrieves a semantically relevant high-resolution reference image from a reference database, and then generates a high-resolution reconstruction conditioned on both the low-quality input and the retrieved reference. It was introduced to overcome a critical limitation of existing Reference-based Super Resolution (RefSR) methods—their reliance on manually curated target-reference image pairs—and is framed as a way to make RefSR flexible and scalable in realistic use cases such as zoos, museums, and exhibitions, where category-specific reference data can be readily collected or pre-curated (Yan et al., 13 Aug 2025).

1. Definition and formal task

RASR is defined relative to two established super-resolution settings. Single Image Super Resolution (SISR) uses only the degraded image. Conventional RefSR uses the degraded image plus one or more manually assigned reference images. RASR differs from both by assuming that only the degraded image is available at test time, while permitting access to a reference database from which a suitable reference is retrieved automatically (Yan et al., 13 Aug 2025).

$$\text{RASR}(I_{LR})=\mathcal{G}(I_{LR}, I_{REF})=\mathcal{G}(I_{LR},\mathcal{R}(\mathcal{DB}, I_{LR})),$$

where $I_{LR}$ denotes the low-resolution input image, $\mathcal{DB}$ a pre-curated database of high-quality reference images, $\mathcal{R}$ a retriever, $\mathcal{G}$ a generator, and $I_{REF}$ the retrieved reference image selected from $\mathcal{DB}$ according to the low-resolution query $I_{LR}$ (Yan et al., 13 Aug 2025).

Setting	Inference input	Reference assumption
SISR	Low-resolution image only	No external image support
Conventional RefSR	Low-resolution image plus reference image(s)	Manually assigned references
RASR	Low-resolution image only, with database access	Automatic retrieval from $\mathcal{DB}$

The practical significance of this formalization lies in its removal of manual reference pairing. In prior benchmarks such as CUFED5 or LMR, each low-quality target is accompanied by predesigned references. RASR instead assumes a searchable gallery and dynamic retrieval, which the paper describes as more open-world retrieval than prior RefSR datasets because references are organized as per-category galleries rather than fixed pairs (Yan et al., 13 Aug 2025).

2. Benchmark design: RASR-Flickr30

A major contribution accompanying the RASR formulation is RASR-Flickr30, described as the first benchmark dataset designed for retrieval-augmented super-resolution. Its key design difference from previous RefSR datasets is that it does not provide fixed target-reference pairs; instead, it provides per-category reference databases so that a target image must query a gallery and obtain references dynamically (Yan et al., 13 Aug 2025).

The dataset is built from publicly available Flickr images. The authors manually crawled and curated high-resolution, aesthetically pleasing animal photographs, prioritizing high image quality, visual clarity, relevance, and sufficient images per species. The benchmark focuses on animals because that domain has natural visual diversity while preserving strong category consistency within species. RASR-Flickr30 contains 30 animal species, including both domestic and wild animals, with intra-class diversity in poses, backgrounds, lighting, and viewing angles (Yan et al., 13 Aug 2025).

Per-species partition	Count
Training images	40
Test images	5
Reference database images	Approximately 100

From this organization, the reported total scale is approximately $30 \times 40 = 1200$ training images, $I_{LR}$0 test images, and roughly $I_{LR}$1 reference database images. The paper explicitly reports only training and testing partitions; it does not provide a separate validation split description in the main text (Yan et al., 13 Aug 2025).

For training, all training images and reference database images are cropped into patches of size $I_{LR}$2, and for each training sample paired patches are randomly cropped to $I_{LR}$3. To produce LR-HR pairs for both training and testing, the benchmark uses the Real-ESRGAN degradation pipeline, so the degradation model is realistic synthetic degradation rather than simple bicubic downsampling. Evaluation uses both full-reference and no-reference metrics: PSNR, SSIM, LPIPS, DISTS, FID, NIQE, MUSIQ, and CLIPIQA (Yan et al., 13 Aug 2025).

The benchmark is explicitly contrasted with datasets such as CUFED5 and LMR. Previous datasets provide pre-designed references per input, have lower training resolution, are less suitable for realistic diffusion-based training, and do not support dynamic retrieval. RASR-Flickr30 instead provides higher-quality images, per-category reference galleries, dynamic retrieval, and a more open-world usage pattern within category-specific galleries (Yan et al., 13 Aug 2025).

3. Retrieval and generation in RASRNet

The baseline system proposed for the task is RASRNet, which contains two modules: RIRR (Real-world Image Reference Retriever) and RefDSR (Reference Diffusion Super Resolution). The pipeline is defined in four steps: input a low-quality image $I_{LR}$4, use a semantic retriever to find the most relevant HR reference $I_{LR}$5 in the database, feed $I_{LR}$6, $I_{LR}$7, and text prompts into a diffusion-based generator, and generate the super-resolved output $I_{LR}$8 (Yan et al., 13 Aug 2025).

The retrieval module is intentionally semantic rather than texture-based. The paper argues that texture matching is unreliable on heavily degraded LR images because blur and noise corrupt local patterns, whereas semantic retrieval based on high-level visual embeddings is more robust under realistic degradations. The retriever uses a vision encoder, with DINOv2 cited as the example and ultimately chosen as the default in experiments; the paper also compares VGG16, CLIP, and DINOv2. All database images are encoded into visual embeddings and stored in a searchable index. At inference, the LR image is encoded using the same vision encoder, cosine similarity is computed between the query embedding and all database embeddings, and the nearest neighbor is returned. Retrieval is therefore semantic, nearest-neighbor, cosine-similarity based, and top-1 at inference (Yan et al., 13 Aug 2025).

During training, the system uses a different reference-selection procedure. For each training sample, five reference images are pre-selected based on cosine similarity to the GT image, and one of those five is randomly sampled for training. The paper does not describe end-to-end training or fine-tuning of the retriever; experimentally it behaves as a frozen retrieval module (Yan et al., 13 Aug 2025).

The generator is written as a conditional reference-guided diffusion pipeline:

$I_{LR}$9

where $\mathcal{DB}$0 is the frozen VAE encoder for the LR input, $\mathcal{DB}$1 is the frozen VAE decoder, $\mathcal{DB}$2 is the pretrained latent diffusion backbone for SISR, $\mathcal{DB}$3 denotes the trainable reference-conditioning branch, $\mathcal{DB}$4 denotes the encoder for the reference image, and $\mathcal{DB}$5 denotes text conditioning (Yan et al., 13 Aug 2025).

In implementation, the pretrained SISR diffusion backbone is OSEDiff, and the pretrained model for the ControlNet branch is SD 2.1-base. The architecture includes a frozen VAE encoder, a frozen VAE decoder, a frozen latent diffusion network, and a trainable ControlNet-like branch. All pretrained backbone parameters are frozen to preserve the original generative capability. The retrieved reference $\mathcal{DB}$6 is encoded by a separate frozen VAE encoder $\mathcal{DB}$7, and the encoded latent $\mathcal{DB}$8 is used as conditioning input to the ControlNet branch. The reference encoding is explicitly done without text conditioning to obtain an unconditioned latent representation (Yan et al., 13 Aug 2025).

A central architectural choice is that reference conditioning is injected only in the first three decoder blocks, while text cross-attention is reserved for the final decoder block. The rationale given is that the retrieved reference is only semantically similar rather than spatially aligned, so injecting it too early or too broadly can mislead low-level reconstruction. The fusion rule is

$\mathcal{DB}$9

This means that in decoder blocks $\mathcal{R}$0, the UNet features are augmented by adding corresponding ControlNet features, while text-based modulation is omitted in these blocks (Yan et al., 13 Aug 2025).

4. Conditioning, losses, and optimization

Text conditioning is not treated as incidental. Text prompts are extracted separately from the LR image and the retrieved reference, then concatenated as the text input for the UNet. During training, prompts are extracted from GT images. During inference, the system uses a combination of prompts from the LR input and the retrieved reference because a fine-tuned text extractor on LR alone can be unreliable. The reported ablation is qualitative but directional: no prompt is worse, LR-only prompts help, REF-only prompts help more, and REF+LR is best (Yan et al., 13 Aug 2025).

The paper also modifies feature fusion at inference. Using the same full-strength feature fusion at inference gives high fidelity but poorer perceptual quality, so the reference injection strength is reduced by a factor of $\mathcal{R}$1:

$\mathcal{R}$2

This is a practical implementation detail intended to balance fidelity and realism (Yan et al., 13 Aug 2025).

Training is described in terms of image-level supervision on the generated output rather than a standard diffusion noise-prediction exposition. The losses are MSE loss, LPIPS perceptual loss, GAN loss, and Gram loss. The Gram loss is defined as

$\mathcal{R}$3

where $\mathcal{R}$4 is the Gram matrix of VGG-16 features at layer $\mathcal{R}$5, $\mathcal{R}$6 is the layer weight, and $\mathcal{R}$7 is the ground-truth image. The total loss is a weighted sum of MSE, LPIPS, Gram, and GAN losses, and the discriminator for the GAN loss uses a fixed DINO backbone, following DINO-GAN (Yan et al., 13 Aug 2025).

The generator is trained on RASR-Flickr30 and LSDIR. For LSDIR, two random $\mathcal{R}$8 patches are cropped from each image to form target-reference pairs, which the paper uses to increase local diversity while preserving global semantic consistency. The training procedure targets the generator rather than joint optimization of retriever and generator. The retriever appears separate and frozen, and the pretrained diffusion backbone is also frozen. Only lightweight modules are trained: LoRA modules and zero-convolution layers in the DownBlocks of the ControlNet branch. The optimizer is AdamW, the learning rate is $\mathcal{R}$9, the batch size is 24, the hardware is 4 NVIDIA A800 GPUs, and training time is approximately 33 hours. Reported hyperparameters include LoRA rank 4, $\mathcal{G}$0, $\mathcal{G}$1, and $\mathcal{G}$2. The paper does not report contrastive retriever training, hard negative mining, or joint retrieval learning (Yan et al., 13 Aug 2025).

5. Evaluation and empirical findings

The central experimental claim is that RASRNet consistently improves over SISR baselines on RASR-Flickr30, achieving +0.38 dB PSNR and -0.0131 LPIPS, while generating more realistic textures. The strongest baseline is reported to be OSEDiff, and the improvement is framed as evidence that retrieval augmentation can bridge the gap between academic RefSR research and real-world applicability (Yan et al., 13 Aug 2025).

The benchmark evaluates the entire retrieval-augmented pipeline rather than only the reconstruction stage. For RefSR baselines in the retrieval-based setting, the authors use references retrieved by their own RIRR module. This evaluation protocol matters because it removes the simplifying assumption of oracle or manually assigned references and instead measures performance under the intended deployment condition of dynamic retrieval (Yan et al., 13 Aug 2025).

The reported ablations clarify which forms of conditioning contribute to the baseline’s performance. Prompt ablation indicates that semantic text extracted from both the LR input and the reference is preferable to either source alone. Architectural ablation favors semantic retrieval over texture matching for degraded LR inputs, and the selective injection of reference features into the first three decoder blocks reflects the paper’s position that semantically similar but spatially unaligned references should guide higher-level decoding rather than directly perturb the entire network. A plausible implication is that RASR, in the form presented here, treats retrieval primarily as a semantic prior rather than as an alignment-ready texture donor (Yan et al., 13 Aug 2025).

6. Scope, misconceptions, and neighboring super-resolution paradigms

A common misconception is to treat any super-resolution method with auxiliary information as a RASR method. The defining condition of RASR is narrower: input at inference is only the degraded image $\mathcal{G}$3, but the system can access a reference database $\mathcal{G}$4 and retrieve a suitable reference automatically. Methods that use manually paired references belong to conventional RefSR rather than RASR, and methods with no retrieval module belong elsewhere (Yan et al., 13 Aug 2025).

This distinction is visible in adjacent work on artifact-aware super-resolution for compressed video. A lightweight CNN for joint compression artifact reduction and super-resolution is formulated as a single-stage restoration-and-upscaling problem for low-bitrate compressed streams, and the paper explicitly states that it processes only one frame at a time and contains no retrieval module. Its relevance to RASR is instead problem-formulation oriented: it highlights that practical degradations in streaming settings are mixed—downsampling plus codec distortion plus bitrate dependence—and that super-resolution-only approaches tend to amplify artifacts along with details (Ma et al., 2024). This suggests that practical RASR systems may require degradation models beyond clean LR inputs or generic synthetic blur/downsampling.

The boundary is also clear in optics-oriented super-resolution. The one-bit coded diffraction pattern framework studies a setting in which phase-less Fourier measurements are collected under modulation or masking, and the contribution is not retrieval augmentation per se, but a recovery mechanism that turns difficult magnitude-only Fourier measurements into a spectral estimation problem from pairwise order comparisons. In that work, mask modulation is crucial because it mixes spatial frequencies, effectively shifting otherwise inaccessible high-frequency content into the observable passband, thereby enabling super-resolution beyond the diffraction limit of a given imaging system (Mroueh, 2014). A plausible implication is that RASR should be understood as one member of a broader super-resolution landscape in which “augmentation” may arise from retrieved references, coded measurements, or other auxiliary evidence, even though only the first is retrieval-based in the strict sense.

A further neighboring case is RASLF, a representation-aware state-space framework for light field super-resolution. It is explicitly not a retrieval-based SR paper: there is no nearest-neighbor retrieval, no external database, no patch matching, no cross-image memory bank, and no explicit retrieval-conditioned synthesis. Its stated relevance to RASR is structural rather than definitional. The paper argues that different evidence sources should not be processed identically and offers representation-aware fusion mechanisms—Progressive Geometric Refinement, Representation Aware Asymmetric Scanning, and Dual-Anchor Aggregation—that are presented as directly useful to future retrieval-augmented SR systems, especially where retrieved textures must be fused with geometry-aware processing (Wei et al., 17 Mar 2026). This suggests a broader research direction in which RASR moves from simple conditioning on a single retrieved image toward more explicit organization of heterogeneous evidence spaces.