RealIR-Bench: Real-World Restoration Benchmark

• RealIR-Bench is a real-world, non-reference benchmark that evaluates image restoration without relying on clean paired references.
• It combines the Restoration Score (RS) for degradation removal with LPIPS/LPS for measuring content preservation to ensure balanced performance.
• The benchmark curates 464 authentic images across nine degradation types to robustly assess large-scale image restoration models.

RealIR-Bench is a real-world, non-reference benchmark for image restoration under authentic degradations where clean references are unavailable. It was introduced in connection with RealRestorer to address a gap left by benchmarks that are synthetic, reference-based, or restricted to a single degradation type, and it is explicitly designed to evaluate two properties at once: degradation removal and content consistency preservation (Yang et al., 26 Mar 2026).

1. Definition and motivation

RealIR-Bench was created for real-world image restoration rather than restoration under simulated corruption. The underlying motivation is that restoration is a preprocessing stage for downstream systems such as autonomous driving, object detection, remote sensing, and 3D reconstruction, so an evaluation protocol should measure not only whether a method removes blur, haze, rain, flare, or noise, but also whether it preserves scene structure and important content (Yang et al., 26 Mar 2026).

The benchmark is positioned against three limitations of conventional restoration evaluation. First, existing benchmarks often use synthetic corruptions, which are too cleanly modeled compared with real-world degradations. Second, they often assume a clean paired reference, which is usually unavailable for internet-sourced real images. Third, they typically focus on a narrow degradation distribution, whereas real images often contain mixed, spatially uneven, and domain-dependent corruptions such as blur plus flare, or haze plus compression artifacts. The paper also explicitly states that evaluation should not reward models that “over-restore” by changing image content. In that formulation, restoration quality is not reducible to perceptual change alone; it must be constrained by scene faithfulness (Yang et al., 26 Mar 2026).

This design implies a benchmark philosophy distinct from paired restoration datasets. RealIR-Bench does not ask whether an output matches a hidden ground-truth image pixelwise. Instead, it asks whether a model removes the targeted degradation while remaining faithful to the degraded input’s scene identity, structure, and details. A plausible implication is that the benchmark is intended for settings in which restoration errors can propagate into downstream failures even when outputs appear visually plausible.

2. Dataset composition and curation

RealIR-Bench contains 464 degraded images collected from the web and manually curated. It is built entirely from real-world non-reference degraded images, and the benchmark spans nine degradation categories: blur, rain, noise, low-light, moiré patterns, haze, compression artifacts, reflection, and flare (Yang et al., 26 Mar 2026).

The benchmark is designed so that each sample belongs to one of these real-world restoration tasks, and each category is evaluated with a fixed instruction prompt to reduce prompt variability across methods. The appendix further states that the real-world degradation data used for training and curation came from high-quality online sources such as Pexels and Pinterest, and that the benchmark itself is manually filtered to preserve diversity in scene content, degradation severity, and artifact type.

Component	Specification
Total images	464 degraded images
Data type	Real-world non-reference degraded images
Degradation categories	blur, rain, noise, low-light, moiré patterns, haze, compression artifacts, reflection, flare

The dataset design differs from earlier restoration benchmarks in several specific ways. It is non-reference by design, uses authentic real-world degradations rather than simulated corruption alone, covers nine common degradations in one benchmark, explicitly evaluates content consistency, and is benchmarked with large image editing models rather than only with classical restoration networks. This suggests that RealIR-Bench is intended as an all-in-one evaluation set for broad real-world restoration behavior rather than as a narrowly scoped diagnostic benchmark.

3. Evaluation protocol and metric design

Because RealIR-Bench has no paired clean references, the paper introduces a non-reference evaluation protocol based on two complementary metrics: Restoration Score (RS) for degradation removal and LPIPS / LPS for content consistency preservation (Yang et al., 26 Mar 2026).

Restoration Score (RS) is based on a vision-LLM evaluator, specifically Qwen3-VL-8B-Instruct. The evaluator scores both the degraded image and the restored image on a 1–5 degradation-severity scale, where 5 = no visible degradation / clean and 1 = extreme degradation. The instruction requires the evaluator to inspect the image, estimate severity of the target degradation only, and output a score from 1 to 5 while ignoring unrelated aesthetic issues. The appendix gives the prompt logic as: inspect local regions, for example through a 3×3 grid; judge whether the target degradation exists; estimate severity and affected area; aggregate into a single image-level score; and output only Degradation Score: \<1--5>. RS is then defined as the improvement in degradation level after restoration, i.e., the difference between the degradation score of the restored image and that of the degraded image. Higher RS means the restored output is judged cleaner than the input.

LPIPS, reported in the tables as LPS, measures perceptual distance between the restored output and the degraded input. In this setting it serves as a consistency metric: smaller values indicate better content preservation. The paper uses LPS in combination with RS to avoid rewarding outputs that remove degradations by altering scene content too aggressively.

After normalizing both RS and LPS to the same scale, the paper defines the combined score as

$FS = 0.2 \, (1 - LPS) \, RS$

The intended interpretation is explicit. RS asks whether the model actually removes the targeted corruption. LPS / LPIPS asks whether the model keeps the original scene consistent. FS asks whether it does both at once. If a model removes degradation well but changes the scene too much, LPS worsens and FS drops; if it preserves content but fails to remove degradation, RS remains low and FS also drops.

The paper further reports alignment between FS and subjective preference from a user study. The correlations are Kendall’s $\tau_b = 0.2493$, SRCC = 0.3010, and PLCC = 0.2919, with $p = 2.96 \times 10^{-124}$, $p = 4.62 \times 10^{-128}$, and $p = 3.21 \times 10^{-120}$, respectively. The paper characterizes these as modest but statistically significant correlations, supporting the metric’s usefulness (Yang et al., 26 Mar 2026).

4. Benchmarking role in large-scale image editing models

A central theme of the RealRestorer paper is that large-scale image editing models have become strong candidates for restoration because they bring broad semantic priors learned from large image-text or editing corpora. Within that framing, RealIR-Bench serves as the non-reference evaluation environment for testing whether such models can restore authentic degradations while preserving consistency (Yang et al., 26 Mar 2026).

The benchmark is used to evaluate both open-source and closed-source large editing models, including Nano Banana Pro, GPT-Image-1.5, Seedream 4.5, Qwen-Image-Edit-2511, FLUX.1-Kontext-dev, LongCat-Image-Edit, and Step1X-Edit. RealRestorer itself is obtained by fine-tuning Step1X-Edit, described as a diffusion-in-transformer editing model.

The associated training strategy for RealRestorer is two-stage. The first stage is transfer training on synthetic degradation data, which teaches basic restoration behavior across nine tasks. The second stage is supervised fine-tuning on real-world degraded-clean pairs, which adapts the model to authentic real-world statistics while preserving generalization. The paper also uses a progressively mixed training strategy, retaining a small amount of synthetic data in the second stage to avoid overfitting to a narrow real-world distribution.

In this experimental setting, RealIR-Bench functions as a realism-oriented benchmark for testing whether large editing models can generalize beyond synthetic restoration objectives. A plausible implication is that the benchmark is particularly informative for models whose outputs may be semantically plausible but structurally inconsistent, since such behavior is penalized by the LPS term in the final score.

5. Empirical findings on RealIR-Bench

On RealIR-Bench, the paper reports that RealRestorer ranks first among open-source models and third overall, narrowing the gap with Nano Banana Pro to only 0.007 points and surpassing Qwen-Image-Edit-2511 by 0.019 points in the overall score (Yang et al., 26 Mar 2026). The user study reports that Nano Banana Pro had the highest first-ranking rate at 32.02%, followed by GPT-Image-1.5 at 23.83%, while RealRestorer achieved 21.54%.

Across the nine tasks, RealRestorer is reported as best on deblurring, best on low-light enhancement, second on moiré pattern removal, first among open-source methods on five tasks, and second among open-source methods on two tasks. The paper emphasizes that these results reflect a strong balance between restoration strength and content preservation.

For the subset of five tasks in the first table, the reported values are:

• Rain Removal: RS = 1.076, FS = 0.135
• Deblurring: RS = 1.900, FS = 0.159
• Low-light Enhancement: RS = 1.360, FS = 0.110
• Haze Removal: RS = 0.680, FS = 0.090
• Reflection Removal: RS = 1.620, FS = 0.230

For the remaining four tasks and the average:

• Deflare: RS = 1.623, FS = 0.247
• Moiré Patterns Removal: RS = 1.620, FS = 0.142
• Denoise: RS = 0.863, FS = 0.090
• Compression Restoration: RS = 1.067, FS = 0.097
• Avg Total (9): LPS = 0.445, RS = 1.312, FS = 0.146

These results are complemented by evaluation on FoundIR, a paired benchmark on which the paper reports PSNR and SSIM. RealRestorer obtains the best PSNR/SSIM on 5 of 7 degradations among the compared models, with average PSNR = 21.45 and SSIM = 0.70. Although FoundIR is not part of RealIR-Bench, the comparison is used in the paper to show that performance on the non-reference benchmark is not isolated from performance on a paired evaluation setting.

6. Significance, use cases, and limitations

The practical significance of RealIR-Bench is tied directly to downstream robustness. The paper states that real-world restoration matters for autonomous driving and object detection, because poor restoration may leave blur, haze, rain, flare, or noise that harms detectors and planners, while over-aggressive restoration may distort shapes, move objects, or remove evidence needed for recognition (Yang et al., 26 Mar 2026). RealIR-Bench is useful in this context because it tests exactly two properties: whether a method removes the corruption and whether it preserves geometry and content so that detection and planning remain reliable.

Methodologically, the benchmark’s main contribution is its refusal to treat restoration as a purely perceptual enhancement problem. By combining a VLM-based degradation-removal score with a perceptual consistency term, it formalizes a trade-off that earlier synthetic or paired benchmarks often leave implicit. This suggests a broader evaluation principle for real-world restoration: non-reference assessment is not merely a fallback when clean targets are absent, but a mechanism for measuring failure modes that paired metrics may understate.

The benchmark also inherits the limits of its design. It is non-reference by construction, so its evaluation depends on an external judge for degradation severity and on LPIPS-based consistency measurement rather than on direct comparison to aligned clean targets. The reported human correlations are statistically significant but modest. A plausible implication is that RealIR-Bench should be read as a realism-oriented benchmark for authentic degradations rather than as a replacement for all paired restoration benchmarks. Within the paper’s framework, its role is complementary: it measures behavior on uncontrolled internet imagery that traditional PSNR/SSIM-centered evaluation does not faithfully capture.