WideRealSR: Real-World Imaging & Signal Benchmarks

• The paper introduces WideRealSR’s imaging benchmark, which uses naturally degraded, unpaired low-resolution images to evaluate super-resolution model generalization.
• The signal recognition benchmark simulates diverse RF modulations with randomized SNRs, enabling precise time-frequency IoU evaluations for segmentation models.
• Both datasets emphasize the limitations of synthetic training and advocate kernel adaptation and robust augmentation strategies for real-world deployment.

WideRealSR refers to two distinct benchmark datasets within the machine learning literature: a photographic super-resolution testbed (“How Real is Real: Evaluating the Robustness of Real-World Super Resolution” (Deviyani et al., 2022)) and a wideband spectrum recognition collection (“A Wideband Signal Recognition Dataset” (West et al., 2021)). Each dataset targets a fundamentally different domain—image restoration and signal identification—yet both address the core challenge of generalization to real-world, naturally degraded inputs absent idealized ground-truth labels.

1. WideRealSR for Real-World Super-Resolution in Imaging

WideRealSR, as introduced by Deviyani et al. (Deviyani et al., 2022), is a curated collection of real, low-resolution photographs acquired from heterogeneous sources, specifically constructed to benchmark model generalization beyond synthetic bicubic downsampling. Comprising approximately 35–105 images sourced from 35 discrete sensor noise categories, it encompasses mobile phones, tablets, broadcast and surveillance systems, automotive-mounted cameras, drones, microscopy (malaria slides), social network compressions, mapping applications, and satellite platforms. The images vary substantially in their native resolutions, stretching from a few hundred pixels per axis (e.g., microscopy regions) up to several thousand (DSLR or satellite imagery).

Notably, WideRealSR contains only original, unfiltered low-resolution samples; no paired high-resolution ground-truth counterparts are present. Preprocessing is deliberately minimal, limited to cropping for border removal when essential and otherwise preserving all native sensor and encoding artifacts. The dataset includes source/sensor-type metadata solely for facilitating per-domain comparisons. No additional semantic, box, or high-resolution annotations are provided.

2. WideRealSR for Wideband RF Signal Recognition

The WideRealSR signal recognition dataset, as defined by O'Shea et al. (West et al., 2021), is an extensive, programmatically generated collection aimed at spatiotemporal detection, localization, and classification of wideband RF signals. The dataset consists of 260 training and 130 test “SigMF records,” each comprising 100 million complex‐int16 I/Q samples. Every record emulates a real-world deployment profile, with up to 16 frequency layout schemas spanning ISM, cellular, PCS, and public-safety bands.

It contains 13 modulation classes: 2-PSK, 4-PSK, 8-PSK, 16-QAM, 64-QAM, 256-QAM, OFDM (512 subcarriers), 2-FSK, 4-FSK, GMSK, OOK, AM-DSB, AM-SSB, and FM. Parameters such as center frequency, signal bandwidth (tens of kHz to ≈1 MHz), duration (hundreds of µs to tens of ms), and per-burst SNR (+30 dB to –10 dB) are randomized for realism. Every signal event is annotated in SigMF JSON with precise time-frequency bounding intervals, class labels, and optional SNR values.

3. Data Generation, Annotation, and Preprocessing Procedures

For the photographic WideRealSR, all images originate as naturally degraded, in-the-wild captures. HR–LR pairing is not undertaken; instead, degradation kernel estimation (e.g., via KernelGAN) is later leveraged to synthesize LR–HR pairs from external corpora (e.g., DIV2K) for controlled training experiments. The test set exclusively contains unmodified, real-world, low-resolution images from diverse domains, with no artificial downsampling applied.

Conversely, the signal recognition WideRealSR data are entirely synthetic, generated using DeepSig’s waveform libraries with randomized modulation, duration, bandwidth, and symbol content (real audio for analog modulations). Each “burst” is injected into the wideband record and superimposed, with annotation via SigMF specifying burst start/end times, frequency bounds, modulation type, and SNR (for diagnostics). No multipath fading or real-world channel effects beyond overlapping sidelobes are simulated in the baseline release.

Preprocessing in the signal domain consists of conversion to log-magnitude spectrograms (512 × 512) for neural model input, followed by channel-wise normalization.

4. Evaluation Protocols and Metrics

Photographic WideRealSR cannot be assessed using canonical pixel-aligned metrics (PSNR, SSIM) due to the absence of HR ground truth. Instead, the paper proposes human perceptual studies (30 participants per source domain), wherein raters select the most visually plausible reconstruction from five state-of-the-art SR methods (RealSR, USRNet, ESRGAN, EnhanceNet, DPSR). Results indicate pronounced domain specificity, with RealSR excelling for smartphone images but failing on StreetView and other sources—linked to its restricted DPED training distribution. Qualitative illustrations highlight mode failure: spurious artifacts may emerge for out-of-distribution kernels, while retraining on cluster-specific synthetic data ameliorates such shortcomings.

In the signal recognition dataset, evaluation is conducted using time-frequency intersection-over-union (IoU), with a threshold τ = 0.5 to declare a true positive. Precision, recall, and F₁ scores are computed over the entire hold-out test partition, across the SNR continuum ([+30, –10] dB). The baseline U-Net segmentation model achieves recall and precision surpassing 90% for SNRs above +5 dB, with F₁ > 0.9 in favorable conditions. Confusions primarily arise for wideband OFDM and FM signals (merged masks) and for low SNR (<0 dB).

5. Benchmarking Insights, Model Robustness, and Best Practices

The photographic WideRealSR dataset reveals critical inadequacies in SR model generalization when applied beyond synthetic bicubic or DPED-style domains. The overfitting of specialized SR models (e.g., RealSR trained on a narrow phone subset) to their source domain is evident. The authors propose a kernel clustering and adaptation strategy: estimate the degradation kernel of each test image, assign it to the nearest cluster via k-means on KernelGAN outputs, and apply a cluster-trained SR model for inference. Human perceptual criteria are advocated for holistic model selection, as standard quantitative metrics cannot be computed.

In the signal recognition domain, randomized training-time SNR augmentation and cross-profile validation are necessary to avoid overfitting to any particular frequency schema or noise regime. Channel-wise data-driven mask thresholding and filtering of small connected components in U-Net outputs optimize precision at moderate SNRs. Leverage of SigMF metadata supports dynamic augmentation strategies (frequency shifting, amplitude rotation, jitter injection).

6. Limitations, Common Misconceptions, and Future Directions

Both versions of WideRealSR highlight intrinsic challenges in dataset design for robust model evaluation. For image SR, the absence of realistic LR–HR pairs and the preponderance of simplistic downsampling in existing benchmarks have fostered models ill-equipped for natural image degradations. The dataset’s lack of internal train/val splits, limited metadata, and sample size may constrain fully quantitative analysis or granular stratification.

A plausible implication is that unified SR models are fundamentally limited in real-world generalization without explicit adaptation strategies. Future work should prioritize (1) expansion of real-world LR testbeds, (2) development of perceptual metrics tightly coupled to human evaluation, and (3) standardized procedures for kernel clustering and model assignment.

WideRealSR for signal recognition similarly foregrounds the tension between synthetic realism and annotation tractability. The absence of uncontrolled channel impairments may simplify the task relative to field deployment. Recommendations emphasize validation against diverse frequency layouts, robust SNR augmentation, and further investigation into segmentation-based detection under severe noise and class ambiguity.

7. Comparative Overview

Dataset Variant	Domain	Data Type	Ground-Truth Format
WideRealSR (Imaging)	Photography	Real LR images	Sensor/source-type metadata
WideRealSR (Signal Recognition)	RF Signals	Synthetic I/Q & bursts	SigMF time/freq bounding boxes

Both datasets set rigorous standards for robustness evaluation in their respective fields and illustrate the ongoing need for diverse, realistic, and meticulously annotated benchmarks to drive progress in generalizable model design.