RRDataset: Robust Benchmark for AI-Generated Images

• RRDataset is a benchmark dataset designed to evaluate AI-generated image detectors under realistic conditions including scenario generalization, internet transmission, and re-digitization.
• It tests detectors across three dimensions, addressing shortcomings in existing benchmarks by including high-stakes scenarios and real-world degradations.
• The evaluation involves 17 detectors, 10 vision-language models, and a human study with 192 participants, highlighting performance gaps in deployment conditions.

RRDataset is a benchmark dataset for evaluating AI-generated image detectors under realistic, challenging conditions rather than only on idealized, clean test data. It is designed around three evaluation dimensions—scenario generalization, internet transmission robustness, and re-digitization robustness—and is paired with RRBench, which evaluates 17 detectors, 10 vision-LLMs (VLMs), and a human study with 192 participants. The benchmark is intended to bridge the gap between ideal evaluation and real-world deployment in AI-generated image detection (Li et al., 11 Sep 2025).

1. Motivation and position within robustness benchmarking

RRDataset was created in response to two shortcomings that the paper identifies in existing AI-generated image detection benchmarks. First, benchmarks such as GenImage, Fake2M, WildFake, and Chameleon are described as dominated by everyday-life imagery, so detectors are rarely tested on images from war and conflict, disasters, political events, medical and public health settings, culture and religion, or labor and production. Second, existing benchmarks generally ignore the fact that real images often undergo social-media compression and repeated forwarding, loss of resolution and color fidelity, and re-digitization through print, screen, or projection and re-capture. The paper argues that these omissions inflate reported detector performance because test images are often cleaner than the images encountered in practice (Li et al., 11 Sep 2025).

Within the broader literature on real-world robustness benchmarks, RRDataset belongs to a family of datasets that move evaluation away from clean-image assumptions and toward deployment conditions. ImageNet-S studies systematic error caused by inconsistent image decoding and resize implementations in classification pipelines (Wang et al., 2021). R-Bench evaluates large multimodal models under 33 corruption dimensions organized by the path from image capture to model reception (Li et al., 2024). RobustSpring extends corruption robustness benchmarking to dense correspondence tasks—optical flow, scene flow, and stereo—through 20 corruption types applied in a time-, stereo-, and depth-consistent manner (Schmalfuss et al., 14 May 2025). RRDataset is differentiated by its focus on AI-generated image detection and by the combination of high-stakes scenario coverage, internet transmission, and re-digitization (Li et al., 11 Sep 2025).

2. Dataset design and scenario coverage

RRDataset evaluates detectors across three dimensions.

Dimension	Coverage in RRDataset	Role in evaluation
Scenario generalization	Seven scenarios: War and Conflict, Disasters and Accidents, Political and Social Events, Medical and Public Health, Culture and Religion, Labor and Production, and everyday life	Tests whether detectors generalize beyond generic internet photos
Internet transmission robustness	Images undergo 2 to 6 transmission cycles across Telegram, WeChat, Facebook, QQ, WhatsApp, X, Instagram, and Tinder	Tests robustness to compression artifacts, reduced resolution, color distortion, and detail loss
Re-digitization robustness	Scanning a color printout, photographing a color printout, photographing an image displayed on a screen, and photographing a projected digital image	Tests robustness when images are converted from digital to physical and back to digital

For the six non-everyday categories, each scenario is expanded into 10 manually defined themes, and those themes are further enriched using Qwen2.5-7B-Instruct to create detailed prompts. This produces 24,000 prompts, which are used by SD 3.5 Large and Flux.1 to generate 48,000 images. After filtering with CLIP-score $> 0.27$ and an NSFW safety checker, the dataset retains 6,000 high-quality AI-generated images, corresponding to 1,000 per scenario. The real counterparts consist of 6,000 real images collected from openly licensed news and photo sources such as Reuters Pictures, Associated Press Images, BBC News In Pictures, UN Photo, and the International Committee of the Red Cross (Li et al., 11 Sep 2025).

For everyday life, 4,000 real images are gathered from COCO, CC3M-val, and Unsplash. AI counterparts are generated using prompts derived from COCO and CC3M captions with multiple generators: SD 3.5 Large, Flux.1, DALL-E 3, SD v1.4, SD v1.5, and Midjourney. Some AI images are also sampled from the Chameleon dataset, and low-resolution images from StyleGAN and ProGAN are included to better reflect real-world quality variation. The result is described as an everyday-life subset of 4,000 images. The paper also notes that it does not give a single final total image count in a compact formula, because the benchmark combines original, transmitted, and re-digitized images across multiple scenario groups (Li et al., 11 Sep 2025).

This design is explicitly intended to ensure that detectors are not only good at detecting fake social-media-style objects and scenes, but also at detecting AI-generated content in high-risk contexts where the cost of failure is higher. A plausible implication is that scenario coverage is treated not merely as class diversity, but as a robustness variable tied to the social consequences of false negatives and false positives.

3. Internet transmission and re-digitization protocols

The internet transmission dimension applies multiple rounds of transmission to all 10,000 real images and 10,000 AI-generated images. These images are sent through Telegram, WeChat, Facebook, QQ, WhatsApp, X, Instagram, and Tinder. Each image undergoes 2 to 6 transmission cycles, covering both cross-platform transmission and single-platform repeated forwarding. The simulated multi-transmission workflow uses the following transmission-time distributions: 10% for 1–2 times, 25% for 2–3 times, 25% for 2–4 times, 25% for 3–5 times, and 15% for 4–6 times. The paper states that this is meant to mimic levels of information spread such as direct private messaging, limited forwarding, and repeated forwarding of trending news (Li et al., 11 Sep 2025).

The re-digitization dimension applies four methods with equal probability to 10,000 real images and 10,000 AI-generated images: scanning a color printout, photographing a color printout, photographing an image displayed on a screen, and photographing a projected digital image. The resulting re-digitized images are then added to RRDataset. The paper emphasizes that the original real/fake label is determined by the source, not by how the image looks after re-digitization (Li et al., 11 Sep 2025).

These two dimensions are central because they target failure modes that are structurally different from conventional clean-image evaluation. Internet transmission introduces compression artifacts, reduced resolution, color distortion, and detail loss. Re-digitization alters color fidelity, texture, illumination, perspective, and geometric consistency. The benchmark therefore treats deployment transformations as part of the evaluation target rather than as incidental nuisance factors.

4. RRBench protocol, metrics, and evaluated models

RRBench evaluates accuracy on real images and accuracy on AI-generated images as the primary metrics. Because the dataset is balanced, recall and precision are straightforward to compute, though the reported table focuses on class-wise accuracy and overall accuracy. The benchmark compares 17 AI-generated image detectors, 10 VLMs, and human observers (Li et al., 11 Sep 2025).

The 17 detectors are CNNSpot, F3Net, GramNet, DIRE, UnivFD, LNP, LGrad, AIDE, SSP, Fusing, Fredect, DNF, NPR, Freq-Net, SAFE, DRCT, and C2P-clip. The paper states that these include methods from recent top venues, including SOTA approaches from KDD 2025, AAAI 2025, and ICLR 2025. Following prior work, the detectors are pretrained on GenImage-SD v1.4 and then fine-tuned on an RRDataset subset (Li et al., 11 Sep 2025).

The 10 VLMs are GPT-4o-latest, Claude-3.7-sonnet, Gemini-1.5-pro, Gemini-2-flash, GLM-4v-plus, Grok-2-vision, Qwen2.5-VL-72B, YI-vision, Moonshot-preview-vision-128k, and Hunyuan-vision. They are evaluated in zero-shot mode and are prompted to output a JSON prediction of either “AI-generated” or “Real” based on visual analysis (Li et al., 11 Sep 2025).

A notable methodological feature is that RRBench treats robustness as a multi-condition property rather than a single benchmark score on original images. This suggests an evaluation philosophy closer to robustness benchmarking in other subfields, where performance stability under realistic transformation is at least as important as clean-set accuracy.

5. Human benchmark and few-shot adaptation study

The human study includes 192 participants, randomly assigned to a special-scenario group or an everyday-scenario group. In Phase 1, each participant evaluates 120 images: 20 real and 20 AI images from each of the original, transmitted, and re-digitized conditions. For each image, participants answer whether the image is AI-generated or real, and, if AI-generated, why. They choose from 14 reasons grouped into 5 low-level criteria—texture, edge, clarity, distortion, overall hue—5 mid-level criteria—light and shadow, shape, content deficiency, symmetry, reflection—and 4 high-level criteria—layout, perspective, theme, irreality. They also rate confidence on a 5-point Likert scale (Li et al., 11 Sep 2025).

In the few-shot learning phase, participants are shown 2 additional images from RRDataset, without overlap with the main test set. In Phase 2, they repeat the same 120-image test to measure whether the 2-shot exposure improves accuracy. A final comprehensive inquiry asks what they assume when unsure—AI-generated or real—and which areas of the image they focus on most. For the second question, participants are split into three subgroups of 32 and asked to mark regions in AI-generated images from the original, transmitted, and re-digitized conditions (Li et al., 11 Sep 2025).

The paper describes this as the largest human benchmark to date. It is also unusual in that it studies not only raw human accuracy, but human few-shot adaptation and cue selection under degradation. This design supports comparisons among specialized detectors, zero-shot VLMs, and human observers under the same transformations.

6. Empirical findings, interpretation, and significance

The benchmark reports that current detectors still struggle in real-world conditions. The strongest detector reaches 89.59% overall accuracy, and none of the 17 detectors achieves saturated performance. Under internet transmission, 14 of 17 detectors suffer reduced fake-image accuracy. The most severe reported declines are 71.61% for Freq-Net, 79.86% for Fusing, and 97.41% for SAFE. The paper identifies DNF, DIRE, and DRCT-ConvB as relatively stable under transmission, and states that these methods benefit from diffusion-based or reconstruction-based features that are less sensitive to social-media degradation (Li et al., 11 Sep 2025).

Re-digitization is reported as even more difficult. Sixteen of 17 detectors suffer fake-accuracy decreases under re-digitization, with 88.30% for DIRE and 90.57% for DNF as the most severe declines. AIDE is presented as a standout exception, with only 1.89% fake-accuracy reduction and a 4.19% increase in real accuracy. The paper suggests that AIDE’s robustness comes from its reliance on CLIP-extracted semantic and contextual information, which survives re-digitization better than low-level artifacts (Li et al., 11 Sep 2025).

The VLMs provide strong zero-shot baselines, but they also degrade under transmission and re-digitization. GPT-4o is reported to surpass 16 specialized detectors on the original data, yet its performance still drops substantially under transformed conditions. The human study shows a parallel pattern. Before few-shot exposure, the everyday-life group attains 69.17% overall accuracy and the special-scenario group 59.52%. For AI-generated images specifically, the everyday-life group attains 58.29%, whereas the special-scenario group attains 39.64%. After 2-shot learning, overall detection of AI-generated images rises from 42.07% to 55.86%, and real-image accuracy rises from 77.54% to 81.92%. Human confidence also declines under degradation: the proportion of “somewhat certain” and “very certain” judgments falls from 51.8% on original images to 38.3% on transmitted images and 22.7% on re-digitized images (Li et al., 11 Sep 2025).

The paper further reports a “trust crisis” effect. When unsure, 70.21% of the everyday-scenario group guess AI-generated, and 89.31% of the special-scenario group do the same. It also finds that human reasoning shifts with condition: transmission increases reliance on texture, edge, and clarity, while re-digitization increases reliance on light and shadow, reflection, and hue. On the modeling side, the authors propose a human-inspired in-context learning approach for VLMs. For GPT-4o, this improves re-digitization robustness by 5.50% and overall accuracy by 3.90%, reaching 87.47% average accuracy (Li et al., 11 Sep 2025).

Taken together, these findings position RRDataset as a benchmark for evaluating whether AI-generated image detectors remain reliable after online sharing, compression, re-photography, scanning, and other realistic degradation. The broader robustness literature suggests that this kind of shift-aware benchmarking is part of a larger movement across vision tasks: ImageNet-S targets deployment-induced preprocessing mismatch in classification (Wang et al., 2021), R-Bench targets corruption robustness in large multimodal models (Li et al., 2024), and RobustSpring targets corruption robustness in dense correspondence (Schmalfuss et al., 14 May 2025). RRDataset extends that movement to AI-generated image detection in high-stakes scenarios. A plausible implication is that future robust detectors will need to rely less on fragile low-level artifacts alone and more on signals that survive transmission and re-digitization.