GenBuster++: Cross-Modal AI Detection Benchmark
- GenBuster++ is a cross-modal benchmark combining 2,000 images and 2,000 video clips to evaluate AI-generated content detection under a unified interface.
- It uses a two-stage expert filtering and standardized data processing to produce high-quality real and synthetic samples across both modalities.
- Empirical results show that cross-modal training with BusterX++ significantly improves accuracy, highlighting gaps between real and fake video detection.
Searching arXiv for the specified papers and closely related work to ground the article. GenBuster++ is a cross-modal benchmark for detecting AI-generated content, introduced as the companion benchmark to BusterX++ and designed to evaluate multimodal LLMs on images and videos within a unified setting. It contains 4,000 total samples—2,000 images and 2,000 video clips—with exact 1:1 real/fake balance in each modality, and it is intended for one model that handles both modalities under the same interface and training regime rather than as separate image-only and video-only detectors (Wen et al., 19 Jul 2025).
1. Rationale and conceptual scope
GenBuster++ was proposed in response to two limitations in prior evaluation practice. First, existing benchmarks are almost all single-modality: image-only datasets such as GenImage, DMImage, SID-Set, and So-Fake-Set, and video-only datasets such as GenBuster-200K, GenVideo, GenVidBench, and DeCoF. Second, many of these benchmarks rely heavily on automatic generation pipelines with little human curation, which can produce unrealistic or low-quality fakes and can make evaluation less reflective of hard real-world forgeries (Wen et al., 19 Jul 2025).
Within this benchmark, cross-modal has a specific meaning. It simultaneously covers two visual modalities—images and videos—and evaluates whether a single MLLM generalizes across modalities under one interface and one training regime. It does not denote image-text-video or audio-video joint reasoning. The benchmark therefore targets image↔video transfer rather than a broader multimodal fusion problem.
The benchmark also marks a shift from the video-only design of GenBuster-200K, which was introduced with BusterX for AI-generated video forgery detection and explanation. A plausible implication is that GenBuster++ extends the earlier emphasis on high-quality, generator-diverse, standardized evaluation into a unified image-and-video setting while preserving the forensic orientation toward current generative models (Wen et al., 19 May 2025).
2. Dataset composition, sourcing, and curation
GenBuster++ is explicitly constructed in two parts—real content and synthetic content—for both images and videos. The benchmark is intentionally small but curated as a high-quality set of challenging real versus synthetic samples. The exact class balance is uniform across the dataset.
| Modality | Real | Fake |
|---|---|---|
| Images | 1,000 | 1,000 |
| Videos | 1,000 | 1,000 |
The primary real-data source is OpenVid-1M HD. Pre-filtering uses resolution, frame rate, and bitrate thresholds to remove low-quality content, together with deduplication to avoid multiple clips from the same origin. Manual filtering removes heavy watermarks, anime or obviously stylized content, and content with obviously synthetic backgrounds, even if it was labeled real in the original dataset. The final outcome is 1,000 real images and 1,000 real videos (Wen et al., 19 Jul 2025).
The fake-data pipeline is more elaborate. It begins with an initial synthetic pool using MagicArena high-rated samples for both images and videos. The construction process also collects real social-media images via Reddit API to cover a wide range of real-world scenarios, generates textual captions for these images using Qwen-2.5-VL, and then feeds those captions to multiple state-of-the-art generators. The image side uses 7 generators, explicitly including FLUX.1 Kontext and GPT-4o’s image generation. The video side uses 13 generators, explicitly including Seedance 1.0 and SkyReels V1, together with other commercial and open-source generators listed in the paper’s figure caption and references (Wen et al., 19 Jul 2025).
A defining feature of GenBuster++ is its two-stage expert filtering methodology. In Stage 1: Real-likeness screening, experts inspect a mixed pool containing both real and fake samples and select the samples that look real. In Stage 2: Synthetic isolation, experts re-examine that subset and separate the samples that are actually synthetic. Only the synthetic samples that were previously judged as real are retained as fake data. This yields 1,000 fake images and 1,000 fake videos that are visually high-quality and sufficiently confusing that experts initially believed they could be real (Wen et al., 19 Jul 2025).
Post-processing standardizes media to suppress trivial low-level shortcuts. Images are resized or cropped to and stored with standardized encoding. Videos are standardized to , 5 seconds, 24 FPS, and HEVC (x265). The stated benefits are to eliminate encoding or codec artifacts as a trivial discriminative signal, normalize resolution and frame rate across generators, and ensure consistent length for video-level detectors (Wen et al., 19 Jul 2025).
The benchmark covers 9 different categories of real-world scenarios, although the exact label set is not enumerated in the text. Visual examples show landscapes and landmarks, architecture, food and kitchen activities, vehicles, animals, and political signage and street scenes. The dataset remains in the photorealistic domain by manually excluding anime and stylized material (Wen et al., 19 Jul 2025).
3. Labels, task definition, and evaluation protocol
At its core, GenBuster++ is a binary real-versus-fake benchmark. Each sample is labeled
There is no explicit labeling of generator identity, manipulation type, bounding boxes, pixel-level masks, or temporal segments with per-frame labels. Annotation therefore operates at the example level and is centered on authenticity and high-quality visual realism rather than localization or attribution (Wen et al., 19 Jul 2025).
For use with BusterX++, each sample is paired with a task prompt, phrased in the appendix as “Is this image/video real or fake?”. In the BusterX++ evaluation setting, this is instantiated as a multiple-choice question with options A) real and B) fake, and the model is expected to choose one. Although this prompt is part of the evaluation protocol rather than dataset annotation proper, it is integral to how the benchmark is used (Wen et al., 19 Jul 2025).
GenBuster++ is treated as a pure evaluation benchmark rather than as a training dataset for BusterX++. Training uses other datasets, including So-Fake-Set and GenBuster-200K, and GenBuster++ is reserved for testing cross-modal generalization. The paper does not define an internal train/validation/test split within GenBuster++; the entire 4,000-sample set is used for evaluation (Wen et al., 19 Jul 2025).
The principal metric is accuracy:
Results are reported by modality and class: Image Real, Image Fake, Video Real, Video Fake, and an Overall score that is implicitly weighted by equal counts across the four categories. The paper does not use AUROC, F1, ECE, or explanation metrics for GenBuster++ (Wen et al., 19 Jul 2025).
4. Cross-modal evaluation and integration with BusterX++
GenBuster++ is the central evaluation platform for BusterX++’s unified image-and-video training regime. BusterX++ begins from Qwen2.5-VL-7B-Instruct and uses purely RL-based post-training with no SFT cold start. The RL algorithm is DAPO, and training is performed on mixed image+video data. The same backbone and the same policy parameters are used for both modalities, so the benchmark probes whether forgery cues learned from one modality transfer to the other (Wen et al., 19 Jul 2025).
The paper’s ablation on Data Modality is the clearest expression of the benchmark’s cross-modal role:
| Training data | Key accuracies | Overall ACC |
|---|---|---|
| Images only | Image 79.0 / 72.3; Video 81.2 / 52.9 | 71.4 |
| Videos only | Image 78.7 / 65.4; Video 92.4 / 55.0 | 72.9 |
| Images + videos | Image 80.4 / 76.2; Video 95.3 / 57.9 | 77.5 |
These results support the paper’s central cross-modal argument: training on both modalities improves performance on each modality. On images, fake accuracy rises to 76.2 from 72.3 or 65.4. On videos, both real and fake accuracy improve under cross-modal training. The benchmark is therefore designed not merely to compare image and video detectors side by side, but to test whether a unified detector benefits from modality transfer (Wen et al., 19 Jul 2025).
GenBuster++ is also used to evaluate BusterX++’s RL design. In the paper’s comparison of cold-start and non-cold-start pipelines, Cold-start reaches 71.7% overall ACC with Stage-1 only and 72.9% with Stage-1 + Stage-3, whereas Non-cold-start reaches 69.4% with Stage-1 only and 77.4% with Stage-1 + Stage-3. This indicates that the benchmark functions as a stress test for the claim that cold-start SFT can limit RL improvements for AI-generated content detection (Wen et al., 19 Jul 2025).
5. Empirical performance and explanation-oriented use
On GenBuster++, the benchmark separates strong generic MLLMs from BusterX++ by a large margin. Reported overall accuracies are 58.4% for MiMo-VL-7B-RL, 55.5% for InternVL3-8B, 51.9% for Keye-VL-8B, 53.3% for MiniCPM-o 2.6, 55.0% for Qwen2.5-Omni-7B, 55.4% for Qwen2.5-VL-7B, and 68.3% for the prior video-only detector BusterX. By contrast, BusterX++ (thinking mode) reaches Image Real 80.4%, Image Fake 76.2%, Video Real 95.3%, Video Fake 57.9%, and Overall 77.5%; BusterX++ (/no_think) reaches 76.8% overall (Wen et al., 19 Jul 2025).
A notable pattern is the asymmetry between real and fake video performance. Video reals reach 95.3% while video fakes reach 57.9%, which indicates that the human-filtered fake videos are substantially harder. Image fakes, at 76.2%, are also challenging but less resistant than video fakes. Because the synthetic subset is composed of samples that initially passed expert real-likeness screening, this distribution of errors is consistent with the dataset’s design objective of emphasizing hard cases (Wen et al., 19 Jul 2025).
Although GenBuster++ itself carries only binary labels, it is used to assess explanation behavior in combination with BusterX++. In thinking mode, the system prompt enforces the format > ... <answer> A/B </answer>. The generated reasoning may include low-level visual cues such as lighting, shadows, edges, and compression artifacts; high-level semantic consistency; and self-reflective inspection. The paper further introduces a thinking reward model, SophiaVL-R1-Thinking-Reward-Model-3B, so that reasoning is rewarded only when the answer is correct and the explanation quality is high (Wen et al., 19 Jul 2025).
Qualitative examples on GenBuster++ illustrate the intended explanatory use. A fake lighthouse boardwalk image is analyzed through over-perfect perspective and symmetry, too-uniform textures, unrealistically clean edges and lighting, and highly stylized clouds and sky. A real portrait is judged from realistic skin texture, hair, clothing, accessories, and consistent background and lighting. A fake cat-on-roof video is flagged through stylized blossoms and mountains, too-perfect textures and symmetry, and rendered-looking fur and pose. A real cooking video is recognized through natural hand and knife motion, realistic textures of tomatoes and cutting board, and consistent indoor lighting and shadows. No quantitative explanation metric or human rating study is reported (Wen et al., 19 Jul 2025).
6. Limitations, biases, and research trajectory
The benchmark is explicitly small-scale by design: 4,000 samples prioritizing quality over quantity. This makes intensive expert curation feasible, but it also limits coverage. The paper notes that GenBuster++ may not fully cover all cultural or geographical domains, many languages in text overlays, audio, speech, or multi-turn conversational settings. The dataset focuses on photorealistic imagery and intentionally excludes stylized or anime content (Wen et al., 19 Jul 2025).
The paper also identifies generator adaptation as an ongoing problem. GenBuster++ uses primarily latest-generation image and video models, and the authors note that these samples are harder than those in GenBuster-200K. As generative systems evolve, continuous updates are required for the benchmark to remain representative. A plausible implication is that GenBuster++ should be understood as a moving target benchmark for current-generation forensics rather than as a static endpoint (Wen et al., 19 Jul 2025).
Methodologically, the benchmark is also used to expose a possible post-training bottleneck. Even with RL, thinking reward, and hybrid reasoning, performance on the hardest fake videos remains limited. The paper therefore points toward future work involving pre-training modifications, dedicated architecture changes, and additional modalities such as audio and 3D. The broader predecessor literature around BusterX had already framed audio-visual forensics, longer videos, more manipulation types, and explicit explanation evaluation as natural extensions; GenBuster++ can be read as a concrete step toward that broader agenda, but not its completion (Wen et al., 19 May 2025).
The project page is announced as https://github.com/l8cv/BusterX. The paper does not specify license details and indicates that such details are presumably clarified in the repository. For practical research use, the benchmark is positioned as a final evaluation platform for zero-shot cross-modal authenticity classification, with per-category ACC on Image Real, Image Fake, Video Real, Video Fake, and Overall as the primary reporting format (Wen et al., 19 Jul 2025).