SciFigDetect: AI-Generated Figures Benchmark

• SciFigDetect is a benchmark that defines scientific figures as structured, text-dense images requiring specialized detection techniques beyond generic synthetic image analysis.
• It employs a multi-agent pipeline with GPT-based controllers to generate and align real–synthetic figure pairs from licensed scientific papers.
• Extensive experiments show that traditional detectors struggle with domain-specific cues, revealing generator-specific overfitting and robustness challenges.

SciFigDetect is a benchmark for detecting AI-generated scientific figures rather than generic synthetic images such as faces or natural scenes. It was introduced to address a setting in which modern multimodal generators, including Nano Banana Pro and GPT-image-1.5, can produce scientific figures at “near-publishable” quality and in close alignment with research narratives, thereby creating new integrity risks for scholarly communication. The benchmark combines a legally compliant, agent-based construction pipeline with aligned real–synthetic pairs spanning multiple figure types and multiple generators, and it is explicitly designed to expose failure modes of existing AI-generated image detection methods under realistic scientific settings (Hu et al., 9 Apr 2026).

1. Problem setting and formal definition

SciFigDetect is motivated by the claim that scientific figures are a distinct detection target. In the benchmark description, scientific figures are characterized by structured layouts, text-dense composition, symbol-heavy and convention-driven presentation, and tight semantic alignment with surrounding scholarly text. These properties distinguish them from the open-domain imagery on which most existing AIGI detectors are developed. The benchmark therefore targets a distribution in which the cues emphasized by conventional fake-image detectors—texture artifacts, spectral anomalies, up-sampling traces, or stylistic cues from general text-to-image content—are weaker, different, or intentionally suppressed.

The benchmark formalizes construction from a candidate paper pool

$\mathcal{P} = \{p_i\}_{i=1}^{N},$

where each paper $p_i$ has a PDF, metadata, and license. Only papers with commercially permissible licenses are retained:

$$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$

where $\ell_i$ is the license of paper $p_i$ and $\mathcal{L}_{\mathrm{perm}}$ includes licenses like CC BY. For each retained paper $p \in \mathcal{P}^{+}$, a benchmark sample is defined as

$$z = \big(c,\; f_{\mathrm{real}},\; f_{\mathrm{syn}},\; a\big),$$

where $c$ is figure-related paper context, $f_{\mathrm{real}}$ is the original figure, $p_i$0 is an accepted synthetic figure derived from the same content, and $p_i$1 is auxiliary metadata. The full benchmark is

$p_i$2

The principal detection tasks defined on top of this dataset are binary classification, real versus synthetic, and cross-generator generalization, training on one generator and testing on another. A common misconception is that “figure detection” here is simply a variant of generic image forensics. The benchmark is designed around the opposite premise: strong performance on standard benchmarks such as GenImage, CIFAKE, and DeepArt does not translate to reliable detection on scientific figures.

2. Agent-based construction pipeline

SciFigDetect is built through a multi-agent pipeline with a master–worker architecture. A GPT-based Controller Agent orchestrates the workflow, while the worker components are a Chunking Agent, Text Agent, Figure Agent, Prompt Builder, Figure Generation module, and Review Agent. The process has two stages: Understanding & Prompt Planning, and Generation–Review Refinement Loop.

In the first stage, the Chunking Agent segments papers according to section boundaries, paragraph continuity, and figure reference positions such as “see Fig. 3,” with the goal of associating each figure to the text that actually describes its content. The Text Agent then extracts figure-relevant semantics from those chunks, including research background and context, key entities, workflows, and structural relations such as input $p_i$3 process $p_i$4 output. In parallel, the Figure Agent analyzes the original scientific figure in terms of layout and composition, module organization, arrows and edges, legends, color schemes, and spatial hierarchy, and assigns a coarse figure type label: Illustration, Overview, or Experimental Figure.

The Prompt Builder merges signals from the Text Agent and Figure Agent into a structured prompt with separated components for content, style, and structure. The benchmark description states that these prompts are “structure-aware” and are intended to preserve “core semantics, logical organization, and plausible academic style” rather than produce arbitrary text-to-image outputs. This matters because the aim is not pixel-wise reconstruction of the original figure, but generation of new figures that encode the same scientific concept, use scientifically plausible layout, and look like publication-ready figures.

In the second stage, the Figure Generation module calls two modern generators: Nano Banana Pro and GPT-image-1.5. Candidate figures are then passed to the Review Agent, which scores each generated candidate $p_i$5 on academic fidelity $p_i$6, aesthetic consistency $p_i$7, and logical coherence $p_i$8. These are aggregated into a single review score:

$p_i$9

The implementation sets

$$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$0

A candidate is accepted if $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$1; otherwise the Controller revises the prompt and/or regenerates, forming a closed-loop refinement process inspired by self-refinement frameworks.

This construction procedure also clarifies the meaning of alignment in SciFigDetect. Real–synthetic pairs are aligned at the content and structure level, not at the pixel level. A synthetic figure is accepted when it conveys the same scientific message and approximates the same layout category as the corresponding real figure.

3. Dataset composition, labels, and provenance

SciFigDetect contains 72,965 real figures and 150,807 synthetic figures. The synthetic portion comes from two generator sources, Nano Banana Pro and GPT-image-1.5, and the benchmark is organized around three figure categories: Illustration, Overview, and Experimental Figure (Hu et al., 9 Apr 2026).

The category-level counts are specified as follows. Among real figures, Illustrations number 5,773, Overviews 8,882, and Experimental Figures 58,310. Among Nano Banana Pro synthetic figures, Illustrations number 4,616, Overviews 6,608, and Experimental Figures 39,155. Among GPT-image-1.5 synthetic figures, Illustrations number 9,090, Overviews 13,164, and Experimental Figures 78,174.

A key subset consists of aligned real–synthetic pairs in which a real figure has synthetic counterparts from both generators. The benchmark reports 4,616 Illustration pairs, 6,608 Overview pairs, and 39,155 Experimental Figure pairs. These pairs support controlled evaluations in which the same textual and scientific context is associated with one real figure and two synthetic variants of the same concept.

The topic distribution is broad rather than limited to a single niche. The benchmark reports 7,079 diagrams in Generative & Learning (39.8%), 6,478 in Science & Application (36.4%), 2,459 in Vision & Perception (13.8%), and 1,769 in Agent & Reasoning (9.9%). At the same time, the stated limitations note that all topics are within computer-science and AI-centric domains, so discipline coverage should not be interpreted as uniformly cross-scientific.

Each sample includes labels for real versus synthetic, generator type, and figure category; context in the form of figure-related paper text and source paper identifiers; and provenance fields including license information, review scores and review history, and prompt content and generation settings. The licensing constraint is integral rather than incidental: only papers whose license belongs to $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$2, commercially permissible licenses such as CC-BY, are included. The benchmark description presents this as a legal safeguard for reuse of figures and associated text in dataset construction.

4. Tasks, evaluation protocols, and shortcut control

The base task supported by SciFigDetect is binary classification of a scientific figure image as real or synthetic. Experiments are reported overall, per figure type, and per generator. To avoid information leakage, splitting is performed at the paper level, so that all figures from a given paper, across categories and generators, are assigned to the same split. The benchmark uses 10-fold cross-validation with training:validation:test = 8:1:1 in each fold, and reported results are averaged over the 10 folds.

The evaluation metrics follow prior AIGI work. Accuracy uses threshold 0.5 on detector logits or scores, and Average Precision is computed over real-versus-fake classification scores. Some tables additionally report $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$3, $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$4, generator-specific accuracies such as $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$5 and $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$6, and an unweighted mean

$$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$7

Three experimental settings are emphasized. In the zero-shot setting, detectors trained on existing open-domain AIGI datasets are applied directly to SciFigDetect with no finetuning. In the cross-generator setting, models are trained on Banana only, GPT only, or Banana+GPT jointly, and then tested on real figures plus both generator outputs. In the degraded-image robustness setting, detectors trained on clean Banana+GPT data are tested under JPEG compression with quality factors $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$8, WebP compression with $$\mathcal{P}^{+} = \{p_i \in \mathcal{P} \mid \ell_i \in \mathcal{L}_{\mathrm{perm}}\},$$9, Gaussian blur with $\ell_i$0, and Gaussian noise with $\ell_i$1.

A notable methodological feature is preprocessing designed to reduce shortcuts. All images are converted to PNG, and synthetic figures are resolution-aligned to match their real counterparts. For GPT-generated images, blank margins are removed so that only the central content region is retained. For diagram-like figures, the pipeline applies color quantization and “color snapping”: colors are clustered and compressed, near-white and near-black pixels are snapped to canonical colors, dominant clusters are merged, and uniform color blocks and boundaries are refined. The stated purpose is to make it harder for detectors to exploit trivial cues such as compression patterns or color-space differences between generators and real images.

5. Baselines and empirical findings

SciFigDetect benchmarks representative AIGI detection paradigms, including spatial-domain CNNs and patch-based methods such as CNNSpot and PatchFor; gradient- or structure-based methods such as LGrad and NPR; frequency-domain models such as FreqNet; and foundation-model or transformer-based detectors such as UniFD, FatFormer, AIDE, and Effort. The benchmark uses these models to probe transfer, generator specificity, and robustness rather than to claim that any one detector solves the task (Hu et al., 9 Apr 2026).

The zero-shot results are the central empirical finding. LGrad is reported as the best overall zero-shot model, with AvgAcc (All Figures) of 53.68% and AP of 65.30. Most other detectors are near chance in terms of real-versus-fake discrimination. The benchmark highlights a recurring pattern of very high $\ell_i$2 and extremely low $\ell_i$3. CNNSpot has $\ell_i$4 and $\ell_i$5 on all figures; PatchFor has $\ell_i$6 and $\ell_i$7; FreqNet has $\ell_i$8 and $\ell_i$9. Effort reaches 100% $p_i$0 on Illustrations, Overviews, and Experimental Figures, but 0% $p_i$1 on Illustrations and Overviews in zero-shot. The benchmark interprets this as evidence that off-the-shelf detectors almost always predict “real” on scientific figures.

Cross-generator experiments show strong generator-specific overfitting. When trained on Banana only, Effort reaches $p_i$2 but drops to $p_i$3, while UniFD obtains $p_i$4, the best AvgAcc in that training column. NPR is stronger in transfer under Banana-only training, with $p_i$5, $p_i$6, $p_i$7, and $p_i$8. When trained on GPT only, Effort reaches $p_i$9 but only $\mathcal{L}_{\mathrm{perm}}$0, and NPR reaches $\mathcal{L}_{\mathrm{perm}}$1 but only $\mathcal{L}_{\mathrm{perm}}$2. Averaged across methods, training on Banana yields 83.3% accuracy on Banana and 48.7% on GPT, while training on GPT yields 87.5% on GPT and 26.1% on Banana.

Joint training on Banana+GPT improves performance but does not remove generator skew. Effort achieves clean AvgAcc = 95.58%, with $\mathcal{L}_{\mathrm{perm}}$3, $\mathcal{L}_{\mathrm{perm}}$4, and $\mathcal{L}_{\mathrm{perm}}$5. NPR reaches AvgAcc = 93.96%, and LGrad reaches AvgAcc = 92.01%. UniFD remains lower at AvgAcc = 75.40%. PatchFor is an explicit failure case even under joint training: $\mathcal{L}_{\mathrm{perm}}$6, $\mathcal{L}_{\mathrm{perm}}$7, but $\mathcal{L}_{\mathrm{perm}}$8, for AvgAcc = 68.30%.

Robustness under degradation is limited. With clean-data training on Banana+GPT, the baseline clean accuracies are 82.50% for CNNSpot, 75.40% for UniFD, 93.96% for NPR, and 95.58% for Effort. Under JPEG compression at $\mathcal{L}_{\mathrm{perm}}$9, NPR drops to 75.94%, CNNSpot to 75.74%, UniFD to around 61.27%, and Effort to about 71.02%. Under WebP compression at $p \in \mathcal{P}^{+}$0, NPR reaches 66.86%, CNNSpot 72.78%, UniFD 71.33%, and Effort 68.25%. Under Gaussian blur with $p \in \mathcal{P}^{+}$1, CNNSpot remains at 83.45%, NPR falls to 79.13%, Effort to 67.40%, and UniFD to 66.20%. Under Gaussian noise with $p \in \mathcal{P}^{+}$2, NPR drops to 50.49%, CNNSpot to 56.12%, UniFD to 53.00%, and Effort to 66.71%. The benchmark therefore presents degradation sensitivity as a practical weakness even for models that perform well on clean data.

6. Interpretation, limitations, and relation to adjacent detection research

The benchmark’s analysis attributes zero-shot failure primarily to domain shift, artifact mismatch, and a strong prior toward natural-image biases. Scientific figures are structured, text-dense, and stylized differently from the faces, natural scenes, and general text-to-image content used in conventional detector training. This suggests that detectors tuned to up-sampling artifacts, spectral distributions, or subtle textures are poorly matched to vector-like graphics, flat regions, sharp lines, and discrete symbols. The analysis also argues that current systems learn fingerprints of specific generators rather than fundamental differences between synthetic and real figures.

A second implication concerns the role of structure and text. The benchmark explicitly notes that the evaluated detectors do not exploit OCR, figure-text alignment, or multimodal reasoning, even though real-versus-synthetic discrimination in this setting may require checking logical consistency of arrows and labels, internal coherence across panels, or agreement between a figure and the associated paper context $p \in \mathcal{P}^{+}$3. This is one reason the benchmark is positioned as a foundation for future work on “structure, text, and semantics” rather than as a closed benchmark in which artifact-based vision models are sufficient.

The limitations are also clearly delimited. Topic coverage is diverse but remains within computer-science and AI-centric domains. Figure type coverage is restricted to Illustrations, Overviews, and Experimental Figures; other figure types such as detailed charts, tables, histograms, and microscopy photos are not explicitly listed. Generator coverage is limited to Nano Banana Pro and GPT-image-1.5. The benchmark is described as smaller than giant general-image benchmarks like GenImage, and the automated multi-agent pipeline may introduce selection and prompting biases.

In adjacent scholarly-integrity research, the paper “Span-level Detection of AI-generated Scientific Text via Contrastive Learning and Structural Calibration” (Yin et al., 1 Oct 2025) develops a structure-aware, span-level detector for scientific text using section-conditioned style modeling, graph-based structural encoding, boundary confidence modeling, and temperature scaling. A plausible implication is that future figure-forensics systems could borrow analogous ideas—localizing suspicious regions or caption segments, exploiting section-aware priors on figure function, and using calibrated multimodal confidence—rather than limiting detection to whole-image binary labels. That implication remains hypothetical, but it is consistent with SciFigDetect’s own conclusion that robust and generalizable scientific-figure forensics will likely require methods that understand structure, text, and semantics, not only low-level visual artifacts.