- The paper introduces AEGIS, a comprehensive benchmark for forensic analysis of AI-generated academic images, integrating 7 categories, 39 subtypes, and four forgery strategies.
- It employs a robust data curation pipeline from over 4,300 PMC papers and 25 generative models to simulate realistic image manipulations with region-level annotations.
- Experimental results reveal critical performance gaps in both MLLMs and expert detectors, underscoring the need for combined sensitivity and semantic reasoning in forensic systems.
AEGIS: A Holistic Benchmark for Evaluating Forensic Analysis of AI-Generated Academic Images
Motivation and Contributions
AEGIS addresses the systematic gaps in the forensic assessment of AI-generated images in academic contexts. Prior benchmarks emphasize generic scenes, limited forgery types, and single-task detection protocols, thus failing to meet the requirements for high-fidelity scholarly review. AEGIS introduces a domain-specific, multi-dimensional, and adversarially diverse benchmark, encompassing seven academic image categories and 39 annotated subtypes, four principal forgery strategies (Text Constraint Fabrication, Image Inference Forgery, Targeted Region Restoration, Targeted Region Editing), and evaluation across four forensic dimensions: Forgery Scope Discrimination, Textual Artifact Recognition, Manipulation Classification, and Tampering Pinpointing. The dataset comprises over 8,000 highly curated panels and over 20,000 forensic questions, covering a wide range of generative models and manipulation scenarios.
Dataset Construction and Forgery Simulation
The data curation pipeline extracts figures and panels from 4,362 PMC papers, followed by expert annotation and strict filtering. The forgery simulation utilizes 25 advanced generative models spanning diffusion-based, hybrid, and unified multimodal architectures, driving realistic and controllable synthetic image manipulations. Region-level masks are generated using SAM for fine-grained spatial control. Rigorous quality assurance combines multi-stage expert review and automated metrics (IS, FID, CLIP Score), reaching strong alignment with authentic misconduct cases on realism, plausibility, and credibility.
Multi-Dimensional Evaluation Protocol
AEGIS evaluates forensic models across four tasks:
- Forgery Scope Discrimination: Multi-class authenticity judgment (Entire, Partial, or No Forgery).
- Textual Artifact Recognition: Binary detection of AI-generated or manipulated text.
- Manipulation Classification: Reasoning-based discrimination of edit types (Insertion, Removal, Alteration).
- Tampering Pinpointing: Fine-grained spatial localization (region-level for MLLMs, pixel-level for expert models).
Metrics include Accuracy, Macro-F1, Intersection over Union, Correct Localization Accuracy, Over Localization Rate, and a composite Normalized Forensic Index (NFI) to penalize task imbalance and over-localization.
Experimental Results and Analysis
AEGIS systematically benchmarked 25 MLLMs (including GPT-5.1, Gemini, Qwen, LLaVA variants, Janus-Pro-7B), nine expert detectors, and three hybrid models. Key findings are:
- Domain Complexity: Both MLLMs and expert baseline models underperform on dense scientific images (e.g., Stained Micrographs, Medical Imaging). GPT-5.1 achieves only 48.80% balanced NFI, with expert models exhibiting a 30% gap in real-vs-forgery detection F1 and localization plateauing at IoU 30.09%. Performance correlates with image visual density, revealing a structural bias toward geometric regularity and limited generalization.
- Adversarial Generative Diversity: Eleven generative models reduce forensic accuracy below 50%; four degrade it below 30%. Localized manipulations remain challenging—targeted edits consistently yield lower detection and localization rates compared to holistic forgeries.
- Complementary Model Competence: MLLMs excel at semantic artifact recognition (TAR at 84.74% accuracy), manipulation classification (up to 60.07%), but lack spatial precision. Expert detectors reach 79.54% in binary authenticity classification but are fragile under image perturbations. Robust academic forensics requires integration of sensitive detection (experts) with cross-modal reasoning (MLLMs).
- Prompting Strategies: Few-shot prompts trade off improved pattern-level detection for degraded fine-grained reasoning; Chain-of-Thought incrementally boosts detection, localization, but impairs manipulation classification. Both strategies indicate inherent task-specific weaknesses in current architectures.
- Error Patterns: MLLMs often overgeneralize localized edits, misinterpret occlusion as content removal, and default to axis-aligned bounding boxes, missing irregular anatomical contours. Expert detectors display a real-forgery prediction bias due to distributional coverage gaps.
- Perturbation Robustness: Vision-only expert models degrade sharply under blurring, compression, and scaling; MLLMs remain relatively stable, illustrating lower reliance on pixel-level features and greater robustness.
Theoretical and Practical Implications
AEGIS exposes fundamental limitations in current forensic systems:
- Authenticity Modeling: Current architectures fail to model knowledge-intensive, structurally complex distributions characteristic of academic imagery, perpetuating biased predictions and insufficient spatial precision.
- Adversarial Dynamics: The pace of generative model advances outstrips forensic adaptation—localized edits and multimodal generation challenge static forensic cues.
- Complementarity Requirement: Synergizing expert detectors (sensor-level sensitivity) with MLLMs (semantic reasoning) is essential for robust expert AGI applicable to high-stakes scholarly and regulated domains.
- Task-Specific Generalization: Generalized forensic protocols are insufficient—domain-specific training, context-aware evaluation, and region-level annotations are critical for trustworthy deployment.
- Cross-Domain Applicability: Challenges encountered mirror those in legal evidence, financial document forensics, and structured visual inspection, suggesting broader implications beyond academic publishing.
Speculation on Future Developments
The trajectory of forensic analysis will likely involve adaptive hybrid systems leveraging hierarchical fusion of vision-level cues with textual/semantic reasoning, model customization for dense, structured domains, and scalable integration of real-world annotation protocols. Improvements in fine-grained spatial grounding, manipulation provenance tracking, and adversarial resilience are anticipated. Expansion of benchmarks with real-world misconduct cases, as generative misuse detection intensifies, will further drive sophistication in forensic interpretability and generalization.
Conclusion
AEGIS establishes the first rigorous, domain-specific, and multi-perspective benchmark for AI-generated academic image forensics (2604.28177). It reveals persistent model capability gaps rooted in dense scientific visual content, adversarial generative diversity, and forensic-task orthogonality. Future progress in multimodal forensics requires an orchestrated blend of sensitive detection and deep reasoning, tailored for heterogeneous, knowledge-intensive environments. AEGIS offers a robust foundation and diagnostic framework for research on trustworthy, generalizable multimodal image forensics, with downstream applications across scientific, legal, and regulated domains.
