---
title: Human-AIGI Benchmark
url: https://www.emergentmind.com/topics/human-aigi-benchmark
type: topic
---

# Human-AIGI Benchmark

Human-AIGI Benchmark denotes a human-centered benchmarking paradigm for assessing artificial systems against human perceptual, cognitive, and collaborative performance, and, in a narrower usage, the psychophysically curated Human-AIGI dataset introduced for AI-generated image detection in MIRROR [2602.02222]. Across recent work, the unifying principle is that model capability is not judged only by aggregate task accuracy, but by calibration to human perceptual limits, human success rates, human error patterns, human uncertainty, or human interaction structure. This orientation appears in image forensics, multimodal reasoning, common-ground formation, general-assistant evaluation, and human-versus-AI dialogue discrimination [2505.11141][2602.21337][2311.12983].

## 1. Conceptual scope and benchmark landscape

In current arXiv usage, the term spans several related benchmark forms. One form is **perceptual benchmarking**, where the objective is to determine whether detectors exceed human observers on hard AI-generated images. A second is **interaction benchmarking**, where the target is common ground, grounding, repair, and role-flexible collaboration between humans and AI systems. A third is **human-referenced cognitive benchmarking**, where task difficulty or success criteria are anchored to human populations, as in standardized exams, multimodal reasoning sets, or general-assistant tasks [2602.02222][2602.21337][2309.06495].

| Benchmark | Domain | Human reference signal |
|---|---|---|
| Human-AIGI | AIGI detection | Accuracy, confidence, response latency |
| A-Bench | AIGI evaluation by LMMs | Expert-authored QA, human participant accuracy |
| Common Ground benchmark | Human-AI collaboration | User study, task success, grounding acts |
| Inverse Turing Bench | Dialogue discrimination | Human judge baseline |
| Human-Aligned Bench | Multimodal reasoning | Human correct rate, error-prone options |
| AGIBench / AGIEval / GAIA | General intelligence or assistant tasks | Human-referenced difficulty or direct human performance |

This landscape is methodologically heterogeneous but conceptually coherent. Human-AIGI benchmarks typically replace leaderboard-only evaluation with one or more of the following: direct human comparison, psychophysical curation, human-referenced difficulty tiers, human preference data, or process-level interaction analysis. This suggests that the defining attribute is not a single modality or task family, but explicit coupling between machine evaluation and measured human behavior [2502.20502][2304.06364].

## 2. Human-AIGI in AI-generated image detection

The most specific and technically developed instance is the Human-AIGI benchmark introduced by MIRROR, whose stated purpose is to answer when AIGI detectors actually surpass human observers, including visual experts, on images that are hard for humans [2602.02222]. Existing AIGI benchmarks such as CNNSpot, GenImage, AIGCDetect, DRCT-2M, UniversalFakeDetect, Synthbuster, WildRF, and AIGIBench emphasize generator coverage, semantic diversity, or robustness to distortions, but do not directly measure the point at which detectors outperform humans on the most deceptive images. MIRROR therefore frames Human-AIGI around a **“superhuman crossover”**: the point where a detector exceeds the human perceptual limit on near-indistinguishable real versus AI-generated images.

Human-AIGI is built from MSCOCO real images and AI-generated images from **27 generators** spanning diffusion, autoregressive, unified, and closed-source commercial systems, including SD-3-Medium, SD-3.5-Large, FLUX.1-dev, PixArt-α, Qwen-Image, Infinity-8B, BAGEL, Janus-Pro-7B, OmniGen2-7B, Seedream 3.0, Gemini 2.0 Flash, Imagen 4, HunyuanImage-3.0, and GPT-Image [2602.02222]. Its defining element is the **Human-Imperceptible subset** \(D_{\text{hard}}\), created through psychophysical experiments with **50 subjects**, comprising lay users and visual experts. For each image, the benchmark records binary real-versus-fake decisions, confidence \(S(x)\), and response latency \(RT(x)\). Hard examples are selected as “perceptual corner cases” using the rule
\[
D_{\text{hard}} = \{x \in D : S(x) \ge \tau_{\text{real}} \;\lor\; RT(x) > \mu + \sigma_{\text{rt}} \},
\]
so that selected items either deceive humans into a “Real” judgment with high confidence or induce significant hesitation [2602.02222].

Evaluation uses the standard binary detection task on both the original test partition and \(D_{\text{hard}}\). Human baselines are reported as **76.9%** accuracy for lay users and **88.3%** for visual experts. MIRROR reaches **89.6%** accuracy across 27 generators on Human-AIGI and **89.5%** on the Human-Imperceptible subset, thereby surpassing lay users and slightly exceeding visual experts under this operational definition of superhuman performance. On the hard subset, MIRROR outperforms the next-best baseline, DDA at **86.9%**, by **2.6 percentage points** [2602.02222].

Methodologically, MIRROR links the benchmark to a theory of human perception as reference–comparison on a **human cognitive manifold** of real-world regularities. The detector learns a real-image manifold from **200k real images from MSCOCO**, stores it in a discrete memory bank with \(K = 4096\) orthogonal prototypes, projects test images into a manifold-consistent reference, and uses residuals and reconstruct perplexity as detection signals. A plausible implication is that Human-AIGI is designed not merely to compare accuracies, but to test whether explicit modeling of reality priors yields superhuman performance precisely where human perception fails [2602.02222].

## 3. Human-centered evaluation of AIGI judges and explanation systems

The image-forensics strand extends beyond binary detection. A-Bench asks whether large multimodal models are reliable evaluators of AI-generated images, rather than generators or detectors in the usual sense [2406.03070]. It contains **2,864 AIGIs from 16 text-to-image models**, split into **1,408** high-level semantic items and **1,456** low-level quality items. Annotation was performed by **15 experts** with professional experience in photography and AIGI evaluation, with each QA pair reviewed by at least three other experts. Human participants achieve **92.39%–94.00%** on the semantic subset and **90.53%–92.25%** on the quality subset, whereas the best proprietary LMM, Gemini 1.5 Pro, reaches **84.69%** and **69.06%** respectively. The benchmark therefore shows that even the best LMM remains substantially below human performance, especially on generative distortions and quality perception [2406.03070].

AIGI-Holmes defines a related but distinct Human-AIGI regime centered on **human-verifiable explanations** and **human-aligned detection** [2507.02664]. Holmes-Set combines **Holmes-SFTSet** and **Holmes-DPOSet**, covering **18 generators** and **65K+4K** instances with explanations and preference data. Its benchmark logic is not limited to a single leaderboard: it integrates explanation-grounded supervision, preference optimization, and evaluation by automatic metrics, MLLM-as-a-judge scores, and human ELO ratings. On Protocol-III, which tests unseen generators such as Janus, VAR, Infinity, Show-o, PixArt-XL, SD3.5-Large, and FLUX, AIGI-Holmes reaches **99.2%** mean accuracy and **99.9%** mean average precision. On the explanation benchmark, AIGI-Holmes (DPO) reports **BLEU-1 = 0.622**, **ROUGE-L = 0.375**, **METEOR = 0.311**, **CIDEr = 0.107**, and **ELO = 11.420**, outperforming its SFT-only version and raw MLLM baselines [2507.02664]. This broadens the Human-AIGI notion from “can a model detect what humans miss?” to “can it justify its decision in a form humans can verify and prefer?”

The baseline problem has also shifted. “Simplicity Prevails” shows that a single linear classifier on frozen features from modern Vision Foundation Models can now outperform many specialized AIGI detectors, especially on in-the-wild distributions [2602.01738]. DINOv3-Linear reaches **0.964** average accuracy on GenImage and **0.940** average across Chameleon, WildRF, SocialRF, and CommunityAI, compared with **0.850** for DDA and much lower scores for many artifact-centric systems. At the same time, these simple VFM probes remain weak on pure VAE reconstruction and localized edits. For Human-AIGI benchmarking, this creates a new baseline condition: claims about human-level or superhuman perception must now be situated against strong, low-complexity foundation-model probes rather than only against bespoke forensic architectures [2602.01738].

## 4. Collaboration, common ground, and dialogue discrimination

A second major interpretation of Human-AIGI benchmarking concerns collaboration rather than perception. “A Benchmark to Assess Common Ground in Human-AI Collaboration” presents a collaborative puzzle task grounded in Clark and Brennan’s theory of common ground and grounding, and explicitly aligns its motivation with capacities demanded of a **Human–AGI** or **Human–AIGI** collaborator [2602.21337]. The task uses a **4×4 grid work area**, a target pattern with **4 pieces**, and a Worker who sees **24 candidate pieces** while a Helper sees the target pattern but not the full inventory. Communication occurs through text chat, under either **shared-view** or **non-shared-view** conditions. The AI system, GPT-4.1 with vision, alternates with the human between Helper and Worker roles. The benchmark measures exact puzzle success, communication effort, referential conventions, and dialogue acts labeled as presentation, clarification, repair, or acceptance [2602.21337].

The user study includes **40 human participants** in a **2×2 between-subject design**. It reproduces several classical human-human findings: shared visual context improves performance, with success rates significantly higher in the shared-view condition \((\chi^2(1)=11.57,p<0.001)\); Human Helpers significantly reduce word counts across trials in shared view \((F(1,28)=11.67, p<0.01)\); and clarification requests decrease over trials in shared view \((F(1,28)=27.63, p<0.0001\) for the AI Worker). Yet it also reveals divergences: no robust learning effect in overall task accuracy, shallow grounding by the AI, weak belief-state maintenance, and an asymmetrical grounding burden in which humans do more clarification and repair early on but reduce that effort when the AI does not reciprocate [2602.21337]. In benchmark terms, this converts Human-AIGI from static correctness into dynamic mutual understanding.

Inverse Turing Bench supplies an adjacent dialogue-centered module [2606.21844]. It contains **557 pairs** of multi-turn dialogues, each pair sharing the same human interrogator, with one dialogue human–human and the other human–AI. The task is a binary choice:
\[
who\_is\_human \in \{\text{A}, \text{B}\}.
\]
The benchmark tests LLMs and detectors as judges of whether interaction patterns are human-only or human-AI. GPTZero reaches **89.41%** accuracy in the witness-only setting; Claude Opus 4.6 reaches **77.92%**; GPT-5.5 reaches **75.94%**; and the **Human Judge** baseline is **54.58%** [2606.21844]. Persona prompting substantially reduces semantic judges’ performance—Claude Opus 4.6 drops from **95.80%** to **59.04%**, GPT-5.5 from **93.36%** to **57.56%**, and human judges from **68.53%** to **39.85%**—while GPTZero remains comparatively stable. This suggests that a Human-AIGI benchmark for social interaction must measure not only raw discrimination accuracy but also robustness to persona manipulation and shifts in conversational style [2606.21844].

## 5. Human-referenced reasoning and general-assistant evaluation

A broader Human-AIGI paradigm appears in multimodal reasoning and general-assistant benchmarks. Human-Aligned Bench is explicitly designed for fine-grained alignment of multimodal reasoning with human performance [2505.11141]. It contains **9,794** questions spanning **visual reasoning**, **definition judgment**, **analogical reasoning**, and **logical judgment**, in both Chinese and English, with each item annotated by human correct rate and human error-prone option. The average human score rate is **68.82%**. Gemini-2.5-Pro-exp-03-25 reaches **68.41%**, essentially matching this benchmark-wide human average, while visual reasoning remains difficult for all current MLLMs, often near **25–30%** overall. The benchmark is distinctive because it allows comparison not only of accuracy but of whether models deteriorate with human-defined difficulty in a human-like way and whether they choose the same distractors humans tend to choose [2505.11141].

AGIBench operationalizes human-referenced difficulty at larger scale [2309.06495]. It contains **927 questions**, covering **3 ability branches**, **20 knowledge categories**, and **68 knowledge subclasses**, with every question labeled by the four-tuple
\[
q = \langle a, k, d, m \rangle .
\]
Difficulty levels are defined from the empirical accuracy of **millions of well-educated humans**, with Level 1 corresponding to human accuracy in \([80\%,100\%]\) and Level 5 to \([0\%,20\%)\). This design allows a benchmark to ask not only whether a model is correct, but where it sits relative to human performance curves across common sense, reasoning, understanding, knowledge area, and modality [2309.06495].

AGIEval similarly grounds evaluation in **20 tasks** and **8,062 questions** drawn from human standardized exams such as Gaokao, SAT, LSAT, GMAT, GRE-style algebraic reasoning, civil-service exams, the Chinese lawyer qualification test, and AMC/AIME mathematics [2304.06364]. It reports average and top human performance baselines and decomposes model failures into **understanding**, **knowledge**, **reasoning**, and **calculation**. GPT-4 reaches **95%** on SAT Math and **92.5%** on the English test of the Chinese national college entrance exam, while remaining less proficient in tasks requiring complex reasoning or specific domain knowledge [2304.06364]. GAIA pushes in a different direction by selecting **466** real-world questions that are conceptually simple for humans but difficult for AI assistants, with **92%** human accuracy versus **15%** for GPT-4 equipped with plugins [2311.12983]. Its tasks require web browsing, multi-modality, and tool use, and its methodology departs from exam difficulty maximization by treating robust performance on easy-for-humans tasks as a milestone for general AI assistants [2311.12983].

Taken together, these benchmarks define a human-centered AGI evaluation regime in which difficulty is human-referenced, comparison is head-to-head with human baselines, and benchmark structure is intentionally diagnostic rather than leaderboard-only. This suggests that a mature Human-AIGI benchmark is likely to be modular: perceptual, interactive, and reasoning components are all needed because no single task family captures human-like generality [2505.11141][2311.12983].

## 6. Methodological criticisms, limitations, and future directions

A central critique of current benchmarking is that many claims about “human-level” or “superhuman” AI rely on labels and tasks that are not truly human-grounded. “On Benchmarking Human-Like Intelligence in Machines” argues that current evaluation paradigms suffer from **a lack of human-validated labels**, **inadequate representation of human response variability and uncertainty**, and **simplified and ecologically-invalid tasks** [2502.20502]. In a human evaluation study on **10** existing benchmarks, mean agreement with the benchmark label is **63.51%** with standard deviation **20.99**, and **26.67%** of stimuli receive **less than 50%** agreement. Moreover, **57.69%** of ratings fall between **20 and 80**, indicating that human judgments are often graded rather than categorical [2502.20502]. The paper recommends using human data as ground truth, evaluating population-level distributions rather than single labels, capturing gradedness and uncertainty, grounding tasks in cognitive meta-theory, and designing ecologically valid, cognitively rich tasks.

These criticisms map directly onto the current Human-AIGI literature. Human-AIGI in MIRROR is limited to photographic-style images from the MSCOCO domain and a **50-participant** psychophysical study; the exact display conditions are not fully specified, and the benchmark is a snapshot in a rapidly changing generator landscape [2602.02222]. The common-ground benchmark uses a constrained puzzle task, a single fixed model configuration, text-only communication, and no explicit formalization of a common-ground state \(CG_t\) [2602.21337]. Inverse Turing Bench is based on a fixed dataset, a limited persona family, and dialogues produced in an adversarial Turing-test setting that may be easier to classify than ordinary conversations [2606.21844]. GAIA, despite its strong human baseline, is English-only, evaluates final answers rather than traces, and requires substantial manual effort because only **68%** of initially crafted questions survive unambiguous validation [2311.12983].

Future directions converge on several themes. One is **dynamic benchmarking**: GAIA advocates removing broken questions and adding new ones over time; Inverse Turing Bench argues for a live benchmark that evolves with conversational systems; the VFM-based AIGI detection literature implies that evaluation must keep pace with new generators and web distributions [2311.12983][2606.21844][2602.01738]. A second is **richer human-centered instrumentation**: MIRROR integrates confidence and response latency; Human-Aligned Bench adds human error-prone options; AIGI-Holmes adds human preference data and explanation evaluation [2602.02222][2505.11141][2507.02664]. A third is **modal expansion**: several works explicitly point toward video deepfakes, multimodal content, longer time horizons, broader domains, and more detailed psychophysical or interaction protocols [2602.02222][2602.21337]. The resulting trajectory is toward benchmark suites that do not merely score models against answer keys, but locate them on human perceptual, cognitive, and collaborative scales.

Source: https://www.emergentmind.com/topics/human-aigi-benchmark