Human-AIGI Benchmark

• Human-AIGI Benchmark is a human-centered evaluation framework that compares AI-generated image detection, collaborative, and reasoning tasks against measured human performance.
• It employs psychophysical experiments capturing response latency, confidence, and error patterns to determine the point where AI detectors achieve superhuman accuracy.
• The benchmark landscape integrates diverse tasks—from perceptual and interaction to general-assistant evaluations—to provide a comprehensive, human-referenced performance scale.

Human-AIGI Benchmark denotes a human-centered benchmarking paradigm for assessing artificial systems against human perceptual, cognitive, and collaborative performance, and, in a narrower usage, the psychophysically curated Human-AIGI dataset introduced for AI-generated image detection in MIRROR (Liu et al., 2 Feb 2026). Across recent work, the unifying principle is that model capability is not judged only by aggregate task accuracy, but by calibration to human perceptual limits, human success rates, human error patterns, human uncertainty, or human interaction structure. This orientation appears in image forensics, multimodal reasoning, common-ground formation, general-assistant evaluation, and human-versus-AI dialogue discrimination (Qiu et al., 16 May 2025, Poelitz et al., 24 Feb 2026, Mialon et al., 2023).

1. Conceptual scope and benchmark landscape

In current arXiv usage, the term spans several related benchmark forms. One form is perceptual benchmarking, where the objective is to determine whether detectors exceed human observers on hard AI-generated images. A second is interaction benchmarking, where the target is common ground, grounding, repair, and role-flexible collaboration between humans and AI systems. A third is human-referenced cognitive benchmarking, where task difficulty or success criteria are anchored to human populations, as in standardized exams, multimodal reasoning sets, or general-assistant tasks (Liu et al., 2 Feb 2026, Poelitz et al., 24 Feb 2026, Tang et al., 2023).

Benchmark	Domain	Human reference signal
Human-AIGI	AIGI detection	Accuracy, confidence, response latency
A-Bench	AIGI evaluation by LMMs	Expert-authored QA, human participant accuracy
Common Ground benchmark	Human-AI collaboration	User study, task success, grounding acts
Inverse Turing Bench	Dialogue discrimination	Human judge baseline
Human-Aligned Bench	Multimodal reasoning	Human correct rate, error-prone options
AGIBench / AGIEval / GAIA	General intelligence or assistant tasks	Human-referenced difficulty or direct human performance

This landscape is methodologically heterogeneous but conceptually coherent. Human-AIGI benchmarks typically replace leaderboard-only evaluation with one or more of the following: direct human comparison, psychophysical curation, human-referenced difficulty tiers, human preference data, or process-level interaction analysis. This suggests that the defining attribute is not a single modality or task family, but explicit coupling between machine evaluation and measured human behavior (Ying et al., 27 Feb 2025, Zhong et al., 2023).

2. Human-AIGI in AI-generated image detection

The most specific and technically developed instance is the Human-AIGI benchmark introduced by MIRROR, whose stated purpose is to answer when AIGI detectors actually surpass human observers, including visual experts, on images that are hard for humans (Liu et al., 2 Feb 2026). Existing AIGI benchmarks such as CNNSpot, GenImage, AIGCDetect, DRCT-2M, UniversalFakeDetect, Synthbuster, WildRF, and AIGIBench emphasize generator coverage, semantic diversity, or robustness to distortions, but do not directly measure the point at which detectors outperform humans on the most deceptive images. MIRROR therefore frames Human-AIGI around a “superhuman crossover”: the point where a detector exceeds the human perceptual limit on near-indistinguishable real versus AI-generated images.

Human-AIGI is built from MSCOCO real images and AI-generated images from 27 generators spanning diffusion, autoregressive, unified, and closed-source commercial systems, including SD-3-Medium, SD-3.5-Large, FLUX.1-dev, PixArt-α, Qwen-Image, Infinity-8B, BAGEL, Janus-Pro-7B, OmniGen2-7B, Seedream 3.0, Gemini 2.0 Flash, Imagen 4, HunyuanImage-3.0, and GPT-Image (Liu et al., 2 Feb 2026). Its defining element is the Human-Imperceptible subset $D_{\text{hard}}$, created through psychophysical experiments with 50 subjects, comprising lay users and visual experts. For each image, the benchmark records binary real-versus-fake decisions, confidence $S(x)$, and response latency $RT(x)$. Hard examples are selected as “perceptual corner cases” using the rule

$$D_{\text{hard}} = \{x \in D : S(x) \ge \tau_{\text{real}} \;\lor\; RT(x) > \mu + \sigma_{\text{rt}} \},$$

so that selected items either deceive humans into a “Real” judgment with high confidence or induce significant hesitation (Liu et al., 2 Feb 2026).

Evaluation uses the standard binary detection task on both the original test partition and $D_{\text{hard}}$. Human baselines are reported as 76.9% accuracy for lay users and 88.3% for visual experts. MIRROR reaches 89.6% accuracy across 27 generators on Human-AIGI and 89.5% on the Human-Imperceptible subset, thereby surpassing lay users and slightly exceeding visual experts under this operational definition of superhuman performance. On the hard subset, MIRROR outperforms the next-best baseline, DDA at 86.9%, by 2.6 percentage points (Liu et al., 2 Feb 2026).

Methodologically, MIRROR links the benchmark to a theory of human perception as reference–comparison on a human cognitive manifold of real-world regularities. The detector learns a real-image manifold from 200k real images from MSCOCO, stores it in a discrete memory bank with $K = 4096$ orthogonal prototypes, projects test images into a manifold-consistent reference, and uses residuals and reconstruct perplexity as detection signals. A plausible implication is that Human-AIGI is designed not merely to compare accuracies, but to test whether explicit modeling of reality priors yields superhuman performance precisely where human perception fails (Liu et al., 2 Feb 2026).

3. Human-centered evaluation of AIGI judges and explanation systems

The image-forensics strand extends beyond binary detection. A-Bench asks whether large multimodal models are reliable evaluators of AI-generated images, rather than generators or detectors in the usual sense (Zhang et al., 2024). It contains 2,864 AIGIs from 16 text-to-image models, split into 1,408 high-level semantic items and 1,456 low-level quality items. Annotation was performed by 15 experts with professional experience in photography and AIGI evaluation, with each QA pair reviewed by at least three other experts. Human participants achieve 92.39%–94.00% on the semantic subset and 90.53%–92.25% on the quality subset, whereas the best proprietary LMM, Gemini 1.5 Pro, reaches 84.69% and 69.06% respectively. The benchmark therefore shows that even the best LMM remains substantially below human performance, especially on generative distortions and quality perception (Zhang et al., 2024).

AIGI-Holmes defines a related but distinct Human-AIGI regime centered on human-verifiable explanations and human-aligned detection (Zhou et al., 3 Jul 2025). Holmes-Set combines Holmes-SFTSet and Holmes-DPOSet, covering 18 generators and 65K+4K instances with explanations and preference data. Its benchmark logic is not limited to a single leaderboard: it integrates explanation-grounded supervision, preference optimization, and evaluation by automatic metrics, MLLM-as-a-judge scores, and human ELO ratings. On Protocol-III, which tests unseen generators such as Janus, VAR, Infinity, Show-o, PixArt-XL, SD3.5-Large, and FLUX, AIGI-Holmes reaches 99.2% mean accuracy and 99.9% mean average precision. On the explanation benchmark, AIGI-Holmes (DPO) reports BLEU-1 = 0.622, ROUGE-L = 0.375, METEOR = 0.311, CIDEr = 0.107, and ELO = 11.420, outperforming its SFT-only version and raw MLLM baselines (Zhou et al., 3 Jul 2025). This broadens the Human-AIGI notion from “can a model detect what humans miss?” to “can it justify its decision in a form humans can verify and prefer?”

The baseline problem has also shifted. “Simplicity Prevails” shows that a single linear classifier on frozen features from modern Vision Foundation Models can now outperform many specialized AIGI detectors, especially on in-the-wild distributions (Zhou et al., 2 Feb 2026). DINOv3-Linear reaches 0.964 average accuracy on GenImage and 0.940 average across Chameleon, WildRF, SocialRF, and CommunityAI, compared with 0.850 for DDA and much lower scores for many artifact-centric systems. At the same time, these simple VFM probes remain weak on pure VAE reconstruction and localized edits. For Human-AIGI benchmarking, this creates a new baseline condition: claims about human-level or superhuman perception must now be situated against strong, low-complexity foundation-model probes rather than only against bespoke forensic architectures (Zhou et al., 2 Feb 2026).

4. Collaboration, common ground, and dialogue discrimination

A second major interpretation of Human-AIGI benchmarking concerns collaboration rather than perception. “A Benchmark to Assess Common Ground in Human-AI Collaboration” presents a collaborative puzzle task grounded in Clark and Brennan’s theory of common ground and grounding, and explicitly aligns its motivation with capacities demanded of a Human–AGI or Human–AIGI collaborator (Poelitz et al., 24 Feb 2026). The task uses a 4×4 grid work area, a target pattern with 4 pieces, and a Worker who sees 24 candidate pieces while a Helper sees the target pattern but not the full inventory. Communication occurs through text chat, under either shared-view or non-shared-view conditions. The AI system, GPT-4.1 with vision, alternates with the human between Helper and Worker roles. The benchmark measures exact puzzle success, communication effort, referential conventions, and dialogue acts labeled as presentation, clarification, repair, or acceptance (Poelitz et al., 24 Feb 2026).

The user study includes 40 human participants in a 2×2 between-subject design. It reproduces several classical human-human findings: shared visual context improves performance, with success rates significantly higher in the shared-view condition $(\chi^2(1)=11.57,p<0.001)$; Human Helpers significantly reduce word counts across trials in shared view $(F(1,28)=11.67, p<0.01)$; and clarification requests decrease over trials in shared view $(F(1,28)=27.63, p<0.0001$ for the AI Worker). Yet it also reveals divergences: no robust learning effect in overall task accuracy, shallow grounding by the AI, weak belief-state maintenance, and an asymmetrical grounding burden in which humans do more clarification and repair early on but reduce that effort when the AI does not reciprocate (Poelitz et al., 24 Feb 2026). In benchmark terms, this converts Human-AIGI from static correctness into dynamic mutual understanding.

Inverse Turing Bench supplies an adjacent dialogue-centered module (Hager et al., 20 Jun 2026). It contains 557 pairs of multi-turn dialogues, each pair sharing the same human interrogator, with one dialogue human–human and the other human–AI. The task is a binary choice: $who\_is\_human \in \{\text{A}, \text{B}\}.$ The benchmark tests LLMs and detectors as judges of whether interaction patterns are human-only or human-AI. GPTZero reaches 89.41% accuracy in the witness-only setting; Claude Opus 4.6 reaches 77.92%; GPT-5.5 reaches 75.94%; and the Human Judge baseline is 54.58% (Hager et al., 20 Jun 2026). Persona prompting substantially reduces semantic judges’ performance—Claude Opus 4.6 drops from 95.80% to 59.04%, GPT-5.5 from 93.36% to 57.56%, and human judges from 68.53% to 39.85%—while GPTZero remains comparatively stable. This suggests that a Human-AIGI benchmark for social interaction must measure not only raw discrimination accuracy but also robustness to persona manipulation and shifts in conversational style (Hager et al., 20 Jun 2026).

5. Human-referenced reasoning and general-assistant evaluation

A broader Human-AIGI paradigm appears in multimodal reasoning and general-assistant benchmarks. Human-Aligned Bench is explicitly designed for fine-grained alignment of multimodal reasoning with human performance (Qiu et al., 16 May 2025). It contains 9,794 questions spanning visual reasoning, definition judgment, analogical reasoning, and logical judgment, in both Chinese and English, with each item annotated by human correct rate and human error-prone option. The average human score rate is 68.82%. Gemini-2.5-Pro-exp-03-25 reaches 68.41%, essentially matching this benchmark-wide human average, while visual reasoning remains difficult for all current MLLMs, often near 25–30% overall. The benchmark is distinctive because it allows comparison not only of accuracy but of whether models deteriorate with human-defined difficulty in a human-like way and whether they choose the same distractors humans tend to choose (Qiu et al., 16 May 2025).

AGIBench operationalizes human-referenced difficulty at larger scale (Tang et al., 2023). It contains 927 questions, covering 3 ability branches, 20 knowledge categories, and 68 knowledge subclasses, with every question labeled by the four-tuple

$S(x)$0

Difficulty levels are defined from the empirical accuracy of millions of well-educated humans, with Level 1 corresponding to human accuracy in $S(x)$1 and Level 5 to $S(x)$2. This design allows a benchmark to ask not only whether a model is correct, but where it sits relative to human performance curves across common sense, reasoning, understanding, knowledge area, and modality (Tang et al., 2023).

AGIEval similarly grounds evaluation in 20 tasks and 8,062 questions drawn from human standardized exams such as Gaokao, SAT, LSAT, GMAT, GRE-style algebraic reasoning, civil-service exams, the Chinese lawyer qualification test, and AMC/AIME mathematics (Zhong et al., 2023). It reports average and top human performance baselines and decomposes model failures into understanding, knowledge, reasoning, and calculation. GPT-4 reaches 95% on SAT Math and 92.5% on the English test of the Chinese national college entrance exam, while remaining less proficient in tasks requiring complex reasoning or specific domain knowledge (Zhong et al., 2023). GAIA pushes in a different direction by selecting 466 real-world questions that are conceptually simple for humans but difficult for AI assistants, with 92% human accuracy versus 15% for GPT-4 equipped with plugins (Mialon et al., 2023). Its tasks require web browsing, multi-modality, and tool use, and its methodology departs from exam difficulty maximization by treating robust performance on easy-for-humans tasks as a milestone for general AI assistants (Mialon et al., 2023).

Taken together, these benchmarks define a human-centered AGI evaluation regime in which difficulty is human-referenced, comparison is head-to-head with human baselines, and benchmark structure is intentionally diagnostic rather than leaderboard-only. This suggests that a mature Human-AIGI benchmark is likely to be modular: perceptual, interactive, and reasoning components are all needed because no single task family captures human-like generality (Qiu et al., 16 May 2025, Mialon et al., 2023).

6. Methodological criticisms, limitations, and future directions

A central critique of current benchmarking is that many claims about “human-level” or “superhuman” AI rely on labels and tasks that are not truly human-grounded. “On Benchmarking Human-Like Intelligence in Machines” argues that current evaluation paradigms suffer from a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and simplified and ecologically-invalid tasks (Ying et al., 27 Feb 2025). In a human evaluation study on 10 existing benchmarks, mean agreement with the benchmark label is 63.51% with standard deviation 20.99, and 26.67% of stimuli receive less than 50% agreement. Moreover, 57.69% of ratings fall between 20 and 80, indicating that human judgments are often graded rather than categorical (Ying et al., 27 Feb 2025). The paper recommends using human data as ground truth, evaluating population-level distributions rather than single labels, capturing gradedness and uncertainty, grounding tasks in cognitive meta-theory, and designing ecologically valid, cognitively rich tasks.

These criticisms map directly onto the current Human-AIGI literature. Human-AIGI in MIRROR is limited to photographic-style images from the MSCOCO domain and a 50-participant psychophysical study; the exact display conditions are not fully specified, and the benchmark is a snapshot in a rapidly changing generator landscape (Liu et al., 2 Feb 2026). The common-ground benchmark uses a constrained puzzle task, a single fixed model configuration, text-only communication, and no explicit formalization of a common-ground state $S(x)$3 (Poelitz et al., 24 Feb 2026). Inverse Turing Bench is based on a fixed dataset, a limited persona family, and dialogues produced in an adversarial Turing-test setting that may be easier to classify than ordinary conversations (Hager et al., 20 Jun 2026). GAIA, despite its strong human baseline, is English-only, evaluates final answers rather than traces, and requires substantial manual effort because only 68% of initially crafted questions survive unambiguous validation (Mialon et al., 2023).

Future directions converge on several themes. One is dynamic benchmarking: GAIA advocates removing broken questions and adding new ones over time; Inverse Turing Bench argues for a live benchmark that evolves with conversational systems; the VFM-based AIGI detection literature implies that evaluation must keep pace with new generators and web distributions (Mialon et al., 2023, Hager et al., 20 Jun 2026, Zhou et al., 2 Feb 2026). A second is richer human-centered instrumentation: MIRROR integrates confidence and response latency; Human-Aligned Bench adds human error-prone options; AIGI-Holmes adds human preference data and explanation evaluation (Liu et al., 2 Feb 2026, Qiu et al., 16 May 2025, Zhou et al., 3 Jul 2025). A third is modal expansion: several works explicitly point toward video deepfakes, multimodal content, longer time horizons, broader domains, and more detailed psychophysical or interaction protocols (Liu et al., 2 Feb 2026, Poelitz et al., 24 Feb 2026). The resulting trajectory is toward benchmark suites that do not merely score models against answer keys, but locate them on human perceptual, cognitive, and collaborative scales.