AGIEval: Human-Centric Exam Benchmark
- AGIEval is a human-centric benchmark that assesses foundation models using authentic, standardized exam tasks from diverse domains.
- It utilizes objective exam questions from tests such as Gaokao, SAT, LSAT, and math competitions to evaluate understanding, reasoning, and calculation.
- Empirical results show that models like GPT-4 excel on several tasks, yet struggles persist in handling complex reasoning and domain-specific challenges.
AGIEval is a human-centric benchmark for evaluating foundation models on official, public, high-standard exams that more closely resemble real human tests and decision-making tasks than artificial NLP datasets. It was introduced to assess the general abilities of foundation models through standardized exams such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests, and to analyze performance across understanding, knowledge, reasoning, and calculation (Zhong et al., 2023). In subsequent model-evaluation practice, AGIEval has been used not only as an exam-style reasoning benchmark in its original sense, but also as a zero-shot or few-shot probe for generalization, dialogue quality preservation, multilingual capability, out-of-domain reasoning, and mathematical capability transfer (Wang et al., 2023, Mitra et al., 2023, Ming et al., 2024, Balasubramanian et al., 7 Apr 2026).
1. Origins and design rationale
AGIEval was proposed from the premise that traditional benchmarks such as GLUE and SuperGLUE, while useful, often measure narrow machine skills on artificial tasks and do not fully capture human-level cognition. Its construction therefore centers on official standards for assessing human capability rather than synthetic benchmark design. The benchmark asks whether foundation models can perform like humans on standardized exams, how well they handle understanding, knowledge, reasoning, and calculation, and where they continue to fail on tasks humans find challenging (Zhong et al., 2023).
The design is explicitly human-centered. The source materials are standardized exams used for admissions or qualifications, and the benchmark compares models not only against one another but also against human baselines. The original paper treats this as a more meaningful proxy for real-world, human-centered intelligence because the underlying tasks reflect educational and professional standards rather than benchmark-specific annotation schemes (Zhong et al., 2023).
A central construction choice is restriction to objective questions. AGIEval includes only multiple-choice and fill-in-the-blank questions, excluding subjective or essay-style items that would require human expert scoring. This choice preserves automatic evaluation while still drawing from authentic exam settings. The final benchmark contains 8,062 questions across 20 tasks and is bilingual, covering both English and Chinese (Zhong et al., 2023).
2. Task inventory and benchmark composition
AGIEval spans several exam families. It includes general college entrance or admission exams, professional and qualification tests, logic and civil-service-style reasoning, and high-school math competition problems. These families jointly cover language understanding, commonsense and factual knowledge, legal reasoning, mathematical problem solving, multi-step logical deduction, and quantitative reasoning (Zhong et al., 2023).
| Exam family | Included tasks | Source role |
|---|---|---|
| College entrance / admission | Chinese Gaokao, SAT, GRE, GMAT | Admissions-style reasoning and knowledge |
| Professional / qualification | LSAT, Lawyer Qualification Test | Legal and professional reasoning |
| Logic / civil service | LogiQA-English, LogiQA-Chinese | Formal and verbal reasoning |
| Math competition / quantitative reasoning | MATH, AQuA-RAT | Competition-style and GRE/GMAT-like math |
Within those families, the benchmark enumerates Chinese Gaokao History, Math QA, Math Cloze, English, Chinese, Geography, Biology, Chemistry, and Physics; SAT-English and SAT-Math; GRE and GMAT math-related questions; LSAT Logical Reasoning, Reading Comprehension, and Analytical Reasoning; Lawyer Qualification Test tasks JEC-QA-KD and JEC-QA-CA; LogiQA in English and Chinese; MATH problems from AMC/AIME-style sources; and AQuA-RAT quantitative reasoning. The dataset sources include public Gaokao and SAT materials, previous LSAT datasets, LogiQA, MATH, AQuA-RAT, and JEC-QA (Zhong et al., 2023).
Human performance baselines are part of the benchmark specification. The benchmark reports average human performance and top human performance, with score scaling to a 100-point scale for exams such as LSAT, SAT, and Gaokao; for the lawyer qualification test, top human performance is approximated with the top 10%. The original paper notes that these are approximations, but treats them as a useful human reference frame (Zhong et al., 2023).
3. Evaluation protocol and prompt regimes
The original AGIEval evaluation uses classification accuracy for multiple-choice questions and Exact Match and F1 for fill-in-the-blank questions. The evaluated models are GPT-4, ChatGPT, Text-Davinci-003, and Vicuna-13B. Prompting is tested under zero-shot, few-shot, zero-shot CoT, and few-shot CoT settings, with Azure OpenAI API parameters temperature , max tokens , top , and frequency penalty (Zhong et al., 2023).
Prompting methodology is not incidental in AGIEval. The benchmark paper reports that few-shot learning gives only limited improvement over zero-shot for many tasks, suggesting that modern instruction-tuned LLMs already possess strong zero-shot ability relative to earlier systems. Chain-of-Thought prompting helps some tasks, especially AQUA-RAT, MATH, and SAT-Math, but is not universally beneficial. It can hurt on some logic tasks, some Chinese tasks, and some tasks where generated reasoning becomes misleading. The original analysis therefore treats CoT as task-, model-, and language-dependent rather than uniformly advantageous (Zhong et al., 2023).
Later uses of AGIEval vary substantially in protocol. Some studies evaluate all English multiple-choice tasks in zero-shot with the official AGIEval prompt and answer matching; others use no exemplars and no CoT; some use 5-shot settings; some restrict AGIEval to English multiple-choice subsets; some isolate AGIEval-Math; and some score open-ended generations by extracting an option with GPT-4 rather than by direct answer parsing (Wang et al., 2023, Mukherjee et al., 2023, Chen et al., 2024, Mitra et al., 2024, Balasubramanian et al., 7 Apr 2026). This suggests that an “AGIEval score” is only interpretable together with its prompt regime, subset definition, and answer-extraction procedure.
4. Empirical findings in the benchmark paper
The foundational paper reports GPT-4 as the strongest model overall. Under zero-shot CoT, GPT-4 achieves 58.4% average across tasks, compared with 43.2% for ChatGPT and 37.4% for Text-Davinci-003. It exceeds average human performance on several benchmarks, including SAT, LSAT, and math competitions, and attains 95.0% on SAT-Math and 92.5% on Gaokao-English under zero-shot CoT. Additional reported GPT-4 scores include 85.9% on SAT-English, 85.1% on LSAT-RC, 80.6% on LSAT-LR, 73.2% on AQUA-RAT, and 47.7% on MATH (Zhong et al., 2023).
The same paper emphasizes that difficult tasks remain difficult. Across models, the hardest tasks include MATH, LSAT-AR, Gaokao Physics, and Gaokao Math. Even GPT-4 continues to struggle on high-level mathematical and multi-step reasoning tasks, and the paper explicitly states that GPT-4 is less proficient in tasks requiring complex reasoning or specific domain knowledge (Zhong et al., 2023).
A distinctive feature of the benchmark paper is its capability-level error analysis. The authors manually examine 100 incorrectly answered instances per task from ChatGPT under zero-shot CoT and score four dimensions—understanding, knowledge, reasoning, and calculation—with binary indicators. The analysis concludes that models are generally good at understanding question meaning, context, syntax, and linguistic nuances, but weaker on long contexts, similar concepts, variable substitution, and complex symbols. Knowledge failures are prominent in law, biology, physics, chemistry, and domain-specific formulas. Reasoning is identified as the weakest dimension, with errors in strict logical deduction, counterfactual reasoning, multi-hop reasoning, consistent inference maintenance, and contradiction avoidance. Calculation is weaker than understanding and varies by subject, with especially poor performance on symbolic manipulation and multi-step arithmetic or algebra (Zhong et al., 2023).
The benchmark also includes contamination analysis. GPT-4 performance on uncontaminated subsets often drops only slightly, but math-related subjects show more noticeable degradation, including Gaokao-geography , Gaokao-biology , Gaokao-chemistry , Gaokao-physics , Gaokao-MathQA , and Gaokao-mathcloze . The paper treats this as an important caveat while still arguing that AGIEval remains useful (Zhong et al., 2023).
5. Subsequent use as an evaluation instrument
After its introduction, AGIEval was incorporated into a broad range of model studies. These uses are heterogeneous. Some papers employ it as a standardized zero-shot English multiple-choice reasoning benchmark; some use it as a generalization check to test whether chat alignment degrades exam-style competence; some treat it as a 5-shot dialogue benchmark; some analyze it as out-of-domain reasoning; some use it to evaluate reinforcement-learning transfer beyond mathematics; and some focus specifically on AGIEval-Math as a target for mathematical capability transfer (Wang et al., 2023, Mitra et al., 2023, Liao et al., 2024, Akter et al., 15 Apr 2025, Balasubramanian et al., 7 Apr 2026).
| Paper | AGIEval configuration | Reported result |
|---|---|---|
| OpenChat (Wang et al., 2023) | All English multiple-choice tasks; official zero-shot prompt; accuracy | openchat-13b 36.4 average; llama-2-13b 35.0 |
| Orca (Mukherjee et al., 2023) | Zero-shot; no exemplars; no CoT; English multiple-choice | Orca-13B 41.7 average |
| Orca 2 (Mitra et al., 2023) | Zero-shot reasoning benchmark with task-wise appendix metrics | Orca-2-13B 49.93 average |
| AgentInstruct / Orca-3 (Mitra et al., 2024) | Zero-shot; open-ended output parsed by GPT-4; accuracy | Orca-3-7B 56.80; Mistral-7B-Instruct 40.52 |
| Nemotron-CrossThink (Akter et al., 15 Apr 2025) | Benchmark-level aggregate in RL training analysis | Qwen-2.5-7B 48.59; best blend 63.71 |
| Marco-LLM (Ming et al., 2024) | One-language 5-shot benchmark under “General Knowledge” | Marco-7B 68.8; Marco-72B 84.4 |
Other papers further illustrate the benchmark’s range of uses. CodingTeachLLM reports AGIEval only as a 5-shot dialogue evaluation for a quantized 13B model, with 45.27 for the reported AGIEval score (Chen et al., 2024). NesyCD uses AGIEval as an out-of-domain reasoning benchmark with 2,546 English multiple-choice examples and reports 33.6 for LLaMA2-7B under its full method (Liao et al., 2024). AttentionInfluence uses AGIEval-en as a reasoning-heavy downstream target and reports an increase from 24.50 to 26.30 in its main setting (Hua et al., 12 May 2025). Dr.LLM evaluates AGIEval as an out-of-domain exam-style reasoning benchmark for routing policies and reports accuracy drops such as 43.2 to 41.5 for LLaMA8B and 61.2 to 57.2 for Qwen7B, while preserving efficiency (Heakl et al., 14 Oct 2025). UNLOCK focuses on AGIEval-Math and reports that transferring a mathematical reasoning direction raises Qwen3-14B-Base from 61.1% to 71.3%, exceeding the 67.8% achieved by the 14B post-trained model (Balasubramanian et al., 7 Apr 2026).
Taken together, these studies show that AGIEval has become less a single fixed leaderboard than a recurring exam-style testbed for several research questions: reasoning quality, robustness under alignment, post-training gains, multilingual pretraining effects, retrieval- or routing-based efficiency tradeoffs, and capability transfer across models. This suggests that AGIEval’s enduring value lies in its task family and evaluation philosophy as much as in any single headline score.
6. Limitations, interpretive cautions, and reporting issues
Several limitations arise from the original benchmark and from later usage patterns. First, AGIEval is not a monolithic protocol. The original paper evaluates bilingual tasks under zero-shot, few-shot, zero-shot CoT, and few-shot CoT; later papers often restrict evaluation to English multiple-choice tasks, use no-CoT zero-shot settings, adopt 5-shot prompts, or isolate only AGIEval-Math (Zhong et al., 2023, Wang et al., 2023, Mukherjee et al., 2023, Balasubramanian et al., 7 Apr 2026). Direct numerical comparison across papers therefore requires care.
Second, prompt engineering and parsing materially affect reported performance. The benchmark paper shows that CoT is beneficial for some tasks but harmful for others, so AGIEval does not support a simple claim that more explicit reasoning prompts always improve results (Zhong et al., 2023). In AgentInstruct, multiple-choice answers are extracted from open-ended generations by GPT-4, introducing a different evaluation layer than direct option prediction (Mitra et al., 2024). In Dr.LLM, AGIEval is scored through lm-eval-harness using acc_norm from log-likelihoods rather than free-form answer generation (Heakl et al., 14 Oct 2025).
Third, later papers sometimes repurpose AGIEval in ways that differ substantially from its original positioning. CodingTeachLLM reports AGIEval as a dialogue evaluation benchmark, whereas the original benchmark is defined around standardized exams and objective questions (Chen et al., 2024, Zhong et al., 2023). Marco-LLM uses it as a one-language 5-shot “General Knowledge” benchmark rather than as a bilingual exam battery (Ming et al., 2024). Nemotron-CrossThink reports benchmark-level aggregates and appendix subcategory analyses, while UNLOCK focuses on AGIEval-Math alone (Akter et al., 15 Apr 2025, Balasubramanian et al., 7 Apr 2026).
Finally, there are reporting issues around benchmark mention versus benchmark evidence. In the provided record, the LISA paper explicitly contains no AGIEval occurrences in the main paper, appendix, tables, figures, or references, and therefore provides no AGIEval-specific setup or result (Pan et al., 2024). The Paramanu-Ayn record likewise states that the paper does not mention AGIEval in the main body, tables, or appendix and reports no AGIEval scores or comparisons (Niyogi et al., 2024). A plausible implication is that AGIEval citations in model summaries should be distinguished from documented AGIEval evaluations.
AGIEval thus occupies an unusual position in model evaluation. It is simultaneously a concrete bilingual benchmark of 8,062 objective exam questions and a broader evaluative motif for human-centric, exam-style reasoning. Its empirical record demonstrates strong progress by contemporary models, especially GPT-4 in the original study, but it also documents persistent weakness in strict reasoning, domain-specific knowledge, and multi-step calculation. Subsequent work has reinforced its relevance while also fragmenting its operational meaning across zero-shot, few-shot, English-only, dialogue-style, multilingual, out-of-domain, and math-only variants (Zhong et al., 2023, Mitra et al., 2023, Ming et al., 2024, Akter et al., 15 Apr 2025).
