- The paper introduces EduAgentBench, a 150-task, source-grounded benchmark spanning pedagogical judgment, situated tutoring, and Canvas-style teaching workflows with evidence-based verifiers.
- The benchmark finds that models pass 89.6% of judgment tasks but only 35.7% of tutoring and 32.7% of workflow tasks, showing that knowing pedagogy does not ensure effective teaching or reliable tool use.
- The results show separable capabilities and unsaturated performance, with the best model achieving a 63.8% equal-stage pass rate and often failing to connect learner evidence to correct instructional actions and artifacts.
EduAgentBench is a source-grounded benchmark of 150 quality-controlled tasks for evaluating tutor agents across the full scope of teaching work, spanning pedagogical judgment, situated multi-turn tutoring, and teaching-workflow execution in a Canvas-style learning-management environment (2605.14322). Its central empirical finding is a pronounced gap between knowing pedagogy and enacting it: across eleven complete model runs, Stage 1 (pedagogical judgment) passes 89.6% of model–task pairs, while situated tutoring and workflow execution pass only 35.7% and 32.7%, respectively. The best model, GLM-5.1, reaches only a 63.8% equal-stage pass rate, leaving the benchmark far from saturation.
Motivation and positioning
The authors argue that existing evaluations measure ingredients of teaching but not the conjunction that constitutes teacher-level readiness. Educational QA and exam-style benchmarks (MATH, MMLU, E-EVAL, EduEval, EduBench) test subject-matter competence; tutoring-dialogue resources (MathDial, MathTutorBench, TutorBench, LearnLM evaluations) assess local tutor responses; student-modeling corpora (ASSISTments, EdNet, MOOCCubeX) estimate learner state; and general agent benchmarks (Ï„-bench, TheAgentCompany, Toolathlon) test tool use and state mutation without pedagogical validity. None defines success as completing a teaching episode in which the agent must reason from educational evidence, act under pedagogical constraints, and leave verifiable traces. EduAgentBench targets exactly this missing object.
The capability framework is grounded in established pedagogical theory—pedagogical content knowledge (Shulman; Ball et al.), formative assessment (Black and Wiliam; Stiggins), scaffolding and the zone of proximal development (Wood et al.; Vygotsky), self-regulated learning (Zimmerman), instructional design (Branch), and data literacy (Mandinach and Gummer). From these, the benchmark derives six teacher-work capabilities—Diagnose, Design, Create, Teach, Communicate, and Evaluate—instantiated through three staged surfaces.
Benchmark design
Each of the 150 tasks is a measurement contract specifying four elements: a target educational insight, the evidence that makes the insight recoverable, the runtime surface, and the verifier family that rejects plausible shortcuts. Construction follows a backward-design, insight-first pipeline: authors specify the conclusion a competent instructor should reach, then author prompts with sufficient evidence and deliberate distractors (e.g., the lowest-scoring topic may be a symptom rather than the causal prerequisite).
Stage 1 (pedagogical judgment, 50 tasks) isolates teacher-like reasoning with all evidence packaged in the prompt. Evaluation combines deterministic response checks with natural-language assertions verifying that the explanation uses the intended pedagogical mechanism, blocking fluent but incorrect rationales.
Stage 2 (situated tutoring, 40 tasks) tests trajectory-sensitive interaction. Each task derives from a hard source item (including retained MATH and MMLU-Pro items, with licensing-incompatible sources excluded from the release) and a fine-grained knowledge component, converted into a learner scenario with an explicit misconception, affective stance, and tutoring goal. Turn-level checks score contingency, cognitive-load calibration, and answer-dumping avoidance; trajectory-level checks score diagnosis, scaffolding, transfer of responsibility, and adaptation. A weak-model transfer probe—gated by a cold pre-test to ensure a genuine learning opportunity exists—provides an auxiliary cognitive-transfer signal. Tasks with unclear licensing are excluded from the official release, and the released artifact contains transformed scenarios and verifiers rather than raw source copies.
Stage 3 (teaching workflows, 60 tasks) runs in a Canvas-style environment with students, submissions, grades, quizzes, messages, and analytics. Evaluation stacks environment and goal-state assertions, process constraints enforcing evidence-before-action dependencies, natural-language assertions, and artifact rubrics with a 0.85 content-quality threshold for artifact-heavy tasks. This stack explicitly rejects tool-only success: an agent may post a message or create a file and still fail if it used the wrong evidence or missed the target knowledge component.
Quality control audits insight–verifier alignment rather than model scores, including static checks, oracle solvability, and human review for prompt leakage, false process failures, and degenerate outputs.
Main results
The primary statistic is equal-stage aggregation, so the 50/40/60 task mix does not over-weight the larger workflow surface. Headline results for complete model runs:
| Model |
Eq. pass |
Eq. reward |
Judgment |
Tutoring |
Workflow |
| GLM-5.1 |
63.8% |
0.818 |
94.0% |
52.5% |
45.0% |
| Gemini-3.1-Pro |
61.8% |
0.827 |
98.0% |
52.5% |
35.0% |
| GPT-5.5 |
58.7% |
0.827 |
96.0% |
35.0% |
45.0% |
| Qwen3.6-Plus |
57.1% |
0.797 |
92.0% |
47.5% |
31.7% |
| GPT-5.4 |
56.8% |
0.791 |
98.0% |
32.5% |
40.0% |
| DeepSeek-V4 |
55.7% |
0.816 |
92.0% |
40.0% |
35.0% |
| GPT-4.1 |
47.3% |
0.742 |
76.0% |
37.5% |
28.3% |
| MiniMax-M1-27B |
46.6% |
0.721 |
88.0% |
20.0% |
31.7% |
| Gemini-3.1-Flash-Lite |
46.3% |
0.711 |
78.0% |
37.5% |
23.3% |
| Qwen3 |
42.9% |
0.716 |
88.0% |
12.5% |
28.3% |
| GPT-5.1 |
42.6% |
0.703 |
86.0% |
25.0% |
16.7% |
Three findings stand out. First, no model exceeds two-thirds equal-stage pass rate, confirming the benchmark is unsaturated. Second, stage profiles diverge sharply across models: Gemini-3.1-Pro and GPT-5.4 reach 98.0% on judgment but only 35.0% and 40.0% on workflows; Qwen3 passes 88.0% of judgment tasks but 12.5% of tutoring tasks; GPT-5.1 retains 86.0% on judgment yet drops to 16.7% on workflows. If educational agency were a single latent capability, stage ranks would be stable; instead, knowing, teaching, and acting separate. Third, the failure modes differ by stage: workflow pass rate (32.7%) is low relative to mean reward (0.704), indicating partial progress on evidence retrieval, feedback drafting, or state updates before an essential condition is missed, whereas tutoring shows low pass rate and low reward (0.643), suggesting breakdowns in scaffolding and learner-state adaptation rather than a single missing gate.
Task-level calibration is reasonable: 27 all-pass tasks, 30 no-pass tasks, and 52 high-divergence tasks (reward spread ≥ 0.5). The authors note that a naive global threshold of R≥0.70 would over-count 14–25 extra tasks per model, converting "models make partial progress but miss essential constraints" into an overly optimistic workflow story—an important methodological caution for aggregate-reward reporting.
Case study: the evidence-to-action chain
The MM-04 workflow task illustrates the benchmark's diagnostic granularity. The agent must analyze a 120-attempt ECON101 midterm, compute weak knowledge-component rates, edit the assigned Week 5 slide deck (not create a substitute), create a targeted follow-up quiz, notify an advisor, and post a privacy-preserving announcement. GPT-5.5 closes the evidence-to-artifact loop; GLM-5.1 produces superficially useful material but fails the central institutional constraint by leaving the assigned deck unchanged and generating a generic quiz; Gemini-3.1-Pro produces fluent content but misses required numeric evidence and artifact fields. The case demonstrates that high partial rewards can coexist with a broken chain from educational evidence to pedagogical decision to institutional action.
Limitations
The authors are explicit that EduAgentBench is a simulation benchmark, not deployment certification. Mock learners, judge models, weak-model transfer probes, and simulated course environments expose structured failure modes but do not substitute for controlled studies with real students, instructors, and longitudinal learning outcomes. The task set underrepresents classroom orchestration, multimodal tutoring, institution-specific policy, and long-horizon curriculum planning. The evaluation stack inherits proxy limitations: natural-language judges and artifact rubrics can mis-score edge cases, and weak-model gains are auxiliary evidence for transfer rather than direct human learning measurements. Mitigations—deterministic checks where possible, opportunity-gated weak-model scoring, trajectory audits, and release provenance—reduce but do not eliminate these risks.
Conclusion
EduAgentBench provides the first theory-grounded, source-safe benchmark measuring tutor agents as complete teaching systems rather than as answer generators or tool callers. Its evaluation of frontier models establishes that bounded pedagogical judgment is largely solved (89.6% aggregate pass rate) while situated tutoring and evidence-grounded workflow execution remain open (35.7% and 32.7%), and that these capabilities are separable rather than scalar. The benchmark's open question is whether training and evaluation methods can be built to optimize the full evidence-to-teaching-action loop—maintaining educational state, binding claims to source evidence, updating the correct artifacts, and calibrating help—rather than isolated responses.