StudentBench: AI and human tutoring yield equivalent GRE learning gains

Abstract: Artificial intelligence offers an unprecedented opportunity to augment human capabilities, yet progress at the frontier has focused primarily on advancing model capabilities. We introduce StudentBench, a suite of AI teaching evaluations and a public platform that enables large-scale data collection with over 175,000 student-AI messages to study whether LLMs produce learning gains equivalent to human tutoring. Using StudentBench, we measured learning gains on Quantitative and Verbal GRE questions across 2,383 human participants receiving AI tutoring, human tutoring, or no tutoring. We establish that AI tutoring is statistically equivalent to expert human tutoring for GRE learning gains (p = .015), and in five of the seven GRE domains, the best performing AI tutor surpassed the human tutor, on average. In a second study, expert human tutors compared LLM-generated lesson plans and practice problems through 2,028 pairwise rubric evaluations. Together, the two studies clearly separate AI tutors across: (1) lesson planning, (2) practice-problem creation, (3) conversational pedagogy, (4) cost, and (5) engagement. Surprisingly, one AI tutor achieved learning gains equivalent to human tutoring (p = .044) at 918 times lower cost (USD 0.0052 for AI versus USD 4.81 for human, per percentage point gained). For Quantitative GRE sessions, faster AI replies correlated with more student messages, more messages with more correct practice, and more correct practice with larger learning gains (all p < .002). The StudentBench platform is freely available at https://studentbench.org.

• The paper compares AI tutoring, expert human tutoring, and a no-tutoring control across 2,383 participants and 2,469 analyzed sessions, finding statistically equivalent learning gains for AI and human tutoring.
• AI tutoring achieved significant learning gains in both Quantitative and Verbal sections, with a pooled difference of -0.58 percentage points compared to human tutoring and a 90% confidence interval of [-2.18, 1.03].
• Gemma 4 31B, an AI tutor, demonstrated equivalent learning gains to human tutoring with $918 times$ lower cost per percentage point gained

Research question and contribution

“StudentBench: AI and human tutoring yield equivalent GRE learning gains” (2609.28470) evaluates whether general-purpose LLMs can produce learning gains comparable to expert human tutoring under controlled, one-hour instructional sessions. Rather than treating model competence or response quality as proxies for educational value, the study measures unaided performance on post-tests designed to parallel the GRE. Its central claim is not merely that AI can answer educational questions, but that AI tutoring can improve students’ independent performance at a level statistically equivalent to expert tutoring.

The paper frames this result as evidence for the first condition of “recursive human self-improvement” (RHSI): a technology must improve human capabilities. This framing is conceptually broader than the experiment itself. The data establish immediate tutoring-related gains on newly authored GRE assessments; they do not establish the subsequent RHSI conditions, such as whether improved students use AI more effectively or whether that interaction produces continued gains.

StudentBench contributes three principal components. First, it presents a randomized comparison of AI tutoring, expert human tutoring, and a no-tutoring control across 2,383 participants and 2,469 analyzed sessions. Second, it introduces expert-review and transcript-based evaluations that separate lesson planning, practice-problem generation, and conversational pedagogy. Third, it reports cost, latency, engagement, and practice metrics alongside learning outcomes, producing a benchmark intended to evaluate AI as a system for augmenting human performance rather than as an isolated LLM.

Experimental design and measurement

Participants were recruited through Handshake and were predominantly young adults, although the full age range was 18–67. They completed either a Quantitative or Verbal GRE session. Each session consisted of a 27-question Quantitative or 41-minute Verbal pre-test, approximately five minutes reviewing pre-test errors, one hour in an assigned tutoring condition, and a matched post-test. The assessment forms were counterbalanced: approximately half of participants received form P followed by Q, and the remainder received Q followed by P. The questions were newly written by former ETS and Kaplan exam creators and were designed to match prior GRE assessments in difficulty, content coverage, format, and sequencing. Avoiding publicly available legacy GRE questions was intended to reduce contamination from LLM pretraining.

The AI condition included 13 tutors corresponding to 12 models, with some models evaluated at different reasoning settings. The systems received the student’s complete pre-test record, including questions, answers, correctness, response times, and expert difficulty ratings. They generated a lesson plan, selected concepts and time allocations, created practice problems and answer keys, and conducted the interactive tutoring session. The prompts deliberately supplied minimal scaffolding, leaving the models responsible for diagnosis, sequencing, problem generation, and conversational instruction.

Human tutors were former ETS or Kaplan employees or had at least five years of GRE tutoring experience. They taught through live video calls and received the same pre-test information, but neither human nor AI tutors saw the post-test. The control group reviewed its graded pre-test and watched educational videos unrelated to the GRE. The absence of a practice-only control means that the experiment estimates the effect of the assigned tutoring conditions relative to this control, not the incremental effect of tutoring interaction relative to independent GRE practice.

Learning gain was defined as post-test percentage correct minus pre-test percentage correct. The primary analyses used ANCOVA, adjusting for baseline score, its quadratic term, assessment form, and section where appropriate. AI–human comparisons used CR2 cluster-robust covariance estimates and Satterthwaite degrees of freedom to account for repeated participants and students sharing human tutors. Equivalence was evaluated using TOST with bounds of $\pm 0.25$ pooled standard deviations, corresponding to approximately $\pm 4.09$ percentage points in the combined analysis.

Learning outcomes

The primary result is that pooled AI tutoring was statistically equivalent to expert human tutoring across Quantitative and Verbal sessions. The adjusted AI-minus-human difference was $-0.58$ percentage points, with a 90% confidence interval of $[-2.18, 1.03]$. This interval fell within the prespecified equivalence bounds, yielding $p=.015$. Equivalence also held under a stricter $\pm 0.20$-standard-deviation margin, with $p=.023$.

The result is substantively important because AI tutoring did not merely outperform the no-tutoring condition; it reached the performance level of a strong human benchmark. Relative to control, AI tutoring increased learning gain by 6.15 percentage points in the combined sample, with a 95% confidence interval of $[4.08, 8.21]$ and $p<.001$. The section-specific adjusted differences were 6.86 percentage points in Quantitative and 5.47 percentage points in Verbal. Human tutoring also exceeded control by 7.04 percentage points overall. Thus, the equivalence conclusion is consistent with both interventions producing substantial gains over the control condition, rather than with neither intervention being effective.

The section-level pattern is asymmetric. AI and human tutoring were equivalent in Quantitative, with an AI-minus-human difference of 0.88 percentage points and TOST $p=.028$. Equivalence was not established in Verbal: the AI-minus-human estimate was $\pm 4.09$0 percentage points, with a 90% confidence interval of $\pm 4.09$1 and $\pm 4.09$2 under the stated margin. Human tutoring therefore retained a possible advantage in Verbal, although the study could not establish a statistically meaningful difference between sections in the human–AI gap.

The domain-level results show specialization rather than a uniformly dominant tutor. The best-performing AI tutor exceeded the human mean in five of seven domains, while human tutoring remained higher in algebra and sentence equivalence. Different models led different domains: Google models led data analysis, geometry, and reading comprehension; GPT-5.5 Pro led arithmetic; Kimi K2.6 led algebra; and Anthropic models led sentence equivalence and text completion. These domain leaderboards are descriptive and should not be interpreted as definitive model rankings for subgroups, since many domain-specific cells contain relatively small samples and the selected frontier model differs across domains.

The combined AI-tutor comparison did not detect statistically significant differences among the 12 shared tutors ($\pm 4.09$3). This does not imply practical equality among models. Adjusted mean learning gains ranged from 12.1 to 16.0 percentage points, compared with 14.9 for human tutoring and 7.8 for control. Rather, the study had insufficient evidence to distinguish the tutors under its omnibus specification. The distinction matters because the paper simultaneously reports strong separation on other dimensions, particularly cost, latency, expert preferences, and engagement.

Teaching capability evaluations

StudentBench separates several capabilities that are often conflated in evaluations of AI tutors. In the second study, 51 expert GRE tutors conducted 2,028 pairwise evaluations of AI-generated lesson plans and practice problems derived from 381 pre-tests. Bradley–Terry models aggregated preferences over concept relevance, grouping, prioritization, time allocation, test-taking strategy, practice alignment, difficulty, and example accuracy.

Anthropic models, particularly Opus variants, were preferred by experts for lesson planning and practice-problem creation. These rankings demonstrate that expert judgments distinguish model families and model configurations. They also show that high-quality instructional artifacts are not equivalent to high measured learning gains. The paper explicitly treats these leaderboards as evaluations of teaching materials and observed behaviors, not as direct estimates of student learning. Indeed, the model with the strongest expert-preference profile need not be the model with the largest post-test gain.

The practice-problem analyses provide a limited external check on expert judgments. In Quantitative tutoring, models preferred by experts for practice design tended to receive fewer student answer disputes; the combined Spearman correlation was $\pm 4.09$4, $\pm 4.09$5. For example accuracy, the combined correlation was $\pm 4.09$6, $\pm 4.09$7. These associations are exploratory, based on a small number of model-level observations, and the dispute button recorded student disagreement rather than independently verified errors. They therefore support alignment between some expert and behavioral indicators without validating either as a substitute for learning outcomes.

The conversational-pedagogy leaderboard operationalized six indicators: scaffolding cues, requests for explanations, interactive questions, early requests to attempt problems, long solutions after student replies, and reasoning checks. AI tutors clustered by provider family, with distinct behavioral profiles across model families. This suggests that provider-level training or prompting practices may influence tutoring style. However, the authors acknowledge two substantial limitations: the indicators were not validated as predictors of learning, and human tutoring was spoken video interaction whereas AI tutoring was typed chat. Consequently, differences in indicator prevalence cannot be interpreted straightforwardly as differences in pedagogical quality.

Cost, latency, and engagement

The cost analysis is one of the paper’s strongest quantitative claims. AI inference costs included lesson planning, practice-problem generation, and interactive tutoring. Human tutoring was valued at a reference price of $\pm 4.09$80.067 for Gemma 4 31B to $21.24 for GPT-5.5 Pro.

Gemma 4 31B produced a mean learning gain of 12.742 percentage points across 159 sessions, compared with 15.608 percentage points for human tutoring. Its AI-minus-human difference was $\pm 4.09$9 percentage points, with a 90% confidence interval of $-0.58$0 and an equivalence-test $-0.58$1. Dividing mean session cost by mean learning gain yielded $-0.58$24.80508 for the human reference. The resulting ratio was approximately 918 times lower cost per percentage point of learning gain.

This comparison should be interpreted narrowly. The numerator for humans is a market-rate assumption rather than an experimentally observed labor cost, while AI costs reflect inference and reconstructed token charges. The denominator is an immediate post-test gain, not retained learning, credential attainment, or broader educational benefit. In addition, the individual-tutor equivalence tests were conducted without correction for multiple comparisons across tutors. The result is therefore a striking cost-efficiency estimate within the study’s operational definition, but not a general estimate of the total cost of educational deployment.

The Pareto analysis further indicates that cost and latency are not monotonically related to learning gain. Gemini 3.5 Flash achieved a similar learning gain to GPT-5.5 Pro at approximately 20 times lower cost. Gemini 3.1 Pro had higher mean gain, lower cost, and faster replies than GPT-5.5 Pro, thereby Pareto-dominating it on the sample estimates. Because frontier membership was based on observed means without uncertainty intervals, these comparisons describe the empirical trade-off surface rather than statistically confirmed dominance.

Reply latency was strongly associated with engagement at the tutor level. Across 12 shared tutors, faster replies correlated with more student messages, with Spearman $-0.58$3 and $-0.58$4. Reply times ranged from 1.88 seconds for GPT-5.4 mini to 30.99 seconds for GPT-5.5 Pro. The finding implies that responsiveness may affect learning indirectly by changing the amount of student activity that a tutoring session elicits.

Practice as a potential mechanism

The session-level exploratory analyses connect latency, engagement, practice, and learning in Quantitative tutoring. Among 1,137 Quantitative AI sessions, every link in the reported chain was statistically significant after the paper’s multiple-comparison procedure. Ten additional seconds of reply time were associated with 5.19 fewer student messages, with a 95% confidence interval of $-0.58$5 and $-0.58$6. Ten additional student messages were associated with 0.52 more correct practice problems, with a 95% confidence interval of $-0.58$7 and $-0.58$8. Ten additional correct practice problems were associated with a 5.62-percentage-point larger learning gain, with a 95% confidence interval of $-0.58$9 and $[-2.18, 1.03]$0.

The implication is that response latency may matter less as a direct property of model quality than as a determinant of interaction volume and productive practice. This mechanism could help explain why models with weaker expert-evaluated pedagogical profiles nonetheless achieved human-level learning gains: a fast tutor may induce more attempts and more feedback cycles within the fixed one-hour treatment.

The mechanism did not generalize cleanly to Verbal tutoring. The corresponding associations were not statistically significant under the reported analysis: reply time and student messages had $[-2.18, 1.03]$1, student messages and correct practice had $[-2.18, 1.03]$2, and correct practice and learning gain had $[-2.18, 1.03]$3. Verbal participants sent fewer but longer messages, so message count may be a poorer engagement measure in that section. The contrast emphasizes that engagement metrics are construct-dependent and that the Quantitative pathway should not be treated as a general theory of AI tutoring effectiveness.

Methodological strengths and limitations

The design has several notable strengths. It measures unaided transfer rather than performance during AI assistance, uses counterbalanced assessment forms, withholds post-test content from tutors, recruits expert human comparators, and reports both effectiveness and operational cost. The scale of the study is also substantial for a live tutoring experiment, with 176,614 student–AI messages and 45,463 answered practice problems. The public platform, released data, and reproducibility code make the benchmark unusually inspectable for this class of study.

The principal limitation is that learning was measured immediately after a single one-hour session. The results do not establish retention over weeks or months, cumulative effects from repeated tutoring, or transfer beyond GRE-like items. This is consequential for the cost-per-gain claim: a low cost for an immediate gain may not remain low when durable learning requires spaced practice or repeated intervention.

The population was restricted to adults who could read and write in English, had access to Handshake, and participated in a paid study. The findings therefore do not directly establish performance across languages, younger learners, educational institutions, different devices, or students with different motivational and socioeconomic profiles. The study also did not include a self-study practice condition, leaving the added value of interactive AI tutoring relative to independent problem solving unresolved.

Inference about human equivalence is particularly sensitive to the human comparison structure. The combined analysis retained equivalence after several checks, including excluding participants who completed both sections and omitting human tutors individually. However, the effective degrees of freedom for the principal pooled comparison were only 1.59 because human sessions were concentrated among a small number of tutor clusters. The authors also report that a gain-only sensitivity analysis without the principal covariate and assessment-form adjustments did not establish equivalence in either section. Thus, the central conclusion is supported by the prespecified adjusted model but depends materially on its handling of baseline score, assessment form, clustering, and repeated observations.

Finally, the frontier and individual-model claims should be distinguished from the pooled result. Pooled equivalence supports the claim that the evaluated AI-tutor mixture was comparable to the human benchmark. Six individual AI tutors passed uncorrected equivalence tests, but the study did not correct these tests for multiple comparisons. The evidence is consequently strongest for the pooled AI condition and weaker for claims about any particular model, domain, or subgroup.

Conclusion

StudentBench provides evidence that minimally scaffolded LLM tutors can produce immediate GRE learning gains statistically equivalent to expert human tutoring in a large controlled study. The pooled AI-minus-human difference was $[-2.18, 1.03]$4 percentage points, and AI exceeded the no-tutoring control by 6.15 percentage points overall. The result was stronger for Quantitative than for Verbal, while domain-level analyses indicated substantial specialization among models.

The paper’s second contribution is methodological: it evaluates tutoring as a multidimensional system involving learning outcomes, planning, practice generation, dialogue, cost, latency, and engagement. The most consequential economic result is that Gemma 4 31B achieved human-equivalent gains under the study’s equivalence test at approximately 918 times lower cost per percentage point gained. This estimate is bounded by immediate post-test measurement, assumed human pricing, model-specific inference accounting, and uncorrected individual-tutor comparisons.

The principal unresolved question is whether these short-term, GRE-specific gains persist and accumulate under repeated use, particularly across Verbal reasoning, independent practice conditions, languages, and educational settings not represented in the sample.