DeanLLMs: Automated Feedback Evaluators
- DeanLLMs are large language models that automatically evaluate and screen LLM-generated educational feedback based on quality, safety, and alignment with learning objectives.
- They implement a 16-dimension framework—covering content, effectiveness, and hallucination detection—to provide structured and reliable feedback assessment.
- Empirical findings reveal that prompt strategies, fine-tuning, and ensemble approaches significantly influence performance, bias reduction, and overall robustness.
LLM Feedback Evaluators, commonly termed “DeanLLMs,” are LLMs used as automated evaluators of educational feedback generated by other LLMs. In the formulation introduced in “Dean of LLM Tutors: Exploring Comprehensive and Automated Evaluation of LLM-generated Educational Feedback via LLM Feedback Evaluators,” they function as pre-delivery “deans” that screen tutor outputs for low quality and hallucinations, reject problematic feedback, and optionally support an iterative regeneration loop before anything reaches students (Qian et al., 8 Aug 2025). Within the broader LLM-as-a-judge literature, DeanLLMs occupy the educationally specific intersection of feedback quality assessment, pedagogical alignment, and safety auditing.
1. Definition and system role
DeanLLMs are defined as models that “automatically and comprehensively evaluate the quality and safety of educational feedback generated by LLM tutors before delivery to students” (Qian et al., 8 Aug 2025). Their stated goals are pre-delivery evaluation, rejection of low-quality or hallucinated feedback, and support for scalable, reliable feedback delivery in settings where direct human review is costly or infeasible. In the pipeline described in the same work, an LLM tutor first generates feedback from an assignment description and student submission; a DeanLLM then scores that feedback; low-quality or hallucinated outputs are rejected; and the tutor may be prompted to regenerate improved feedback before final delivery (Qian et al., 8 Aug 2025).
This placement distinguishes DeanLLMs from systems that generate feedback directly. Related studies on LLM tutoring in statistics likewise show why such screening is needed: in one mock-exam deployment using GPT-4-0613, approximately of 2,389 feedback instances contained errors, including both technical inconsistencies and conceptually misleading explanations (Herklotz et al., 6 Nov 2025). A plausible implication is that educational deployment increasingly requires a two-model architecture in which one model generates feedback and another audits it.
2. Pedagogical dimensions and evaluation frameworks
The central contribution of the original DeanLLM paper is a 16-dimension evaluation framework comprising feedback content, feedback effectiveness, and hallucination detection (Qian et al., 8 Aug 2025). Content dimensions are graded on a 3-point Likert scale, whereas effectiveness and hallucination dimensions are labeled in binary form. The paper states that there are six content dimensions, seven effectiveness dimensions, and three hallucination types, although five content dimensions are enumerated in detail in the available description (Qian et al., 8 Aug 2025).
| Group | Components | Label type |
|---|---|---|
| Content | Alignment with goals, Specificity, Motivational Tone, Strength, Weakness | 3-point Likert |
| Effectiveness | Feed forward, Feed up, Feed back, task, process, self-regulation, self | Binary |
| Hallucinations | Input-conflicting, Context-conflicting, Fact-conflicting | Binary |
The content dimensions emphasize whether comments are aligned to specific learning goals, are detailed and actionable, maintain a positive and respectful tone, and identify strengths and weaknesses of the submission (Qian et al., 8 Aug 2025). The effectiveness dimensions draw on familiar feedback theory: feed up concerns clarification of goals, feed back concerns comments on progress, and feed forward concerns next steps; the framework also encodes Hattie-and-Timperley-style levels of feedback on task, process, self-regulation, and self (Qian et al., 8 Aug 2025).
This pedagogical orientation is consistent with adjacent educational evaluation work. In experimentation-protocol feedback, LLM-generated feedback was evaluated using six criteria—Feed Up, Feed Back, Feed Forward, Constructive Tone, Linguistic Clarity, and Technical Terminology—and showed no significant difference from teachers and science education experts in overall quality, but a significant disadvantage in Feed Back, the dimension that requires contextual identification and explanation of errors (Seßler et al., 18 Feb 2025). In statistics education, another feedback framework decomposed task feedback into right/wrong, response-oriented, and conceptually-focused subtypes, alongside process, self-regulatory, and self-level feedback, and found that LLM-generated feedback was dominated by response-oriented explanations while deeper conceptual and self-regulatory guidance remained sparse (Herklotz et al., 6 Nov 2025). Together, these studies suggest that DeanLLMs are not merely correctness checkers; they are evaluators of pedagogical form, depth, and instructional usefulness.
3. Data construction, labeling, and model training
The first DeanLLM study built its evaluator dataset by starting from 1,000 real student submissions and assignment descriptions spanning 155 assignments across 59 computer science courses, then synthesizing a privacy-preserving parallel corpus with o4-mini (Qian et al., 8 Aug 2025). For evaluator development, the authors selected 200 synthetic assignment submissions and descriptions covering 85 assignments from 43 computer science courses, then asked eight commercial LLM tutors to generate feedback, producing 25 feedback instances per tutor and 200 instances in total (Qian et al., 8 Aug 2025).
Three researchers with expertise in LLM-generated educational feedback and feedback effectiveness performed the annotation. They first developed a consensus rubric, conducted an inter-rater reliability test on 16 instances, and reported Fleiss’ Kappa , after which the remaining instances were divided across coders (Qian et al., 8 Aug 2025). The annotation protocol also included 45 written explanations for difficult cases, especially hallucinations, and these explanations were later reused to explore explanatory fine-tuning (Qian et al., 8 Aug 2025).
Four OpenAI models were then evaluated as DeanLLMs: GPT-4.1, o4-mini, o3, and o3-pro (Qian et al., 8 Aug 2025). Zero-shot prompting used the assignment description, student submission, rubric, and restricted output formatting. Few-shot prompting added rubric examples with explanations. Fine-tuning was explored only for GPT-4.1, using three curricula: plain-labeled instances, explanatory instances, and a mixed curriculum combining both (Qian et al., 8 Aug 2025).
The reported results differentiate sharply between prompting and fine-tuning regimes. In zero-shot labeling, o3-pro performed best overall with Accuracy and F1 . In few-shot labeling, o4-mini performed best overall with Accuracy and F1 . Fine-tuned GPT-4.1 trained on plain-labeled instances reached Accuracy and F1 , while human coders averaged Accuracy and F1 (Qian et al., 8 Aug 2025). By contrast, training on explanatory data alone, or mixing explanatory with plain-labeled data, reduced performance (Qian et al., 8 Aug 2025). This suggests that label supervision was more useful than rationale augmentation under the training setup reported.
4. Empirical performance and cross-model findings
The original study did not stop at evaluator benchmarking. After identifying the best DeanLLM configuration, it used that evaluator to score 2,000 feedback instances produced by 10 commercial LLMs, 200 per model, split between synthetic submissions and corresponding original real assignments (Qian et al., 8 Aug 2025). For cross-model comparisons, the paper used the mean Likert score across all feedback quality dimensions except hallucinations as “overall feedback quality,” and it did not define an explicit weighted composite beyond those mean scores (Qian et al., 8 Aug 2025).
On that benchmark, Gemini 2.5 Pro obtained the highest overall feedback quality mean at 1.09, with GPT-4.1 close behind at 1.08; the figure’s 0 confidence intervals indicated that this difference was not significant (Qian et al., 8 Aug 2025). Gemini 2.0 Flash and Gemini 2.0 Flash-Lite performed substantially worse, with the Lite variant reported as significantly worse than nine other models in overall quality, effectiveness, and content (Qian et al., 8 Aug 2025). Hallucination behavior was more sharply differentiated: GPT-4.1 nano showed the highest hallucination detection rate at 1 average across the three hallucination types, while Gemini 2.5 Pro showed zero hallucination issues across its 200 feedback instances (Qian et al., 8 Aug 2025).
A broader educational pattern appears in neighboring studies. In statistical education, prompt engineering alone sometimes outperformed heavier adaptation pipelines: zero-shot prompting achieved the strongest balance between quality and cost, while fine-tuning with LoRA required substantially more resources without clear advantages (Ippisch et al., 10 Nov 2025). In essay feedback selection, FeedEval used three dimension-specialized evaluators—specificity, helpfulness, and validity—and showed that feedback filtered by those evaluators improved downstream essay scoring and revision quality relative to low-quality feedback (Chu et al., 8 Jan 2026). These results suggest that DeanLLM performance depends not only on base model choice, but also on whether evaluation is decomposed into pedagogically coherent subproblems.
5. Reliability, bias, and consistency
Because DeanLLMs are themselves LLM judges, they inherit the general reliability problems of LLM-based evaluation. One major concern is pairwise bias. “The Comparative Trap” showed that pairwise comparison amplifies undesirable preferences such as verbosity and authoritative tone, and introduced PRePair, a protocol that first performs pointwise analyses and then makes a pairwise decision; on LLMBar-Adversarial, GPT-3.5 improved from 36.05 accuracy under ordinary pairwise judging to 54.23 under PRePair (Jeong et al., 2024). Position bias is a related failure mode: PORTIA improved consistency rates across six evaluator models with an average relative improvement of 2, raised GPT-4 consistency to as high as 3, and enabled a PORTIA-enhanced GPT-3.5 evaluator to reach 4 agreement with GPT-4 at roughly 5 of the cost (Li et al., 2023).
A second concern is evaluator consistency. A dedicated study of evaluator reliability distinguished Self-Consistency and Inter-scale Consistency and found that strong proprietary models were “not necessarily consistent evaluators” (Lee et al., 2024). Interval scales were more consistent than Likert or binary scales, longer criterion definitions improved consistency modestly, and lower temperatures substantially increased self-consistency for some models (Lee et al., 2024). This is directly relevant to DeanLLM deployment, because the original DeanLLM framework mixes Likert-style and binary judgments across dimensions (Qian et al., 8 Aug 2025).
A third concern is systematic optimism. Work on agreeableness bias reported that LLM judges can reach a True Positive Rate of about 6 while remaining below 7 True Negative Rate on invalid outputs, which makes naive accuracy misleading under class imbalance (Jain et al., 13 Oct 2025). The same paper showed that majority-vote ensembles are insufficient, whereas a minority-veto strategy and a regression-based calibration model materially improved invalid-output detection and reduced maximum absolute error in generator-quality estimation to about 8 with a small human-annotated calibration set (Jain et al., 13 Oct 2025). For pairwise win-rate estimation, Bayesian calibration methods such as Bayesian Win Rate Sampling and Bayesian Dawid–Skene were proposed specifically because naive LLM win rates are biased by evaluator-specific error profiles (Gao et al., 2024).
These findings imply that a DeanLLM should be evaluated not only by Accuracy and F1 against human labels, but also by bias resistance, consistency across repeated judgments, robustness to missing outputs, and calibration of accept/reject decisions.
6. Architectural variants and related evaluator paradigms
The single-model DeanLLM architecture in (Qian et al., 8 Aug 2025) is one instantiation of a broader design space. A complementary strategy is to use multiple specialized evaluators rather than one omnibus judge. AIME, developed for code-generation optimization, decomposed evaluation into syntax, logic, correctness, readability, runtime, and redundancy, and showed that multiple role-specific evaluators could improve error-detection rate by up to 9 and success rate by up to 0 relative to a single-evaluator protocol (Patel et al., 2024). Although that work is not educational, it provides a clear analogue to dimension-specialized DeanLLMs.
FeedEval offers an educational example of that decomposition. It trains separate evaluators for specificity, helpfulness, and validity, normalizes each dimension’s scores across candidate feedbacks, then combines them by equal-weight averaging to select the best feedback (Chu et al., 8 Jan 2026). The framework reported stronger alignment with expert judgments than generic LLM-as-a-judge baselines and improved downstream essay-scoring performance when used to filter feedback (Chu et al., 8 Jan 2026). This suggests that the future of DeanLLMs may lie less in one monolithic evaluator than in ensembles or committees of dimension-specialized judges.
User-facing evaluation interfaces also matter. EvalAssist compared direct assessment with pairwise comparison in a study of machine-learning practitioners and found that users completed more evaluations with direct assessment, while rubric-style explanations were rated more helpful than pairwise explanations (Ashktorab et al., 2024). A plausible implication is that DeanLLM outputs may be more usable when they expose rubric-grounded dimension scores and evidence, rather than only a global accept/reject verdict.
7. Limitations and future directions
The present DeanLLM literature remains early and domain-bounded. The founding DeanLLM study was built on synthetic computer-science assignments and feedback, not on a live multi-institutional deployment, and it explicitly notes unresolved issues such as the under-specified sixth content dimension, prompt sensitivity across models, and the absence of explicit accept/reject thresholds in the deployed screening loop (Qian et al., 8 Aug 2025). Explanatory fine-tuning unexpectedly reduced performance in that setting, indicating that rationale-augmented training remains methodologically unsettled (Qian et al., 8 Aug 2025).
Generalization across domains, languages, and feedback genres is also open. Multilingual evaluator studies show that LLM-based evaluators correlate more strongly with human judgments in high-resource languages than in low-resource languages, and that fine-tuning on one language can improve performance across others, though not uniformly (Chang et al., 6 Mar 2025). Educational studies in statistics and science inquiry further show that LLM feedback remains strongest on correctness judgments and weakest on contextual error explanation, conceptual enrichment, and self-regulation guidance (Seßler et al., 18 Feb 2025, Ippisch et al., 10 Nov 2025). This suggests that future DeanLLMs will need stronger support for multilingual calibration, discipline-specific rubric grounding, and deeper pedagogical dimensions beyond correctness and tone.
A final trajectory is closed-loop governance. The original DeanLLM paper already frames the evaluator as a pre-delivery screen that can trigger tutor regeneration (Qian et al., 8 Aug 2025). Subsequent work points toward richer versions of that loop: pointwise-first comparison to reduce evaluator bias (Jeong et al., 2024), specialist evaluator committees (Patel et al., 2024, Chu et al., 8 Jan 2026), and Bayesian or regression-based calibration layers to correct systematic judge error (Gao et al., 2024, Jain et al., 13 Oct 2025). Taken together, these developments suggest that DeanLLMs are evolving from simple quality filters into a distinct research area concerned with pedagogically grounded, safety-aware, and statistically calibrated governance of AI-generated educational feedback.