Human-Aligned LLM Grading Workflow

• Human-Aligned LLM-Assisted Grading Workflow is a human-in-the-loop system that uses LLMs for bounded grading tasks while preserving expert control over final assessments.
• The workflow decomposes grading into stages such as rubric design, LLM draft scoring, and human verification, enabling controlled automation and reliable calibration.
• Empirical evidence indicates enhanced grading consistency, focused feedback quality, and improved efficiency through selective LLM deployment and human intervention.

A human-aligned LLM-assisted grading workflow is a human-in-the-loop assessment pipeline in which LLMs are used for bounded grading functions—such as rubric application, first-draft feedback, triage, benchmark comparison, confidence estimation, or rubric refinement—while humans retain authority over the scoring philosophy, benchmark cases, final grades, and escalation rules. In the recent literature, this workflow appears not as a single architecture but as a family of designs for proof grading, short-answer scoring, project-report assessment, essay evaluation, and handwritten mathematics, with alignment operationalized through agreement with expert labels, controllability, calibration, and explicit preservation of human oversight (Mahinpei et al., 27 Feb 2026, Li et al., 7 Apr 2025, Raikote et al., 12 Mar 2026).

1. Workflow structure and task decomposition

Across domains, the workflow is consistently decomposed into separable stages: human rubric construction; preparation of reference context such as instructor solutions, key concepts, report requirements, or paper summaries; LLM-based scoring or feedback generation; automated checks for inconsistency or uncertainty; and human verification or rubric revision. This decomposition is central because the literature repeatedly distinguishes between tasks that are predominantly evaluative and normative, and tasks that are explanatory, clerical, or triage-oriented.

Workflow archetype	Human-controlled components	LLM-controlled components
Proof-based course workflow (Mahinpei et al., 27 Feb 2026)	Final grading judgment, feedback curation	Rubric attempt, feedback draft
GradeHITL (Li et al., 7 Apr 2025)	Core rubric, answers to targeted questions	Grading, inquiring, rubric optimization
CHiL(L)Grader (Raikote et al., 12 Mar 2026)	Low-confidence grading, correction data	Score prediction, calibrated confidence, routing
CoGrader (Chen et al., 28 Jul 2025)	Metric selection, benchmark choice, final scores	Metric proposals, initial grading, benchmark-based regrading
Handwritten mathematics workflow (Vanhoyweghen et al., 13 Mar 2026)	Solution keys, grading keys, mandatory verification	Five-pass scoring, consistency checks, provisional score

These systems differ in interface and domain, but they share a common architectural principle: the LLM is not treated as a monolithic grader. It is embedded inside a workflow that localizes its authority, exposes intermediate artifacts, and creates points of intervention. This suggests that “human alignment” in grading is less a property of a model in isolation than of the allocation of responsibility across the pipeline.

2. Rubrics, scales, and the operationalization of human standards

The strongest recurrent pattern is that human alignment depends first on how the grading standard is represented. In GradeHITL, the rubric is explicitly structured as $G = \{G_{qs} \| G_{kc} \| G_{sr} \| G_{ar}\}$, where question stem, key concepts, and scoring rubric remain fixed, and only the adaptation rules $G_{ar}$ are iteratively updated (Li et al., 7 Apr 2025). That design anchors the scoring philosophy in human-authored text while allowing machine-oriented clarifications, examples, and threshold cases to accumulate without rewriting the human core.

Rubric wording itself can dominate alignment outcomes. In LLM-based pretest question evaluation, rubric revision had a larger effect on human-machine agreement than rationale-first evaluation: under direct evaluation, openness agreement rose from $55.7\%$ to $83.3\%$, and depth agreement from $70.5\%$ to $95.0\%$, when the rubric was revised to separate constructs from misleading surface cues, specify valid evidence for each level, and make adjacent boundaries explicit (Tseng et al., 22 Jun 2026). The same paper shows that rationale-first prompting and rubric revision are complementary, but not interchangeable.

Scale design also affects alignment. In LLM-as-a-judge experiments across six benchmarks and three continuous scales, pooled human–LLM agreement was highest on the $0\!-\!5$ scale, with panel-to-panel ICC $0.853$ and nMAE $0.111$, compared with $0.805/0.122$ on $G_{ar}$0 and $G_{ar}$1 on $G_{ar}$2 (Li et al., 6 Jan 2026). Human and LLM panels were internally reliable on all scales, which implies that cross-group misalignment is not reducible to noise. This suggests that workflow design must treat score scale as part of rubric engineering rather than as a neutral formatting choice.

Project-report workflows add a further layer: metric co-design. CoGrader has instructors upload project requirements, then lets an LLM propose “Report Objective Metrics,” “Extra Potential Metrics,” and customized metric definitions, after which instructors mark each metric as “Auto Grade” or “Score Reference” (Chen et al., 28 Jul 2025). That mechanism operationalizes a distinction between dimensions the instructor is willing to delegate and dimensions that remain advisory only. In practice, the workflow therefore begins not with model inference but with the explicit partition of assessment criteria by acceptable autonomy.

3. Allocation of authority, confidence routing, and verification

Human-aligned workflows consistently avoid full autonomy. CHiL(L)Grader formalizes this most directly: a fine-tuned grader predicts a grade $G_{ar}$3 and calibrated confidence $G_{ar}$4, and a confidence gate accepts the prediction only if $G_{ar}$5; otherwise the response is routed to a human grader (Raikote et al., 12 Mar 2026). With post-hoc temperature scaling and continual learning, the framework automatically scores $G_{ar}$6 of responses at expert-level quality, defined as $G_{ar}$7, and reports a QWK gap of $G_{ar}$8 between accepted and rejected predictions. The workflow’s logic is therefore selective prediction, not universal automation.

A different but compatible mechanism appears in handwritten mathematics assessment. There, GPT-5.1 grades each handwritten answer five times independently; variance, spread, and anomaly statistics are computed; and the operational provisional grade is the maximum of the five scores, explicitly favoring the student, while final acceptance remains subject to mandatory human verification (Vanhoyweghen et al., 13 Mar 2026). No stable automatic threshold could replace human review, so internal consistency statistics function as attention-directing signals rather than auto-adjudicators.

The proof-based-course study makes the same point at the level of pedagogy rather than uncertainty estimation. It argues that grading is “situated judgment, not rubric execution,” especially when later subproblems depend on earlier ones and partial credit depends on course-specific conventions about omitted steps, forgivable propagated errors, and acceptable deviations from the instructor’s solution (Mahinpei et al., 27 Feb 2026). In that setting, the recommended workflow separates grading and feedback: humans retain final evaluative authority, while LLMs are used for drafting, explaining, and triaging.

CoGrader turns the same principle into interface logic. “Auto Grade” metrics can be scored with relatively little intervention, but “Score Reference” metrics remain explicitly advisory, and instructors retain final authority for metric selection, benchmark choice, score editing, comment editing, and feedback release (Chen et al., 28 Jul 2025). Across these systems, the human-aligned workflow is therefore defined less by whether the LLM produces a grade than by whether the workflow makes that grade contingent, reviewable, and scoped to appropriate task types.

4. Empirical evidence across assessment types

The empirical record is mixed but structured. In proof-based undergraduate grading, overall pooled agreement on Solution Quality was $G_{ar}$9 for GTA–UTA, but only $55.7\%$0 for GTA–LLM and $55.7\%$1 for UTA–LLM; on Writing Quality the corresponding figures were $55.7\%$2, $55.7\%$3, and $55.7\%$4 (Mahinpei et al., 27 Feb 2026). The asymmetry becomes severe on dependent later subproblems: for subproblem E, GTA–UTA SQ agreement was $55.7\%$5, but both GTA–LLM and UTA–LLM fell to $55.7\%$6. Yet the same study found that when all feedback options were available, LLM feedback was ranked first 28 times, ahead of GTA feedback at 25, UTA feedback at 15, and writing from scratch at 13. The evidence therefore supports feedback assistance much more strongly than autonomous proof grading.

In short-answer quizzes and project reports, GPT-4o shows stronger aggregate alignment. In an undergraduate computational linguistics course, quiz-score correlation with human graders reached $55.7\%$7 overall, and exact score agreement occurred in $55.7\%$8 of quiz cases; however, GPT-4o graded lower than humans in $55.7\%$9 of cases and higher in only $83.3\%$0, indicating a conservative bias (Byun et al., 13 Nov 2025). For project reports in the same study, section-level differences were nonsignificant except in Approach and Results, where GPT-4o was systematically more conservative.

Project-report assessment with collaborative benchmarking shows a different strength profile. In CoGrader, benchmarking improved perceived reliability of regraded scores to $83.3\%$1 and comments to $83.3\%$2, and participant grades showed strong convergence with instructor ground truth, with Kendall’s $83.3\%$3, Spearman’s $83.3\%$4, and Pearson’s $83.3\%$5 (Chen et al., 28 Jul 2025). Here the workflow’s gains come from metric co-design and benchmark-based recalibration rather than from one-shot model scoring.

For handwritten mathematics, the hybrid pipeline reduced grading time by approximately $83.3\%$6, with a geometric-mean digital/manual time ratio of $83.3\%$7, while agreement in the digital condition was comparable to or tighter than manual double-marking in most question instances (Vanhoyweghen et al., 13 Mar 2026). For nationwide Estonian essay exams, automated scoring reached performance comparable to human raters and tended to fall within the human scoring range, while producing fine-grained subscore profiles for feedback and moderation (Karjus et al., 22 Jan 2026). This suggests that open-ended assessment is not uniformly hostile to LLM assistance; rather, performance depends on task structure, rubric explicitness, and the degree to which human review is embedded into the workflow.

5. Recurrent failure modes, controversies, and sources of misalignment

The literature converges on the claim that human–LLM disagreements are systematic rather than random. In proof-based grading, the LLM’s divergences are characterized by stricter rubric interpretation, higher expectation of explicit reasoning, and sensitivity to the instructor’s reference solution; humans often disagree because one grader missed an error, whereas LLM–human disagreements more often arise because the LLM notices an error but penalizes it more harshly (Mahinpei et al., 27 Feb 2026). This is a calibration problem, not merely a noise problem.

A related issue is rubric shortcutting. In science-response scoring, LLM-generated analytic rubrics show a measurable alignment gap with human analytic rubrics, and the model can “resort to shortcuts, bypassing deeper logical reasoning expected in human grading” (Wu et al., 2024). High-quality analytic rubrics designed to reflect human grading logic mitigate this gap and improve scoring accuracy. The result is conceptually important: a rubric can be correct for human graders yet still be under-specified for an LLM, and the model may learn keyword surrogates for conceptual distinctions unless the rubric encodes the intended logic chain.

Criteria themselves can also drift. EvalGen identifies “criteria drift” as the phenomenon that users need criteria to grade outputs, but grading outputs helps users define criteria; some criteria become dependent on the specific outputs observed rather than being fully specifiable a priori (Shankar et al., 2024). This challenges workflows that assume a stable rubric exists before model interaction begins. In grading contexts, it implies that rubric design and error analysis are mutually constitutive rather than sequentially separable.

Scale and subgroup heterogeneity introduce another controversy. The grading-scale study shows that pooled reliability can mask benchmark heterogeneity, and that human–LLM alignment varies across task types and across gender-stratified panels even when within-group reliability is high (Li et al., 6 Jan 2026). The paper’s “reliability illusion” has direct workflow implications: a grader that looks stable in aggregate may be unstable on the most subjective subtask.

Finally, operational context can itself generate unfairness. In graduate reading-report grading, continuous interaction history drove systematic drift in model grading standards away from human expert scores, while simple ensemble approaches failed to improve alignment (Zhou et al., 7 Jun 2026). In national essay grading, prompt injection appended to student essays increased GPT-4.1 scores by an average of $83.3\%$8 points on a $83.3\%$9 scale in a test sample, demonstrating that rubric-driven scoring pipelines remain vulnerable if inputs are not hardened and filtered (Karjus et al., 22 Jan 2026). These findings frame “human alignment” not only as a modeling question but as a protocol question involving context management, security, and auditability.

6. Canonical design patterns and future directions

A mature human-aligned grading workflow therefore exhibits several recurring design patterns. First, it separates evaluative and formative tasks: proof-based-course evidence explicitly recommends not using a single LLM workflow for both grading and feedback, because grading requires contextual normative judgment while feedback can benefit from exhaustive, explicit explanation if a human edits the output (Mahinpei et al., 27 Feb 2026). Second, it treats the human-authored rubric as immutable core policy and limits machine adaptation to an explicit, inspectable layer such as $70.5\%$0 in GradeHITL (Li et al., 7 Apr 2025).

Third, it routes uncertainty and subjectivity rather than pretending to eliminate them. Confidence gates, benchmark-based recalibration, multi-pass variance checks, and mandatory review are all mechanisms for localizing human effort on cases where model reliability is weakest (Raikote et al., 12 Mar 2026, Chen et al., 28 Jul 2025, Vanhoyweghen et al., 13 Mar 2026). Fourth, it makes human clarifications reusable. GradeHITL stores validated human–LLM Q&A pairs and uses a retriever–reflector–refiner loop to update grading behavior while preserving the human core rubric (Li et al., 7 Apr 2025). EvalGen likewise turns human labels into an alignment report card for candidate evaluators, with explicit coverage and false-failure trade-offs (Shankar et al., 2024).

Fifth, it exposes controllable interface parameters rather than only final scores. The studies recommend controls for feedback verbosity, accept/partial-accept/reject actions, visualization of score distributions and benchmark comparisons, and structured outputs that can be audited or exported (Chen et al., 28 Jul 2025, Vanhoyweghen et al., 13 Mar 2026). This suggests that the workflow should be designed as an interactive assessment environment, not merely an API call.

Future work in the literature points toward live deployments with real instructors, multi-rater consensus modeling, subgroup fairness audits, multimodal grading, adaptive thresholding, and richer integration of textual feedback with numeric scoring (Raikote et al., 12 Mar 2026, Chen et al., 28 Jul 2025). A plausible synthesis is that the field is moving toward partial-automation systems in which LLMs act as calibrated decision support, first-draft feedback engines, and rubric refinement tools, while humans remain the final arbiters of grades, policy, and contestability.