TrueGradeAI: Selective AI Grading

• TrueGradeAI is an AI-assisted grading system that combines rubric-grounded LLM evaluation, handwriting transcription, and confidence estimation for both digital and handwritten assessments.
• The framework integrates multimodal inputs and retrieval-augmented evidence to automatically score routine cases while routing ambiguous items to human graders.
• It employs threshold-based acceptance using psychometric models and uncertainty metrics to optimize grading accuracy, workload management, and auditability.

Searching arXiv for the cited TrueGradeAI paper and closely related grading-workflow papers to ground the article. TrueGradeAI is a name used in the recent grading literature both for a specific retrieval-augmented digital examination framework and, in several methodological syntheses, for a broader class of AI-assisted grading systems that combine rubric-grounded LLM evaluation, handwriting transcription or multimodal parsing, confidence estimation, and human-in-the-loop review. Across these usages, the central design objective is not unrestricted automation, but selective automation: routine cases are handled automatically, while uncertain, ambiguous, low-confidence, or policy-sensitive cases are routed to human graders in order to preserve grading accuracy, auditability, and fairness (Thakur et al., 26 Sep 2025, Kortemeyer et al., 2024, Li et al., 7 Apr 2025).

1. Conceptual scope and problem setting

TrueGradeAI addresses a recurrent problem in educational assessment: open-ended and partially open-ended responses are expensive to grade, difficult to standardize, and vulnerable to both human inconsistency and AI overconfidence. The literature associates the label with multiple settings, including handwritten physics and calculus exams, rubric-based short-answer grading, CS1 code grading, collaborative project moderation, and digital tablet-based examinations with retrieval-augmented evidence grounding (Kortemeyer et al., 2024, Memon et al., 17 Nov 2025, Thakur et al., 26 Sep 2025).

In handwritten STEM contexts, the core difficulty is not only scoring but representation. Responses may contain free-form derivations, figures, symbolic expressions, diagrams, and numerically dense work, often written in irregular layouts or outside designated answer regions. The exploratory thermodynamics study at ETH Zurich described 252 consenting examinees, 3041 pages, and item types such as multi-step derivations, conceptual questions, diagram-based responses, and numerical problems, with OCR failures linked to regional handwriting styles and figure interpretation (Kortemeyer et al., 2024). Related studies on mathematics and chemistry report the same bottleneck in different forms: current multimodal models can often parse textual or equation-heavy responses, but graphical tasks, loosely structured page layouts, and fine symbolic distinctions remain substantially less reliable (Caraeni et al., 2024, Cvengros et al., 12 Sep 2025).

A second recurring theme is that grading validity is inseparable from rubric structure. In the cited work, AI systems do not simply “score answers”; they operationalize grading criteria at the level of rubric items, score categories, or evidence-linked claims. This is explicit in the retrieval-augmented digital examination framework, where scoring is modeled as a criterion-weighted aggregation, $$\mathrm{Score} = \sum_{j=1}^{m} w_j\,s_j - \sum_k \lambda_k\,p_k,$$ with criterion scores, weights, and rubric-defined penalties tied to retrieved evidence (Thakur et al., 26 Sep 2025). In short-answer grading, the GradeHITL framework similarly treats grading as rubric-conditioned classification, with a “policy layer” that is not rewritten by the optimizer; only appended “Adaption Rules” are revised during human-in-the-loop refinement (Li et al., 7 Apr 2025).

2. System architecture and grading pipeline

The most explicit architectural formulation of TrueGradeAI appears in the digital examination framework "TrueGradeAI: Retrieval-Augmented and Bias-Resistant AI for Transparent and Explainable Digital Assessments" (Thakur et al., 26 Sep 2025). That system begins with secure tablet stylus capture in a locked exam environment, stores both pen-stroke traces and page images, transcribes handwriting with TrOCR, retrieves relevant faculty and supporting material through dual retrieval modules, and uses an LLM to assign scores with structured, evidence-linked rationales. The retrieval layer is split into RAG1, a faculty knowledge base of model answers, and RAG2, supporting materials such as textbooks and references, with HOT/COLD caches used to retain frequently accessed facts and broader context (Thakur et al., 26 Sep 2025).

This retrieval-augmented design differs from pure OCR-plus-grading pipelines in handwritten exam studies. In the ETH Zurich thermodynamics workflow, GPT-4o is used for OCR of handwritten pages into LaTeX and textual graph descriptions, while GPT-4-Turbo grades each rubric item part by part across ten independent runs (Kortemeyer et al., 2024). In the handwriting-heavy calculus study, multimodal GPT-5 receives two images per grading cycle—combined student pages and the rubric image for that prompt—and returns rubric-aligned item scores with fractional credit permitted (Kortemeyer et al., 4 Oct 2025). In the chemistry system, one page of student work and its corresponding rubric page are graded jointly by a multimodal model, which returns per-part JSON objects containing problem_part, awarded_points, and a concise explanation (Cvengros et al., 12 Sep 2025).

TrueGradeAI therefore appears in two implementation families. The first is a document-centric workflow: scan or capture handwritten work, transcribe or parse it, grade rubric elements, and route uncertain cases. The second is a retrieval-centric digital assessment workflow: preserve handwriting natively, transcribe it with transformer OCR, ground grading in institutional knowledge and trusted references, and log evidence, source identifiers, and rationales for later review (Thakur et al., 26 Sep 2025). A plausible implication is that these two families are complementary rather than competing: document-centric pipelines optimize compatibility with existing paper-based assessment, whereas retrieval-augmented digital frameworks optimize auditability, explainability, and institutional memory.

3. Confidence, psychometrics, and routing

A defining feature of TrueGradeAI in its strongest formulations is confidence-aware routing rather than unconditional automation. The psychometric version developed for handwritten physics exams uses Item Response Theory, specifically a two-parameter logistic model, to estimate expected performance on each rubric item, $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$ where $a_i$ is item discrimination, $b_i$ item difficulty, and $\theta_j$ student ability (Kortemeyer et al., 2024). The AI produces a partial-credit score $s_{ij} \in [0,1]$, and uncertainty is defined as the discrepancy $u_{ij} = \left| p_i(\theta_j) - s_{ij} \right|.$ An AI decision is accepted only when it passes a correctness threshold $C$, an uncertainty threshold $U$, and, optionally, a gate $P$ that routes all AI-graded “incorrect” items to human review: $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$0 This formulation turns workload reduction into an explicit acceptance-rate optimization problem rather than an uncalibrated confidence guess (Kortemeyer et al., 2024).

The same basic idea appears in the handwritten calculus study, but with a slightly different notation. There, the 2PL model generates a model-expected normalized score $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$1, and risk is defined as $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$2 The operational rule is conservative: accept an AI decision only if $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$3 and $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$4; otherwise route the item to human grading (Kortemeyer et al., 4 Oct 2025). The paper explicitly frames this as a workload–quality trade-off. Stricter thresholds improve alignment with TA grading, but they also increase the proportion of items requiring human review.

Text-based grading frameworks use analogous logic even when they are not explicitly psychometric. GradeHITL collects low-confidence or misgraded examples, generates rubric clarification questions, validates human answers against a validation set, and updates only the clarifying “Adaption Rules” of the rubric while preserving the expert-authored policy layer (Li et al., 7 Apr 2025). This suggests that TrueGradeAI, in its broader sense, is less a single model than a graded-decision architecture: score first, estimate confidence or risk second, and only then decide whether a machine-generated grade is admissible.

4. Domain-specific instantiations

TrueGradeAI has been instantiated, or used as a design label, across markedly different educational modalities. In handwritten STEM grading, the strongest evidence concerns selective automation rather than end-to-end autonomy. The thermodynamics study at ETH Zurich used detailed item-level rubrics, GPT-4o OCR, GPT-4-Turbo grading, ten independent grading runs per part, and IRT-based uncertainty to gate accepted grades (Kortemeyer et al., 2024). The calculus study used multimodal GPT-5 against rubric images and found that calibrated filtering could raise agreement to “human-level accuracy,” but only by routing roughly 70% of items to human graders under stricter settings (Kortemeyer et al., 4 Oct 2025). In general chemistry, textual and reaction-equation items were relatively reliable, whereas numerical derivations were weaker and graphing yielded a normed F1 of $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$5, worse than chance (Cvengros et al., 12 Sep 2025).

In computer science, the name is used for both assignment grading and collaborative project moderation. For CS1 programming assignments, the literature distinguishes a Direct method, where the LLM applies a rubric directly to code, and a Reverse method, where the model repairs code, classifies fixes as minor or major, and infers deductions from a perfect score (Memon et al., 17 Nov 2025). For collaborative projects, TrueGradeAI denotes a semi-automated system combining repository mining, communication analytics, and machine learning to generate a project quality score $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$6, an individual contribution score $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$7, and a moderated final grade $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$8 (Yu et al., 5 Oct 2025).

In short-answer and essay-type grading, the emphasis shifts from psychometric routing toward rubric clarification, template grounding, and discrepancy analysis. GradeHITL treats grading as rubric-conditioned classification with human-authored questions and validated Q&A pairs stored in a retrieval database (Li et al., 7 Apr 2025). In an introductory sociology course, GPT-4o and GPT-4o-mini were used as “second graders” on handwritten essay responses; correlations with the human grader improved substantially when template answers were supplied, with GPT-4o reaching $$p_i(\theta_j) = \frac{1}{1 + \exp\left(-a_i(\theta_j - b_i)\right)},$$9 and average differences near zero in the best conditions (Olivos et al., 11 Jan 2025).

A broader misconception is that TrueGradeAI necessarily implies a single universal grader. The literature instead suggests a federated design pattern: the same name is reused for domain-specific workflows that differ in input modality, rubric structure, target metric, and human oversight policy. What persists across these settings is the attempt to bind LLM outputs to external constraints—rubrics, psychometric expectations, retrieval evidence, repository traces, or appeal procedures—so that scoring becomes inspectable rather than purely generative.

5. Empirical performance and workload trade-offs

The empirical literature on TrueGradeAI is notable for reporting explicit trade-offs between acceptance coverage and grading agreement. In the psychometric thermodynamics workflow, unfiltered AI grading reached $a_i$0, but confidence gating with $a_i$1, $a_i$2, and $a_i$3 increased $a_i$4 to $a_i$5 while reducing acceptance from 80% to 50%. A stricter configuration achieved $a_i$6 when about 21% of student-items were auto-graded, and the highest observed $a_i$7 was 0.98 when only 3% of student-items were auto-graded (Kortemeyer et al., 2024). The same study reports token costs of approximately \$7 per exam, with ten grading runs per part and the observation that as few as three runs often suffice for clearly correct or incorrect responses (Kortemeyer et al., 2024).

The calculus study shows a comparable pattern. Unfiltered AI–TA agreement on total score yielded $a_i$8, slope $a_i$9, and offset $b_i$0 points. Filtering with $b_i$1 and $b_i$2 raised $b_i$3 to $b_i$4 at an acceptance rate of $b_i$5, while the stricter setting $b_i$6, $b_i$7 yielded $b_i$8 and slope $b_i$9 at an acceptance rate of only $\theta_j$0 (Kortemeyer et al., 4 Oct 2025). In general chemistry, total exam score agreement was already strong at $\theta_j$1 unfiltered, but full-credit-only filtering pushed the fit toward $\theta_j$2 while reducing coverage below one-third of all parts (Cvengros et al., 12 Sep 2025).

The retrieval-augmented digital assessment framework reports a different performance profile because it is evaluated on textbook-aligned digital QA pairs rather than handwritten, page-segmented high-stakes exam work. On 10,000 QA pairs from NCERT Class 12 History, Political Science, and Geography textbooks, it reports Pearson correlation $\theta_j$3, Spearman $\theta_j$4, and Cohen’s kappa $\theta_j$5, with preprocessing around 6 hours, RAG evaluation around 1.5 hours, and average inference around 1.2 seconds per QA (Thakur et al., 26 Sep 2025). This suggests that much of the residual difficulty in handwritten assessment lies upstream of scoring—in OCR, segmentation, spatial anchoring, and diagram interpretation—rather than in rubric-grounded scoring alone.

These results also correct another common misconception: strong correlation does not imply deployability without qualification. Several studies report moderate or strong regression fits while simultaneously concluding that human oversight remains essential. GPT-4o grading of handwritten probability exam solutions improved from MAE $\theta_j$6 to $\theta_j$7 and correlation from $\theta_j$8 to $\theta_j$9 when correct answers and rubrics were included, yet the study still judged accuracy too low for real-world deployment without human review (Caraeni et al., 2024). TrueGradeAI, as documented in the literature, is therefore best understood as a selective assistance system whose usefulness depends on where acceptance thresholds are set.

6. Governance, limitations, and future directions

Governance is central to the TrueGradeAI literature because most of the targeted use cases are high-stakes or educationally consequential. The cited work repeatedly endorses human-in-the-loop oversight, audit trails, and appeal pathways. In the EU regulatory context, AI for learning outcome evaluation is described as “high-risk,” and the thermodynamics study explicitly notes that the $s_{ij} \in [0,1]$0 gate can be used to ensure that only AI-graded “correct” items are auto-accepted while all “incorrect” items are routed to humans (Kortemeyer et al., 2024). The digital assessment framework similarly foregrounds anonymization, randomized script allocation, retrieval-grounded explanations, and auditable logs of prompts, retrieval hits, scoring rationales, and final decisions (Thakur et al., 26 Sep 2025).

Limitations are equally consistent across domains. Rubric refinement by statistical diagnostics alone had limited or even negative impact in the thermodynamics study, and some transcription errors were not caught by negative discrimination or correlation anomalies (Kortemeyer et al., 2024). Handwritten math and chemistry grading remains especially vulnerable to diagrammatic tasks, graphical clutter, ambiguous notation, and work written outside designated answer regions (Kortemeyer et al., 4 Oct 2025, Cvengros et al., 12 Sep 2025). In K–12 deployment work, teachers valued rapid narrative feedback but distrusted automated scoring, and students expressed skepticism toward AI-only grading even when they appreciated fast revision-oriented comments (Zewei et al., 9 Jun 2025). In AI-PAT, an appeal-resolution system for CS education, 185 appeal cases were processed and 74% resulted in grade changes, underscoring the instability of both AI and human grading configurations when transparent appeal channels are absent (Aytutuldu et al., 18 Apr 2025).

A plausible implication is that the most durable form of TrueGradeAI will not be a fully autonomous grader but a layered institutional workflow. The evidence favors systems that preserve expert-authored policy, expose evidence for each deduction, quantify uncertainty explicitly, retain detailed logs for appeals and audits, and reserve human judgment for responses that are ambiguous, multimodal, adversarial, pedagogically rich, or simply badly captured. In that sense, TrueGradeAI marks a shift away from monolithic “automatic grading” and toward reliability-aware grading infrastructures in which automation is admitted only when its evidentiary and psychometric conditions are satisfied.