CoGrader: Collaborative Project Grading

• The paper introduces a calibrated human-AI grading workflow that leverages instructor-selected benchmarks and automated metric design for open-ended project reports.
• It employs a multi-stage process including initial AI scoring, benchmark-driven regrading, and AI-assisted feedback synthesis to ensure grading consistency.
• Empirical evaluation demonstrates high efficiency and reliability, with instructors actively engaging in metric customization and score adjustments.

CoGrader is a human-AI collaborative grading system for project report assessment in project-based learning (PBL). It was designed for open-ended project reports whose evaluation depends on customized metrics, subjective judgment, peer-comparative calibration, and instructor knowledge of the class context. Rather than replacing instructors with a one-shot LLM judge, CoGrader reorganizes grading into a workflow of human-LLM collaborative metrics design, initial AI grading, benchmark-driven regrading, and AI-assisted feedback generation, while preserving instructor authority over criteria, benchmark choice, score adjustment, and final feedback (Chen et al., 28 Jul 2025).

1. Problem domain and conceptual framing

Project reports occupy a different assessment regime from quizzes, multiple-choice items, grammar checks, or other standardized tasks. In the CoGrader formulation, they are used to evaluate complex learning outcomes such as originality, creativity or innovation, practical application of knowledge, technical depth, peer-comparative effort or achievement, and design sophistication. These dimensions are difficult to reduce to deterministic grading rules, and the diversity of report topics, structures, and solution approaches makes rigid rubric application difficult (Chen et al., 28 Jul 2025).

The system is motivated by four interlocking problems. First, project-report grading is time-consuming. Second, instructors often grade over extended periods, so their standards evolve as they discover stronger and weaker reports, producing grading drift. Third, retroactive adjustment is burdensome because earlier reports are often revisited after later benchmarks emerge. Fourth, detailed and cohesive feedback is difficult to sustain at scale, even though project-based learning benefits from nuanced, comparative commentary. CoGrader addresses these issues by making metric design, calibration, benchmark selection, regrading, and feedback synthesis explicit stages of a collaborative workflow rather than implicit parts of ad hoc manual grading (Chen et al., 28 Jul 2025).

A central conceptual distinction in CoGrader is between lower-judgment and higher-judgment criteria. This distinction is operationalized by separating metrics into “Auto Grade” and “Score Reference.” Objective or relatively standardized criteria can be delegated more directly to AI, whereas subjective or context-dependent criteria remain advisory and instructor-governed. This suggests that CoGrader is best understood not as an autonomous grader but as a calibrated assessment environment for complex reports.

2. Formative study and extracted design requirements

The system design was grounded in a formative study with six university instructors: two assistant professors, three lecturers, and one postdoctoral researcher. All had at least three years of experience designing and grading project reports. Each semi-structured interview lasted about 70 minutes, divided into a 30-minute discussion of existing workflows and pain points and a 40-minute brainstorming session on which grading activities could be automated by AI and which should remain under human control. With participant consent, three co-authors independently reviewed notes and recordings, used open coding, clustered key concepts into themes, and resolved discrepancies over two rounds of discussion (Chen et al., 28 Jul 2025).

The study identified a four-stage grading workflow. In S1, instructors design appropriate grading metrics aligned with learning outcomes and project requirements. In S2, they perform a quick overview of submissions and select high and low benchmark reports. In S3, they conduct detailed evaluation, benchmark adjustments, and retroactive grading. In S4, they synthesize comprehensive feedback and class-level insights. The study found that AI was viewed as useful for generating and recommending metrics from project requirements, standardized report evaluation, and summarizing report content, but that benchmark selection, retroactive grading, and evaluation of subjective qualities such as creativity and novelty still required substantial human judgment (Chen et al., 28 Jul 2025).

These observations were distilled into six design requirements: R1 automatic extraction of core metrics from project requirements, R2 collaborative, multi-source metric enrichment and customization, R3 automated preliminary analysis and AI-driven insights, R4 AI-assisted report benchmarking, comparison, and retroactive grading, R5 AI-assisted, instructor-centered feedback and insight generation, and R6 preservation of instructor authority with balanced AI reliance. The sixth requirement is foundational. CoGrader is intentionally structured so that AI outputs function as suggestions, references, and evidence-supported drafts rather than final judgments.

3. Workflow, views, and benchmark-driven calibration

CoGrader organizes the grading process into three views: Metric View, Benchmarking View, and Feedback View. The workflow begins when the instructor uploads project requirement documents and student report files. In Metric View, clicking “Analyze Requirement” causes the system to produce Report Objective Metrics, which are directly grounded in the uploaded requirements, and Extra Potential Metrics, which are AI-suggested criteria that may be educationally useful even if not explicitly listed. The view also includes Standard Writing Metrics and Customized Metric Design, where instructors describe desired metrics in natural language and the AI converts them into grading metrics with explanations and suggested formulas (Chen et al., 28 Jul 2025).

Each metric can then be selected and classified as either Auto Grade or Score Reference. This classification controls how strongly the AI participates in scoring versus advisory comparison. After metric selection, instructors enter Benchmarking View and click “Grade Reports.” The system grades all reports according to the selected metrics and returns initial metric scores, AI-generated comments, and average metric scores shown in report cards. Instructors can inspect metric-level comments and edit AI-proposed scores and comments report by report (Chen et al., 28 Jul 2025).

Benchmark-driven calibration is the defining mechanism of the system. As instructors review reports in detail, they designate some submissions as Benchmark High and Benchmark Low. These serve as anchors for class-level standardization. Clicking “Regrade Reports” causes the system to compare each report against the selected metrics, the current scores and comments, and the chosen high and low benchmarks. The resulting regrading makes peer-relative judgment explicit rather than implicit. The interface supports this with radar charts comparing a focused report against benchmark reports and with metric distribution views that allow instructors to sort reports by a chosen metric in descending order (Chen et al., 28 Jul 2025).

The workflow concludes in Feedback View, where instructors inspect full report contents, highlight text, annotate sections, revise scores and comments, and then trigger “Generate Feedback.” The AI synthesizes the report summary, instructor-updated scores, metric comments, highlights and annotations, and other grading notes into personalized feedback. The output is deliberately ordered to prioritize instructor-authored content, then instructor-edited AI suggestions, and only then pure AI-generated comments where gaps remain. This sequencing is central to the system’s human-governed design.

4. Technical architecture and allocation of labor

The implementation uses OpenAI’s ChatGPT-4o API, specifically the Assistant API, Structured Response, and File Search via the Assistant API. The web stack is a React.js frontend and a Python Flask API backend communicating through RESTful APIs. The backend mirrors the grading workflow through three LLM modules: Collaborative Metric Design, Benchmarking-driven Grading, and Feedback Generation (Chen et al., 28 Jul 2025).

In the metric-design stage, the LLM receives project requirements and optional instructor-authored metric descriptions, then produces Report Objective Metrics, Extra Potential Metrics, and Customized Metrics after a Redundancy Check. In initial grading, the grading agent is prompted to “think” and “reason” about each report based on selected metrics, search for supporting evidence within the report, assign initial scores, and produce comments with supporting excerpts. In benchmark-based regrading, the model is prompted to “compare” the target report and its current evaluations against the selected benchmarks. In feedback generation, a separate agent synthesizes user-updated scores, comments, annotations, and report summaries into a structured student-facing response (Chen et al., 28 Jul 2025).

The division of labor is explicit. The model extracts candidate metrics, suggests extra metrics, checks redundancy, summarizes reports, proposes first-pass scores and comments, regrades relative to benchmarks, and synthesizes feedback. The instructor chooses and refines final metrics, decides which metrics are Auto Grade versus Score Reference, inspects report evidence, edits scores and comments, selects and revises benchmarks, triggers regrading iteratively, annotates reports freely, and approves the final feedback. The paper does not provide explicit scoring formulas, optimization objectives, or confidence estimation equations in the main text; the technical description is procedural rather than equation-based (Chen et al., 28 Jul 2025).

This architecture implies a grading philosophy different from execution-grounded autograders for code. Systems such as “Code Generation Based Grading” transform a student explanation into executable code and infer correctness from unit tests, thereby measuring whether the explanation contains enough semantic content to synthesize correct behavior (IV et al., 2023). By contrast, CoGrader addresses open-ended report evaluation where abstraction quality, contextual judgment, and peer-relative calibration are integral to the construct being graded.

5. Empirical evaluation and observed use patterns

The user study involved 12 participants: 4 university professors or lecturers and 8 senior PhD students with teaching assistant experience. Gender distribution was 8 male and 4 female. Average experience in designing or grading project reports was 3.25 years; 7 participants had used LLMs for grading before and 5 had not. Each session lasted about 80 minutes: 50 minutes of system exploration and grading, 15 minutes of questionnaire, and 15 minutes of post-interview. Participants worked with a sample project requirement and five sample project reports from a real-world master-level data visualization course from spring 2024. Ground-truth scores were a weighted average of evaluations from one course instructor and two senior teaching assistants, and the five reports were selected to span letter grades from B- to A (Chen et al., 28 Jul 2025).

Collaborative metric generation received strong ratings. Efficiency was rated at mean = 6.5, stdev = 0.522. Reliability ratings were mean = 5.91, stdev = 0.793 for Report Objective Metrics, mean = 6.16, stdev = 0.937 for Extra Potential Metrics, and mean = 5.83, stdev = 1.115 for Custom Metric Design. Initial AI grading was viewed as efficient but less reliable, with mean = 5.92, stdev = 1.083 for efficiency, mean = 4.91, stdev = 1.083 for reliability of initial scores, and mean = 4.83, stdev = 1.030 for reliability of initial comments. Benchmark-driven regrading received the strongest ratings: mean = 6.42, stdev = 0.515 for efficiency, mean = 6.08, stdev = 0.900 for reliability of regraded scores, mean = 5.83, stdev = 1.11 for reliability of regraded comments, and mean = 6.00, stdev = 0.853 for reliability of the benchmark-driven grading process overall (Chen et al., 28 Jul 2025).

Feedback and summary generation were also rated positively. Efficiency of summary generation was mean = 6.33, stdev = 0.778; reliability of summary and feedback processes was mean = 5.08, stdev = 0.996; efficiency of feedback generation was mean = 6.00, stdev = 1.279. Overall workflow ratings were mean = 5.83, stdev = 1.193 for reliability, mean = 5.92, stdev = 0.996 for efficiency, mean = 6.33, stdev = 0.492 for ease of use, and mean = 6.00, stdev = 0.739 for overall satisfaction (Chen et al., 28 Jul 2025).

Alignment with course ground truth was reported through rank and linear correlation: Kendall’s Tau = 0.799, Spearman’s Rank Correlation = 0.899, and Pearson Correlation = 0.900. Report-level averages were also close to the ground truth, for example R01: 90.9 vs 89.1, R02: 75.0 vs 75.6, R03: 74.5 vs 71.5, R04: 84.1 vs 79.3, and R05: 83.3 vs 82.9. The paper also measured instructor engagement with AI outputs rather than passive acceptance. On average, participants selected 4.50 Automatic Metrics per report and 5.83 Reference Metrics per report, wrote 5.25 manual personalized comments per report, and adjusted or overrode 64.09% of AI-generated scores and comments for Reference Metrics. Additionally, 9/12 participants manually checked most AI-referenced facts in Automatic Metrics against the original reports, while the remaining 3 checked high-impact criteria selectively (Chen et al., 28 Jul 2025).

6. Significance, limitations, and relation to adjacent systems

CoGrader’s most distinctive contribution is benchmark-anchored, peer-comparative grading for project reports. The fairness mechanism is not a claim of automated impartiality; it is a calibration mechanism based on instructor-selected anchors, retroactive regrading, report-comparison visualizations, and class-wide metric distributions. This differs from LLM-centric scoring systems such as CodEv, which embed rubric criteria directly into prompts, use Chain-of-Thought criterion decomposition, and stabilize scores through repeated sampling and voting (Tseng et al., 10 Jan 2025). It also differs from multi-agent rubric systems such as AGACCI, which distribute evaluation across specialized agents for execution, results, visualization, and interpretation in code-oriented assignments (Park et al., 7 Jul 2025). CoGrader’s niche is open-ended report assessment where peer-relative judgment and evolving instructor standards are intrinsic to the grading task.

The paper nonetheless acknowledges substantial limitations. Participants observed score variation across attempts, so LLM inconsistency remains an open problem. Some comments were generic, and stronger linkage from AI comments to specific report excerpts was requested. The current system focuses primarily on text and does not evaluate non-textual or external artifacts well; it also does not incorporate external materials such as code, which constrains fact verification in technical domains. The evaluation used only five reports from one master-level data visualization course and did not include a controlled A/B comparison against another software baseline or a purely manual timing baseline with statistical significance testing (Chen et al., 28 Jul 2025).

These limitations are significant when CoGrader is placed alongside adjacent grading architectures. Execution-grounded programming graders such as Mark My Works and Autograder+ integrate test execution with LLM-generated explanation and semantic clustering, thereby grounding correctness in runtime behavior rather than in report text alone (Qiu et al., 15 Jan 2026, Sahu et al., 30 Oct 2025). LaTA shows that local, on-premises, rubric-line grading with binary per-item scoring can be operationally viable at scale in STEM settings, though in a LaTeX-native derivation workflow rather than an open-ended report workflow (Rodríguez, 6 May 2026). Semi-automated collaborative project moderation frameworks separate team artifact quality from individual contribution evidence using repository mining, issue tracking, and review analytics, addressing a different assessment construct from project-report grading (Yu et al., 5 Oct 2025). A plausible implication is that CoGrader’s architecture is especially suited to assignments where contextual, comparative, and explanatory judgment dominate, but it would require extension or hybridization for code-execution-intensive, multimodal, or collaborative-software assessment contexts.

Future directions named in the paper include finer-grained calibration at metric, section, and whole-report levels; targeted regrading of specific metrics or sections; interactive dialogue with the LLM for regrading refinement; inclusion of external artifacts such as code files; stronger transparency mechanisms; and adaptation to other domains such as business presentations and design projects (Chen et al., 28 Jul 2025). In that sense, CoGrader represents not a completed theory of AI grading, but a structured model of how instructors and LLMs can collaboratively govern the difficult parts of project-report assessment without collapsing them into opaque one-shot automation.