FeedbackWriter: AI Grading Mediation

• FeedbackWriter is an AI-mediated feedback system that integrates rubric-aligned suggestions with teaching assistant oversight to enhance student essay revisions.
• It employs a three-stage GPT-4o pipeline for extractive question answering, rubric judgment, and feedback generation, ensuring precise and actionable comments.
• The system demonstrated a significant 5 percentage-point improvement in revision quality through a randomized controlled trial and robust teaching assistant adoption.

Searching arXiv for the cited FeedbackWriter-related papers to ground the article in current preprints. FeedbackWriter is an AI-mediated feedback system for student writing in which a LLM generates rubric-aligned suggestions to teaching assistants while they grade essays, and teaching assistants retain the capacity to adopt, edit, or dismiss every suggestion before any comment reaches students. In a randomized controlled trial in a large introductory economics course, the system was deployed on knowledge-intensive essays and was associated with significantly higher-quality student revisions than handwritten teaching-assistant feedback alone (Lu et al., 18 Feb 2026). In the broader literature on AI writing support, closely related systems include the Arabic writing assistant ARWI, which combines grammatical error detection, correction, automated essay scoring, and feedback generation (Chirkunov et al., 16 Apr 2025), as well as design sketches explicitly framed as “FeedbackWriter” systems that use reward modeling and self-reflective revision to improve generated text (Chakrabarty et al., 10 Apr 2025, Zhao et al., 16 Mar 2026).

1. System definition and runtime workflow

FeedbackWriter is built as a full-stack web application with a React frontend and Django backend, and it integrates with Canvas via its API (Lu et al., 18 Feb 2026). Its core is a three-stage, rubric-aligned LLM pipeline powered by GPT-4o with temperature $0.05$.

The first stage is extractive question answering over the student essay and the rubric. For each rubric item, the system identifies all sentences in the essay “where the student attempts to address this criterion.” These sentence spans are returned to the interface as draggable highlights. The second stage is rubric judgment. The prompt includes the rubric text, the highlighted sentences, and the instruction “Rationale, then: met/not-met.” The model then performs a satisfiability check, deciding whether every required element is present and accurate, generating a rationale and then a binary judgment. The third stage is feedback generation. The prompt includes the rubric text, the judged status of the item, and three instructor-supplied exemplars for few-shot-style hints. The output consists of two side-by-side suggestions: AI-generated personalized feedback, described as Socratic questions and indirect hints, and historic feedback drawn from prior semesters (Lu et al., 18 Feb 2026).

The teaching-assistant interface is organized as a split pane. Each rubric item appears as a collapsible feedback box containing the rubric statement with a color-coded checkbox, drag-and-drop sentence highlights anchored in the essay, two suggestion cards labeled “Historic” and “GPT,” and a blank student-facing feedback textarea that requires teaching assistants to explicitly insert and/or edit any suggestion. Because no AI-generated text is sent to students until a teaching assistant clicks “Add,” all student-facing comments are vetted and modifiable. Teaching assistants may reposition highlights, flip judgments, adopt, edit, or generate suggestions anew, or write free-form comments (Lu et al., 18 Feb 2026).

A central property of the system is therefore mediation rather than automation. The AI operates as a rubric-grounded suggestion engine embedded in an existing grading workflow, while instructional control remains with the teaching assistant. This suggests that FeedbackWriter is designed less as an autonomous grader than as a human-AI coordination interface for feedback provision.

2. Course deployment and randomized evaluation

The randomized evaluation took place in a large semester-long Introduction to Economics course at an R1 university with $N=354$ students, two writing-to-learn assignments per student, and four submissions per student: two first drafts and two final drafts (Lu et al., 18 Feb 2026). Eleven teaching assistants, each responsible for one discussion section, were pre-trained in rubrics and grade-norming. No student or teaching assistant opted out.

Students were randomly assigned at the semester’s start to one of two cross-over groups. Group A received baseline human-only feedback on Assignment 1 and FeedbackWriter-mediated feedback on Assignment 2. Group B received FeedbackWriter-mediated feedback on Assignment 1 and baseline feedback on Assignment 2. Across both assignments, a total of 1,366 essays were graded under feedback conditions (Lu et al., 18 Feb 2026).

The grading rubrics comprised 35 knowledge-intensive items per assignment, including items such as “Identify consumer vs. producer deadweight loss” and “Propose solution to the externality with efficient size.” For each rubric item, teaching assistants awarded met or missing judgments and provided comments. Essay quality was later quantified by an independent GPT-4o-based rubric scorer that reused the same rubric definitions to assign $0/1$ values for each item and then aggregated them via the teaching assistants’ weighting scheme into a percentage score:

$$\text{EssayScore}= \frac{\sum_{i=1}^{35} w_i\,\mathbf{1}_{\{\text{item}_i\ \text{met}\}}}{\sum_{i=1}^{35} w_i}\times 100\%.$$

The experimental design is methodologically notable because it evaluates not merely feedback generation but student response to feedback. The relevant outcome is the quality of the revised draft rather than the text of the feedback in isolation. That emphasis distinguishes FeedbackWriter from many AI writing-assistance studies that stop at model outputs or annotator preferences.

3. Effects on revision quality and post-test performance

Revision quality was analyzed using a linear mixed-effects regression in which final revision score was the outcome, with first-draft score, feedback condition, provisional teaching-assistant score, and assignment indicator as fixed effects, and student and teaching assistant as random intercepts (Lu et al., 18 Feb 2026). In the reported model, the coefficient for AI-mediated feedback was $\widehat\beta=0.05$ with $t=5.02$ and $p<0.001$, while the coefficient for first-draft quality was $\widehat\beta=0.56$ with $t=10.99$ and $p<0.001$. Controlling for draft quality and teaching-assistant score, the estimate indicates a 5 percentage-point increase in revision quality under AI-mediated feedback.

The paper also reports a standardized effect size of approximately $N=354$0, described as equivalent to moving a student from the 50th to the 70th percentile (Lu et al., 18 Feb 2026). In descriptive terms, students receiving AI-mediated feedback improved their revision scores by an average of $N=354$1 from $N=354$2 to $N=354$3, whereas students under human-only feedback improved by $N=354$4 from $N=354$5 to $N=354$6. The difference in improvements was reported as significant with $N=354$7.

A parallel mixed model examined an optional extra-credit post-test taken immediately after final-draft submission. Final-draft quality predicted post-test performance, with $N=354$8, $N=354$9, and $0/1$0, but the feedback condition itself was not significant, with $0/1$1 (Lu et al., 18 Feb 2026). The result distinguishes short-term revision gains from immediate transfer to post-test outcomes.

Outcome	Estimate	Interpretation
AI-mediated feedback in revision model	$0/1$2	5 percentage-point increase in revision quality
Standardized effect size	$0/1$3	50th to 70th percentile shift
Final-draft quality in post-test model	$0/1$4	Higher final quality predicts higher post-test score
Feedback condition in post-test model	$0/1$5	Not significant

These results support a narrow but important claim: FeedbackWriter improved the quality of revised essays, but the reported evidence did not establish a direct condition effect on the optional immediate post-test. A plausible implication is that the system’s strongest demonstrated effect lies in revision support rather than in short-horizon content learning as measured in that quiz format.

4. Feedback uptake, annotation dimensions, and teaching-assistant behavior

The study also analyzed how teaching assistants interacted with AI suggestions. Across 703 final-draft grade logs under FeedbackWriter, teaching assistants wrote on average 8.7 comments per essay, broken down into 3.41 AI-constructive comments, 0.91 AI-praise comments, 1.92 historic comments, and 1.27 extra comments (Lu et al., 18 Feb 2026). AI suggestions were adopted 51.3% of the time, corresponding to 3,141 of 6,121 comments, whereas historic reuse was adopted 22.2% of the time. Teaching assistants corrected 11.3% of AI judgments by flipping the met or missing checkbox.

To examine whether uptake mediated revision gains, the revision model was extended with teaching-assistant log covariates, including numbers of historic comments, AI positive comments, AI constructive comments, flipped judgments, and extra feedback. The number of AI-constructive comments adopted was a significant positive predictor of final-draft quality, with $0/1$6, $0/1$7, and $0/1$8 (Lu et al., 18 Feb 2026). The reported interpretation is that each additional AI-constructive comment adopted by a teaching assistant predicts a further 0.5 percentage-point gain in final-draft quality.

Feedback quality itself was evaluated using an LLM-driven annotation pipeline validated at Cohen’s $0/1$9. Feedback segments were tagged on four binary dimensions: content accuracy, actionability, promotion of independent learning, and tone or supportiveness. Under AI-mediated feedback relative to baseline, the reported rates were 0.893 versus 0.758 for actionability, 0.926 versus 0.823 for independent learning, 0.974 versus 0.808 for tone or supportiveness, and 0.976 versus 0.960 for content accuracy, with all reported differences significant and content accuracy described as comparable (Lu et al., 18 Feb 2026).

Interview themes clarify how teaching assistants interpreted the system’s utility. They valued AI suggestions for surfacing overlooked rubric items, providing well-phrased Socratic hints without giving away answers, helping standardize rubric interpretations, and allowing them to verify and edit rather than write from scratch, thereby preserving agency and trust (Lu et al., 18 Feb 2026). These observations are consistent with the system’s interface design, which centers review and selective uptake rather than direct model-to-student communication.

5. Position within AI writing-assistance research

Within the broader literature, FeedbackWriter belongs to a family of systems that combine automated assessment with revision guidance, but it occupies a specific niche: rubric-grounded, human-mediated feedback on course assignments. ARWI, described as the first publicly available Arabic writing assistant, addresses a different problem space. It provides a prompt database for different proficiency levels, an Arabic text editor, grammatical error detection and correction, and automated essay scoring aligned with the Common European Framework of Reference. Its pipeline proceeds from preprocessing to grammatical error detection, grammatical error correction, automated essay scoring, feedback generation, and presentation in the text editor with progress tracking (Chirkunov et al., 16 Apr 2025). A preliminary user study reported that ARWI provides actionable feedback, helping learners identify grammatical gaps, assess language proficiency, and guide improvement (Chirkunov et al., 16 Apr 2025).

A second line of work treats “FeedbackWriter” as a generative writing-improvement architecture rather than a teaching-assistant tool. In “AI-Slop to AI-Polish? Aligning LLMs through Edit-Based Writing Rewards and Test-time Computation,” the proposed blueprint uses the Writing Quality Benchmark with 4,729 writing quality judgments, specialized Writing Quality Reward Models, and a Best-of-$$\text{EssayScore}= \frac{\sum_{i=1}^{35} w_i\,\mathbf{1}_{\{\text{item}_i\ \text{met}\}}}{\sum_{i=1}^{35} w_i}\times 100\%.$$0 revision pipeline in which $$\text{EssayScore}= \frac{\sum_{i=1}^{35} w_i\,\mathbf{1}_{\{\text{item}_i\ \text{met}\}}}{\sum_{i=1}^{35} w_i}\times 100\%.$$1 revised candidates are generated and ranked (Chakrabarty et al., 10 Apr 2025). Human evaluation with 9 experienced writers found that reward-model-based selection produced writing samples preferred by experts 66% overall and 72.2% when the reward gap exceeded 1 point (Chakrabarty et al., 10 Apr 2025).

A third line, “Writer-R1: Enhancing Generative Writing in LLMs via Memory-augmented Replay Policy Optimization,” formulates a detailed design sketch for a “FeedbackWriter” system powered by a multi-agent collaborative workflow based on Grounded Theory and by Memory-augmented Replay Policy Optimization. In that formulation, dynamic criteria are generated by open-coding, axial-coding, and selective-coding agents, and generation proceeds through a generation-reflection-revision loop with optional memory retrieval (Zhao et al., 16 Mar 2026). The reported main result is that Writer-R1-4B with MRPO achieves overall accuracy 0.87 on WritingBench, compared with 0.86 for Kimi-K2 and 0.84 for LongCat-Flash-Chat (Zhao et al., 16 Mar 2026).

These systems differ in domain, supervision regime, and end user. FeedbackWriter in the economics-course trial is pedagogical and workflow-embedded; ARWI is learner-facing and language-proficiency-oriented; the reward-model and MRPO variants are writing-generation architectures. This suggests a broader conceptual family in which “FeedbackWriter” denotes systems that structure revision through explicit criteria, intermediate judgments, and guided rewriting, while differing substantially in whether the human in the loop is a student, a teaching assistant, or the model itself.

6. Design principles, limitations, and open directions

The reported design recommendations for FeedbackWriter emphasize rubric engineering, cognitive alignment, human-in-the-loop controls, and iterative refinement (Lu et al., 18 Feb 2026). Rubric engineering refers to investing in precise, operational rubrics, including acceptable alternatives and depth requirements, so that AI judgments are accurate and teaching assistants can audit disagreements easily. Cognitive alignment refers to decomposing feedback by rubric item and surfacing evidence, judgments, and suggestions in a single pane in order to minimize context-switching and cognitive load. Human-in-the-loop controls require that raw AI text not be piped directly to students, and that teaching assistants explicitly review, adopt, and edit each suggestion. Iterative refinement uses teaching-assistant disagreements and edits to identify under-specified rubrics and improve prompts or future fine-tuning (Lu et al., 18 Feb 2026).

The study also states several limitations. It relied on an LLM-based rubric scorer, although that scorer achieved 84.6% accuracy versus a human expert. It did not include an AI-only feedback arm. It also reported limited dosage for post-test learning gains (Lu et al., 18 Feb 2026). These limitations matter because they constrain the scope of the causal claims: the trial supports AI-mediated oversight-rich feedback, not autonomous feedback delivery, and it supports revision-quality gains more directly than durable learning gains.

The stated future directions are AI-only versus AI-mediated comparison, rubric-prompt auto-tuning by mining teaching-assistant edits and flips, and longitudinal deployments across multiple courses to assess sustained learning benefits and workload trade-offs (Lu et al., 18 Feb 2026). In the context of related work, a plausible implication is that future FeedbackWriter systems may increasingly combine the rubric-grounded mediation of the economics-course deployment with the reward modeling, dynamic criteria generation, and iterative revision mechanisms explored in adjacent writing-assistance research (Chakrabarty et al., 10 Apr 2025, Zhao et al., 16 Mar 2026).