REVA: Human-AI Feedback Validation

• REVA is a human-AI system that validates LLM-generated programming feedback by sequencing submissions to reduce cognitive context shifts and applying instructor-driven revisions.
• The system employs attention-based sequencing to reorder review items and batch similar errors, streamlining feedback validation in large programming classes.
• Empirical evaluations indicate that REVA reduces validation time and improves misconception coverage precision and recall, leading to higher overall feedback quality.

REVA is a human-AI system for validating large volumes of LLM-generated programming feedback by sequencing submissions to minimize cognitive context shifts and by propagating instructor-driven revisions across semantically similar instances. It is designed for programming education settings in which LLMs can generate personalized feedback at scale but the resulting outputs are not trustworthy enough to send directly to students without instructor inspection, correction, and adaptation. Its central technical premise is that review is not a set of independent items: many code-feedback pairs are semantically similar, and instructors expose actionable attention signals through highlighting and selection during review. REVA operationalizes those signals into an adaptive workflow that reorders upcoming items, supports in-situ revision, and reuses validated edits while preserving instructor control and pedagogical judgment (Tang et al., 15 Jul 2025).

1. Problem formulation and design rationale

REVA addresses the validation bottleneck that arises when LLM-generated programming feedback must be reviewed across large classes. The paper frames the bottleneck as a task-switching or context-switching problem: each move between different programming concepts, code patterns, and pedagogical judgments incurs cognitive switching costs. In that formulation, the primary systems objective is not autonomous grading, but reduction of repetitive cognitive work while keeping instructors in control of final feedback (Tang et al., 15 Jul 2025).

The design rationale emerged from a formative study with 20 instructors in a between-subject study using real classroom data from a large programming course. Participants interacted with a system that allowed component-based feedback generation and visual augmentations, and the resulting findings were distilled into three design patterns. First, instructors develop implicit mental models for prioritizing feedback content, often focusing on severity or on the pedagogical value of individual feedback components. Second, they prefer to review similar issues in batches to reduce context switching, but coarse clustering is insufficient. Third, they employ generalizable revision strategies across similar instances, especially abstraction changes and reusable phrasing. These observations became the system’s three design goals: attention-based seamless review, reusable revisions, and flexible revision support.

A common misconception is that the system primarily automates feedback authoring. The paper instead positions REVA around validation: LLMs draft the feedback, but instructors inspect, revise, accept, or reject outputs. The system therefore targets review throughput and feedback quality under human oversight rather than end-to-end automatic generation.

2. Workflow and feedback representation

REVA organizes the end-to-end process into three stages. Instructors first upload student submissions and generate structured feedback drafts with an LLM. They then review and revise the AI-generated feedback in REVA. Finally, they send the validated feedback to students (Tang et al., 15 Jul 2025).

A central representation is the component-based feedback template, inspired by Hattie and Timperley’s feedback model. The template structures each feedback instance into five components and is intended to make the output easier to inspect and ground in source evidence.

Component	Role in the template
Issue	What the problem is
Strategy	What has been done
Solution	What should be done
Example	Supporting illustration
Next step	Follow-up action

The interface shows highlighting that traces which pieces of code or conversation support each component. This grounding mechanism is important because the paper treats inspectability as a prerequisite for scalable validation. Rather than presenting feedback as a monolithic paragraph, REVA exposes a structured decomposition that supports targeted revision of misconception coverage, abstraction, and tone.

This structure also clarifies the division of labor between model output and instructor judgment. The LLM provides a draft consistent with the template; the instructor validates whether the components are pedagogically appropriate, sufficiently specific, and supported by the underlying submission.

3. Attention-based sequencing

The first major REVA mechanism is user attention-based adaptation. The system infers what the instructor is currently paying attention to from interactions, especially highlighting and selection actions, and uses those signals to adapt the review queue (Tang et al., 15 Jul 2025).

In practice, instructors create semantic filters from what they highlight in code or feedback. A filter can be based on code content or feedback content and represents a concept the instructor wants to focus on, such as a misconception about list concatenation. When the instructor highlights a problematic fragment and clicks “Add Filter,” REVA uses an off-the-shelf LLM to interpret the selected context and infer the semantic intent of that selection. The resulting filter is then used to reorder the queue so that similar code-feedback pairs are reviewed next.

The paper emphasizes that this is not simple clustering. The filter is tied to the instructor’s current focus and can be applied dynamically during review. Similar items are highlighted in the list, and relevant matches within code or feedback are visually emphasized. The intended effect is to keep the instructor in one conceptual region for longer, thereby reducing context shifts and shortening the time needed to orient to a new item.

Qualitative findings indicate that this mechanism changes review behavior as well as queue order. Participants reported that semantic filters helped them narrow attention to a specific misconception and avoid bouncing between unrelated errors. Some preferred predefined filters at the beginning of a session because they reduced decision fatigue and provided an initial scaffold before more targeted semantic filtering.

4. Revision propagation and flexible editing

The second major REVA mechanism is revision propagation. The paper treats a validated edit as a reusable action rather than a one-off local correction. When an instructor makes a revision and accepts it, REVA extracts the high-level revision goal together with the code and feedback patterns involved, searches upcoming code-feedback pairs for matches, and applies the same or analogous revision automatically, subject to instructor verification (Tang et al., 15 Jul 2025).

This design targets recurrent error modes in programming assignments. If many students omit a return statement, for example, the instructor need not repeatedly rewrite the same correction. Instead, the system proposes propagated revisions that the instructor reviews and accepts or rejects. The paper presents this as a way to convert individual edits into reusable review operations.

REVA supports several revision classes. It can revise content-level errors by adding or correcting misconception coverage. It can revise abstraction level, making feedback more specific or more conceptual depending on the severity of the issue and the instructor’s pedagogical goals. It can also revise personal style, including tone, encouragement, or grammar. Both general revision queries and in-situ revision are supported: instructors can highlight the relevant code, problem statement, or feedback text and ask the system to revise on the basis of that selection.

The interface includes shortcuts for common operations such as mentioning issues, expanding feedback, removing feedback, or adjusting abstraction. Proposed revisions are shown inline with a description of what changed, and each change requires instructor acceptance or rejection. That acceptance step is central to the system’s control model. A common misunderstanding would be to treat propagation as unchecked automation; the paper explicitly describes it as supervised reuse of validated edits.

5. Architecture and empirical evaluation

REVA is implemented as a web-based system using React and FastAPI, with GPT-4o and text-embedding-3-large for LLM and similarity-related features, plus Firebase for interaction logging. The prompting pipeline uses persona settings to make the model behave like an instructor, chain-style prompting for multi-step tasks, and few-shot prompts to improve performance. The prompts were iteratively engineered and stress-tested, with response time around three seconds and a low error rate in trial runs. The paper states that hallucination risk is bounded by template constraints and mandatory human verification (Tang et al., 15 Jul 2025).

The main evaluation used a within-subjects lab study with 12 instructors from a four-year university, all with programming and teaching experience. Each participant used both the baseline system and REVA, with order and datasets counterbalanced. The task was to validate and send 30 feedback instances within 20 minutes per condition using authentic code submissions from introductory programming courses. The baseline retained component-based generation and predefined filters but lacked semantic filters and revision propagation. Participants completed NASA-TLX and semi-structured interviews, and the study logged 475 validated feedback messages and 840 revisions.

Measure	Baseline	REVA
Average validation time per code-feedback pair	88.9 s	79.7 s
Average number of revisions	26.0	44.0
Precision for misconception coverage	0.71	0.90
Recall for misconception coverage	0.55	0.86
Reaction time before first concrete action	29.8 s	18.7 s

The paper reports that REVA produced significantly more revisions and significantly better overall feedback quality than the baseline. NASA-TLX results suggested relatively low perceived mental and physical demand with REVA, and the authors observed that participants remained more engaged in revision throughout the session. The reaction-time measure is particularly important because it serves as a proxy for context-switching cost, directly connecting the system’s attention-based sequencing design to an observable behavioral outcome.

6. Interpretation, limitations, and broader relevance

The qualitative results help explain the quantitative gains. Participants valued semantic filters because they supported misconception-focused batching, in-situ revision because it felt more natural than switching among separate tools or rewriting prompts from scratch, and abstraction-level control because it supported calibration of feedback detail to student needs. Revision propagation was especially salient because it eliminated repetitive work while preserving review and acceptance at the point of use (Tang et al., 15 Jul 2025).

The paper interprets REVA as an instance of reciprocal human-AI collaboration. As the system learns from instructor actions, instructors also adapt their reviewing strategies, using propagated edits as cognitive anchors for future items. This suggests a broader systems principle: attention signals such as highlighting and selection can be a general design pattern for large-scale review tasks involving semantically similar items and expensive validation. The authors explicitly suggest essay grading, legal review, patent analysis, and medical chart work as domains in which the same principles may apply.

Several limitations constrain interpretation. The evaluation was conducted in a lab rather than a real grading environment, so ecological validity is limited. Each condition lasted only 20 minutes, much shorter than an actual course grading session. The sample size was small, and the study focused on instructor outcomes rather than student learning outcomes. The paper also notes that REVA does not implement a fully global task-sequencing algorithm; it uses attention-based adaptation without sophisticated end-to-end optimization of review order. Some participants experienced higher early load because initial items happened to contain particularly complex misconceptions. More fundamentally, the paper stresses that LLMs still lack robust pedagogical judgment, context awareness, and deterministic reliability.

Future work is framed around more realistic deployment and richer integration with educational workflows. The authors propose studying student outcomes after instructors validate LLM feedback, examining how validated feedback can inform instructors later, exploring deeper combinations of deterministic program-analysis techniques with LLMs, and testing the system in longer-form grading settings. In that sense, REVA functions less as a fixed product than as a blueprint for scalable human-AI collaboration in review-intensive tasks: systems should learn from human attention and revision intent, reorder work to match human cognition, and reuse validated edits across semantically similar cases.