AGACCI: Grading Agents for Coding Assignments

• AGACCI is a multi-agent framework for rubric-centric grading of Jupyter notebook submissions, integrating code execution, visualization, and interpretive reasoning.
• It decomposes the assessment into specialized tasks such as technical validation, quantitative result evaluation, and qualitative feedback generation.
• Empirical results demonstrate that AGACCI outperforms single-model baselines, offering enhanced rubric accuracy and more consistent, pedagogically meaningful feedback.

AGACCI, short for Affiliated Grading Agents for Criteria-Centric Interface, is a multi-agent evaluation framework for educational coding assignments, especially Jupyter Notebooks that combine code, executable outputs, graphs/visualizations, and explanatory markdown text (Park et al., 7 Jul 2025). Its target problem is criteria-centric educational assessment: instructors specify rubric items, and the system must determine whether each criterion is satisfied while also generating qualitative feedback that is specific, pedagogically meaningful, and aligned with instructional intent. The framework addresses settings in which grading depends simultaneously on technical validity, observable quantitative results, visual communication, and interpretive reasoning, rather than on code correctness alone (Park et al., 7 Jul 2025).

1. Definition, scope, and disambiguation

AGACCI was introduced as a framework for assessing code-based assignments that may require reasoning over whether code is present and functionally plausible, whether measurable outputs or performance thresholds are achieved, whether visualizations are appropriate and interpretable, and whether written explanations reflect actual reasoning rather than superficial description (Park et al., 7 Jul 2025). The paper positions the system against standard VLM or single-agent LLM approaches, which are described as struggling with generic praise, hallucinated reasoning, weak alignment to fine-grained rubrics, and unstable judgments across repeated runs (Park et al., 7 Jul 2025).

The acronym is specific to the 2025 paper "AGACCI : Affiliated Grading Agents for Criteria-Centric Interface in Educational Coding Contexts" (Park et al., 7 Jul 2025). It does not appear in the 2022 paper "Attribute-Guided Multi-Level Attention Network for Fine-Grained Fashion Retrieval", which instead introduces AG-MAN and AGA for fine-grained fashion retrieval (Xiao et al., 2022). That paper explicitly contains no occurrence of AGACCI in its title, abstract, main text, figure captions, tables, appendix, code reference, or cited related work, making AG-MAN or AGA the most plausible source of acronym confusion when AG-prefixed attention methods are discussed in unrelated contexts (Xiao et al., 2022).

This separation is substantive rather than nominal. AG-MAN concerns attribute-specific fashion retrieval with a ResNet50 backbone and an attribute-guided attention design (Xiao et al., 2022), whereas AGACCI concerns rubric-based educational assessment of multimodal notebook submissions (Park et al., 7 Jul 2025).

2. Problem setting and motivation

The framework is designed for assignments in which useful grading cannot be reduced to a single pass/fail execution check. The paper emphasizes that educational coding tasks often involve executable artifacts and measurable outcomes, so assessment may depend on whether code runs, produces expected outputs, reaches performance thresholds, visualizes results properly, and includes thoughtful interpretation (Park et al., 7 Jul 2025).

This leads to several evaluative layers that AGACCI treats as distinct but coordinated:

• Technical validity: whether the code runs and performs the expected steps.
• Observable results: whether reported metrics or outputs satisfy quantitative rubric criteria.
• Presentation and communication: whether charts, labels, legends, and visual structure support understanding.
• Interpretive reasoning: whether the student explains patterns, anomalies, or implications rather than merely restating outputs.
• Rubric grounding: whether feedback is explicitly tied to the instructor’s criteria (Park et al., 7 Jul 2025).

The paper argues that single-model prompting tends to compress all of these subproblems into one response, which can blur evaluative dimensions and produce mismatches between evidence, rubric decisions, and final feedback (Park et al., 7 Jul 2025). AGACCI’s response is to distribute these functions across specialized agents and then reconcile them through a meta-level checking and aggregation stage. A plausible implication is that the framework treats grading as a coordination problem over heterogeneous evidence types rather than as a single generation task.

3. Multi-agent architecture and workflow

AGACCI is described as a pipeline of modular agents with both parallel specialization and hierarchical control (Park et al., 7 Jul 2025). The workflow is explicitly summarized as:

1. Rubric interpretation
2. Submission analysis
3. Parallel evaluation
   • execution and result analysis
   • visualization assessment
   • interpretation assessment
4. Meta-level consistency checking
5. Final judging and aggregation
6. Structured summarization for delivery (Park et al., 7 Jul 2025)

The system is organized around a criteria-centric interface, meaning that the rubric is first transformed into structured operational targets and that downstream judgments are expected to refer back to those targets rather than to general impressions of the submission (Park et al., 7 Jul 2025).

Agent	Primary function	Output focus
Rubric Interpreter	Converts rubric descriptions into operational evaluation targets	objectives, prerequisites, subgoals, evidence types
Submission Analyzer	Maps notebook contents onto rubric-relevant structure	main goal, logical flow, subgoal coverage
Execution Evaluator	Checks functional validity	run status, core steps, expected outputs
Result Evaluator	Assesses quantitative rubric criteria	binary pass/fail with metric-based justification
Visualization Evaluator	Judges charts and graphs	interpretability and communicative adequacy
Interpretation Evaluator	Assesses reasoning quality	explanation, inference, rigor, justification
Meta Evaluator	Checks cross-agent consistency	contradiction flags, override suggestions, confidence adjustments
Final Judge	Aggregates prior outputs	binary rubric satisfaction and human-readable feedback
Summarizer	Produces compact learner-facing output	scores, findings, recommendations

The Rubric Interpreter does not treat rubrics as static checklists. It identifies implicit dependencies, ordering constraints, and minimum performance expectations, and the appendix shows JSON-like outputs containing "final_objective", "prerequisite_items", "subgoals", and "evidence_types" (Park et al., 7 Jul 2025). The Submission Analyzer then reads the notebook holistically and identifies the main goal, logical structure, purpose and sequence of code blocks, commentary and outputs, and alignment with rubric criteria (Park et al., 7 Jul 2025).

The three parallel evaluators specialize over different evidence modalities. The Execution Evaluator checks whether code runs without errors and produces expected outputs; the Result Evaluator determines whether quantitative criteria such as accuracy thresholds, leaderboard scores, or other numeric outputs are satisfied; the Visualization Evaluator judges whether visual outputs match the data and effectively communicate insights; and the Interpretation Evaluator checks for genuine reasoning, including causal or inferential explanations, meaningful treatment of patterns or anomalies, and logical rigor (Park et al., 7 Jul 2025).

The Meta Evaluator then cross-validates specialized outputs and flags contradictions, unsupported assessments, and weak evidence-to-judgment links, possibly suggesting overrides or confidence adjustments (Park et al., 7 Jul 2025). The Final Judge resolves ambiguities across agents, determines binary rubric satisfaction scores, and produces human-readable feedback, after which the Summarizer converts the decision into a compact format, possibly JSON, containing essential findings, recommendations, rubric scores, and structured output suitable for logging or display (Park et al., 7 Jul 2025).

The paper does not provide a formal coordination protocol, debate mechanism, voting rule, consensus algorithm, pseudocode, or optimization objective (Park et al., 7 Jul 2025). Coordination is therefore procedural rather than mathematically formalized.

4. Inputs, representations, and formalization boundaries

AGACCI is intended for multifaceted student submissions, especially Jupyter notebooks, and the paper explicitly mentions handling code, graphs / visualizations, markdown explanations, and other multimodal artifacts (Park et al., 7 Jul 2025). In the experiments, the system input consists of the student’s code and the corresponding rubric, while the evaluative evidence may include printed metrics, logs, plots, and notebook outputs (Park et al., 7 Jul 2025).

Rubrics begin as natural-language descriptions written by instructional staff. The Rubric Interpreter converts them into structured operational representations, with example fields including objective, prerequisites, subgoals, and evidence types (Park et al., 7 Jul 2025). Outputs produced by AGACCI include:

• binary rubric scores for each rubric item;
• textual feedback summarizing strengths, weaknesses, and suggestions;
• a structured summary, potentially in JSON (Park et al., 7 Jul 2025).

The paper notes a compact score format such as Score: 111, which appears to encode pass/fail outcomes across three rubric items, but the exact encoding rule is not formally specified (Park et al., 7 Jul 2025).

Formalization is intentionally limited. The paper states that AGACCI has no stated loss functions, no optimization objective, no formal aggregation equation, and no theorem-level derivation (Park et al., 7 Jul 2025). The most formal aspects are instead the evaluation setup:

• each rubric item is a binary decision, 0 or 1;
• because each assignment has three rubric items, the task is treated as a multi-label binary classification problem;
• the primary quantitative metric is average rubric accuracy across submissions;
• qualitative feedback is evaluated on a 5-point scale over Feedback Accuracy, Relevance to Rubrics, Consistency, and Coherence;
• to stabilize LLM-based scoring, each comment is evaluated with a G-Eval-style prompt 20 times, and the average is used;
• both AGACCI and the baseline are run for six independent evaluation rounds because of stochastic output variation (Park et al., 7 Jul 2025).

A plausible implication is that AGACCI should be understood as a systems and evaluation contribution rather than as a mathematically specified learning algorithm.

5. Experimental design and empirical results

The experiments use a real university-course dataset comprising 360 submissions from 60 participants, where each participant completed six graduate-level programming assignments (Park et al., 7 Jul 2025). The assignments span machine learning, computer vision, and natural language processing, with the six referenced task domains being:

• ML: regression task
• CV1: face detection / sticker alignment task
• CV2: segmentation / portrait mode task
• NLP1: text classification
• NLP2: summarization
• NLP3: chatbot interaction (Park et al., 7 Jul 2025)

Each assignment had a domain-specific rubric created by instructional staff, with three evaluation criteria per assignment, and domain experts annotated the submissions with binary rubric scores and qualitative feedback (Park et al., 7 Jul 2025). All experimental artifacts were written in Korean (Park et al., 7 Jul 2025).

The main baseline is a single GPT-based baseline, referred to as SLI, and both AGACCI and SLI use the same backbone model, GPT-4o mini, so the comparison isolates the effect of multi-agent structure rather than model size (Park et al., 7 Jul 2025). For qualitative scoring, the evaluation model is GPT-4o, selected because the data are in Korean and the authors cite its strong Korean capability (Park et al., 7 Jul 2025).

The headline result is that AGACCI achieved 12 percentage points higher average rubric accuracy than the single-model baseline (Park et al., 7 Jul 2025). The paper reports consistent outperformance across the six assignment domains, with especially strong gains in NLP tasks, which it attributes to the importance of interpretive reasoning and coherent language generation in those assignments (Park et al., 7 Jul 2025).

Several criterion-level gains are explicitly reported:

• ML: Preprocessing, training, and visualization — AGACCI: 0.7337 ± 0.0978; SLI: 0.1739 ± 0.0178
• CV1: Natural sticker alignment on face — AGACCI: 0.7455 ± 0.0268; SLI: 0.1786 ± 0.0484
• CV2: Solution for portrait mode errors — AGACCI: 0.6798 ± 0.0439; SLI: 0.3860 ± 0.0516
• CV2: Clear identification of portrait errors — AGACCI: 0.6535 ± 0.0088; SLI: 0.4342 ± 0.0461
• NLP1: Improved accuracy using Word2Vec — AGACCI: 0.6509 ± 0.0189; SLI: 0.4057 ± 0.0109
• NLP2: Extractive vs. abstractive comparison — AGACCI: 0.8673 ± 0.0204; SLI: 0.4541 ± 0.0510
• NLP3: Stable Transformer convergence — AGACCI: 0.9694 ± 0.0204; SLI: 0.5765 ± 0.0918
• NLP3: Korean response generation model — AGACCI: 0.9592 ± 0.0000; SLI: 0.5102 ± 0.0957 (Park et al., 7 Jul 2025)

The paper links these gains to rubric items requiring multi-step reasoning, visual judgment, error diagnosis, comparative analysis, or deep-learning process understanding (Park et al., 7 Jul 2025).

The qualitative evaluation shows that AGACCI outperformed the baseline on most dimensions. It had a higher mean and lower variance on Consistency, produced more integrated feedback on Coherence, and scored higher on Feedback Accuracy (Park et al., 7 Jul 2025). On Relevance to Rubrics, the systems performed comparably, although AGACCI showed somewhat reduced variance (Park et al., 7 Jul 2025). The authors suggest that AGACCI sometimes goes beyond strict rubric satisfaction by adding pedagogically helpful advice or broader analysis, which may aid educational usefulness while appearing as off-rubric elaboration under a narrow relevance metric (Park et al., 7 Jul 2025).

6. Limitations, implementation context, and broader significance

The reported gains are not uniform. The paper identifies rubric items on which the baseline performs better:

• ML: Kaggle submission status — AGACCI: 0.4728 ± 0.0109; SLI: 0.5870 ± 0.0589
• ML: Leaderboard accuracy threshold met — AGACCI: 0.2391 ± 0.0000; SLI: 0.6848 ± 0.0416
• CV1: Implementation of sticker compositing — AGACCI: 0.7232 ± 0.0536; SLI: 0.8795 ± 0.0225
• CV2: Portrait mode implementation — AGACCI: 0.5570 ± 0.0088; SLI: 0.8114 ± 0.0088
• NLP1: Embedding comparison and analysis — AGACCI: 0.4057 ± 0.0189; SLI: 0.5802 ± 0.0322
• NLP2: Preprocessing for summarization pipeline — AGACCI: 0.9286 ± 0.0204; SLI: 0.9643 ± 0.0195 (Park et al., 7 Jul 2025)

The paper emphasizes one recurring cause: some rubric items depend on external actions or external evidence, such as actual Kaggle submission and leaderboard verification, which may not be inferable from notebook code alone unless explicitly documented (Park et al., 7 Jul 2025). AGACCI is therefore strongest when evidence for a criterion is present within the submission itself and weaker when rubric satisfaction depends on hidden or off-platform verification (Park et al., 7 Jul 2025).

Implementation is carried out on top of AutoGen, which is used for multi-agent orchestration (Park et al., 7 Jul 2025). Both AGACCI and SLI use GPT-4o mini, a choice justified by limited computational resources, budget constraints, the need for responsive classroom feedback, sufficient reasoning ability for rubric-aligned educational assessment, and efficient support for parallel multi-agent deployment (Park et al., 7 Jul 2025). The appendix includes prompts for the agents and for qualitative evaluation, but the supplied text does not include their full wording (Park et al., 7 Jul 2025). Likewise, the paper does not specify a sandboxed execution environment, exact notebook runner implementation, token accounting, runtime, or latency numbers (Park et al., 7 Jul 2025).

The principal limitations acknowledged are dependence on observable in-submission evidence, difficulty with ambiguous or conflicting rubrics, lack of validated generalization beyond code- and visualization-centric assignments, absence of formal robustness analysis, missing cost/runtime accounting, and the constrained dataset scope of 60 participants, 360 submissions, and a specific Korean curricular context (Park et al., 7 Jul 2025). Future directions proposed in the paper include dynamic calibration or clarification mechanisms, tighter human-in-the-loop feedback loops, extension to more open-ended text assignments, and study of learner trust, perceived clarity, and behavioral impact of the generated feedback (Park et al., 7 Jul 2025).

Within the literature represented here, AGACCI is best understood as a criteria-centric, modular grading system for complex notebook-based assignments rather than as a general-purpose auto-grader or a mathematically formalized multi-agent learning framework (Park et al., 7 Jul 2025). Its central claim is empirical: distributing rubric interpretation, submission analysis, execution checking, quantitative result evaluation, visualization judgment, interpretive assessment, consistency validation, and final aggregation across affiliated agents can improve rubric accuracy, feedback accuracy, consistency, and coherence relative to a matched single-model baseline in educational coding contexts (Park et al., 7 Jul 2025).