---
title: 'AGACCI: Grading Agents for Coding Assignments'
url: https://www.emergentmind.com/topics/agacci
type: topic
---

# AGACCI: Grading Agents for Coding Assignments

AGACCI, short for **Affiliated Grading Agents for Criteria-Centric Interface**, is a **multi-agent evaluation framework** for **educational coding assignments**, especially **Jupyter Notebooks** that combine **code, executable outputs, graphs/visualizations, and explanatory markdown text** [2507.05321]. Its target problem is **criteria-centric educational assessment**: instructors specify **rubric items**, and the system must determine whether each criterion is satisfied while also generating **qualitative feedback** that is specific, pedagogically meaningful, and aligned with instructional intent. The framework addresses settings in which grading depends simultaneously on **technical validity**, **observable quantitative results**, **visual communication**, and **interpretive reasoning**, rather than on code correctness alone [2507.05321].

## 1. Definition, scope, and disambiguation

AGACCI was introduced as a framework for assessing **code-based assignments** that may require reasoning over whether code is present and functionally plausible, whether measurable outputs or performance thresholds are achieved, whether visualizations are appropriate and interpretable, and whether written explanations reflect actual reasoning rather than superficial description [2507.05321]. The paper positions the system against **standard VLM or single-agent LLM approaches**, which are described as struggling with generic praise, hallucinated reasoning, weak alignment to fine-grained rubrics, and unstable judgments across repeated runs [2507.05321].

The acronym is specific to the 2025 paper **"AGACCI : Affiliated Grading Agents for Criteria-Centric Interface in Educational Coding Contexts"** [2507.05321]. It does **not** appear in the 2022 paper **"Attribute-Guided Multi-Level Attention Network for Fine-Grained Fashion Retrieval"**, which instead introduces **AG-MAN** and **AGA** for fine-grained fashion retrieval [2301.13014]. That paper explicitly contains **no occurrence** of **AGACCI** in its title, abstract, main text, figure captions, tables, appendix, code reference, or cited related work, making **AG-MAN** or **AGA** the most plausible source of acronym confusion when AG-prefixed attention methods are discussed in unrelated contexts [2301.13014].

This separation is substantive rather than nominal. AG-MAN concerns **attribute-specific fashion retrieval** with a **ResNet50** backbone and an **attribute-guided attention** design [2301.13014], whereas AGACCI concerns **rubric-based educational assessment** of multimodal notebook submissions [2507.05321].

## 2. Problem setting and motivation

The framework is designed for assignments in which useful grading cannot be reduced to a single pass/fail execution check. The paper emphasizes that educational coding tasks often involve **executable artifacts** and **measurable outcomes**, so assessment may depend on whether code runs, produces expected outputs, reaches performance thresholds, visualizes results properly, and includes thoughtful interpretation [2507.05321].

This leads to several evaluative layers that AGACCI treats as distinct but coordinated:

- **Technical validity**: whether the code runs and performs the expected steps.
- **Observable results**: whether reported metrics or outputs satisfy quantitative rubric criteria.
- **Presentation and communication**: whether charts, labels, legends, and visual structure support understanding.
- **Interpretive reasoning**: whether the student explains patterns, anomalies, or implications rather than merely restating outputs.
- **Rubric grounding**: whether feedback is explicitly tied to the instructor’s criteria [2507.05321].

The paper argues that single-model prompting tends to compress all of these subproblems into one response, which can blur evaluative dimensions and produce mismatches between evidence, rubric decisions, and final feedback [2507.05321]. AGACCI’s response is to distribute these functions across **specialized agents** and then reconcile them through a meta-level checking and aggregation stage. A plausible implication is that the framework treats grading as a coordination problem over heterogeneous evidence types rather than as a single generation task.

## 3. Multi-agent architecture and workflow

AGACCI is described as a **pipeline of modular agents** with both **parallel specialization** and **hierarchical control** [2507.05321]. The workflow is explicitly summarized as:

1. **Rubric interpretation**
2. **Submission analysis**
3. **Parallel evaluation**
   - execution and result analysis
   - visualization assessment
   - interpretation assessment
4. **Meta-level consistency checking**
5. **Final judging and aggregation**
6. **Structured summarization for delivery** [2507.05321]

The system is organized around a **criteria-centric interface**, meaning that the rubric is first transformed into structured operational targets and that downstream judgments are expected to refer back to those targets rather than to general impressions of the submission [2507.05321].

| Agent | Primary function | Output focus |
|---|---|---|
| Rubric Interpreter | Converts rubric descriptions into operational evaluation targets | objectives, prerequisites, subgoals, evidence types |
| Submission Analyzer | Maps notebook contents onto rubric-relevant structure | main goal, logical flow, subgoal coverage |
| Execution Evaluator | Checks functional validity | run status, core steps, expected outputs |
| Result Evaluator | Assesses quantitative rubric criteria | binary pass/fail with metric-based justification |
| Visualization Evaluator | Judges charts and graphs | interpretability and communicative adequacy |
| Interpretation Evaluator | Assesses reasoning quality | explanation, inference, rigor, justification |
| Meta Evaluator | Checks cross-agent consistency | contradiction flags, override suggestions, confidence adjustments |
| Final Judge | Aggregates prior outputs | binary rubric satisfaction and human-readable feedback |
| Summarizer | Produces compact learner-facing output | scores, findings, recommendations |

The **Rubric Interpreter** does not treat rubrics as static checklists. It identifies **implicit dependencies**, **ordering constraints**, and **minimum performance expectations**, and the appendix shows JSON-like outputs containing `"final_objective"`, `"prerequisite_items"`, `"subgoals"`, and `"evidence_types"` [2507.05321]. The **Submission Analyzer** then reads the notebook holistically and identifies the **main goal**, **logical structure**, **purpose and sequence of code blocks**, **commentary and outputs**, and **alignment with rubric criteria** [2507.05321].

The three parallel evaluators specialize over different evidence modalities. The **Execution Evaluator** checks whether code runs without errors and produces expected outputs; the **Result Evaluator** determines whether quantitative criteria such as **accuracy thresholds**, **leaderboard scores**, or other numeric outputs are satisfied; the **Visualization Evaluator** judges whether visual outputs match the data and effectively communicate insights; and the **Interpretation Evaluator** checks for genuine reasoning, including **causal or inferential explanations**, meaningful treatment of patterns or anomalies, and logical rigor [2507.05321].

The **Meta Evaluator** then cross-validates specialized outputs and flags **contradictions**, **unsupported assessments**, and **weak evidence-to-judgment links**, possibly suggesting **overrides** or **confidence adjustments** [2507.05321]. The **Final Judge** resolves ambiguities across agents, determines **binary rubric satisfaction scores**, and produces human-readable feedback, after which the **Summarizer** converts the decision into a compact format, possibly **JSON**, containing essential findings, recommendations, rubric scores, and structured output suitable for logging or display [2507.05321].

The paper does **not** provide a formal coordination protocol, debate mechanism, voting rule, consensus algorithm, pseudocode, or optimization objective [2507.05321]. Coordination is therefore procedural rather than mathematically formalized.

## 4. Inputs, representations, and formalization boundaries

AGACCI is intended for **multifaceted student submissions**, especially **Jupyter notebooks**, and the paper explicitly mentions handling **code**, **graphs / visualizations**, **markdown explanations**, and other **multimodal artifacts** [2507.05321]. In the experiments, the system input consists of **the student’s code** and **the corresponding rubric**, while the evaluative evidence may include printed metrics, logs, plots, and notebook outputs [2507.05321].

Rubrics begin as natural-language descriptions written by instructional staff. The **Rubric Interpreter** converts them into structured operational representations, with example fields including objective, prerequisites, subgoals, and evidence types [2507.05321]. Outputs produced by AGACCI include:

- **binary rubric scores** for each rubric item;
- **textual feedback** summarizing strengths, weaknesses, and suggestions;
- a **structured summary**, potentially in JSON [2507.05321].

The paper notes a compact score format such as `Score: 111`, which appears to encode pass/fail outcomes across three rubric items, but the exact encoding rule is **not formally specified** [2507.05321].

Formalization is intentionally limited. The paper states that AGACCI has **no stated loss functions**, **no optimization objective**, **no formal aggregation equation**, and **no theorem-level derivation** [2507.05321]. The most formal aspects are instead the evaluation setup:

- each rubric item is a **binary decision**, 0 or 1;
- because each assignment has **three rubric items**, the task is treated as a **multi-label binary classification problem**;
- the primary quantitative metric is **average rubric accuracy across submissions**;
- qualitative feedback is evaluated on a **5-point scale** over **Feedback Accuracy**, **Relevance to Rubrics**, **Consistency**, and **Coherence**;
- to stabilize LLM-based scoring, each comment is evaluated with a **G-Eval-style prompt 20 times**, and the average is used;
- both AGACCI and the baseline are run for **six independent evaluation rounds** because of stochastic output variation [2507.05321].

A plausible implication is that AGACCI should be understood as a systems and evaluation contribution rather than as a mathematically specified learning algorithm.

## 5. Experimental design and empirical results

The experiments use a real university-course dataset comprising **360 submissions** from **60 participants**, where each participant completed **six graduate-level programming assignments** [2507.05321]. The assignments span **machine learning**, **computer vision**, and **natural language processing**, with the six referenced task domains being:

- **ML**: regression task
- **CV1**: face detection / sticker alignment task
- **CV2**: segmentation / portrait mode task
- **NLP1**: text classification
- **NLP2**: summarization
- **NLP3**: chatbot interaction [2507.05321]

Each assignment had a **domain-specific rubric** created by instructional staff, with **three evaluation criteria** per assignment, and domain experts annotated the submissions with **binary rubric scores** and **qualitative feedback** [2507.05321]. All experimental artifacts were written in **Korean** [2507.05321].

The main baseline is a **single GPT-based baseline**, referred to as **SLI**, and both AGACCI and SLI use the same backbone model, **GPT-4o mini**, so the comparison isolates the effect of **multi-agent structure** rather than model size [2507.05321]. For qualitative scoring, the evaluation model is **GPT-4o**, selected because the data are in Korean and the authors cite its strong Korean capability [2507.05321].

The headline result is that **AGACCI achieved 12 percentage points higher average rubric accuracy** than the single-model baseline [2507.05321]. The paper reports consistent outperformance across the six assignment domains, with especially strong gains in **NLP tasks**, which it attributes to the importance of **interpretive reasoning** and **coherent language generation** in those assignments [2507.05321].

Several criterion-level gains are explicitly reported:

- **ML: Preprocessing, training, and visualization** — AGACCI: **0.7337 ± 0.0978**; SLI: **0.1739 ± 0.0178**
- **CV1: Natural sticker alignment on face** — AGACCI: **0.7455 ± 0.0268**; SLI: **0.1786 ± 0.0484**
- **CV2: Solution for portrait mode errors** — AGACCI: **0.6798 ± 0.0439**; SLI: **0.3860 ± 0.0516**
- **CV2: Clear identification of portrait errors** — AGACCI: **0.6535 ± 0.0088**; SLI: **0.4342 ± 0.0461**
- **NLP1: Improved accuracy using Word2Vec** — AGACCI: **0.6509 ± 0.0189**; SLI: **0.4057 ± 0.0109**
- **NLP2: Extractive vs. abstractive comparison** — AGACCI: **0.8673 ± 0.0204**; SLI: **0.4541 ± 0.0510**
- **NLP3: Stable Transformer convergence** — AGACCI: **0.9694 ± 0.0204**; SLI: **0.5765 ± 0.0918**
- **NLP3: Korean response generation model** — AGACCI: **0.9592 ± 0.0000**; SLI: **0.5102 ± 0.0957** [2507.05321]

The paper links these gains to rubric items requiring **multi-step reasoning**, **visual judgment**, **error diagnosis**, **comparative analysis**, or **deep-learning process understanding** [2507.05321].

The qualitative evaluation shows that AGACCI outperformed the baseline on most dimensions. It had a **higher mean** and **lower variance** on **Consistency**, produced more integrated feedback on **Coherence**, and scored higher on **Feedback Accuracy** [2507.05321]. On **Relevance to Rubrics**, the systems performed comparably, although AGACCI showed somewhat reduced variance [2507.05321]. The authors suggest that AGACCI sometimes goes beyond strict rubric satisfaction by adding pedagogically helpful advice or broader analysis, which may aid educational usefulness while appearing as off-rubric elaboration under a narrow relevance metric [2507.05321].

## 6. Limitations, implementation context, and broader significance

The reported gains are not uniform. The paper identifies rubric items on which the baseline performs better:

- **ML: Kaggle submission status** — AGACCI: **0.4728 ± 0.0109**; SLI: **0.5870 ± 0.0589**
- **ML: Leaderboard accuracy threshold met** — AGACCI: **0.2391 ± 0.0000**; SLI: **0.6848 ± 0.0416**
- **CV1: Implementation of sticker compositing** — AGACCI: **0.7232 ± 0.0536**; SLI: **0.8795 ± 0.0225**
- **CV2: Portrait mode implementation** — AGACCI: **0.5570 ± 0.0088**; SLI: **0.8114 ± 0.0088**
- **NLP1: Embedding comparison and analysis** — AGACCI: **0.4057 ± 0.0189**; SLI: **0.5802 ± 0.0322**
- **NLP2: Preprocessing for summarization pipeline** — AGACCI: **0.9286 ± 0.0204**; SLI: **0.9643 ± 0.0195** [2507.05321]

The paper emphasizes one recurring cause: some rubric items depend on **external actions** or **external evidence**, such as **actual Kaggle submission and leaderboard verification**, which may not be inferable from notebook code alone unless explicitly documented [2507.05321]. AGACCI is therefore strongest when evidence for a criterion is present within the submission itself and weaker when rubric satisfaction depends on hidden or off-platform verification [2507.05321].

Implementation is carried out on top of **AutoGen**, which is used for **multi-agent orchestration** [2507.05321]. Both AGACCI and SLI use **GPT-4o mini**, a choice justified by **limited computational resources**, **budget constraints**, the **need for responsive classroom feedback**, **sufficient reasoning ability for rubric-aligned educational assessment**, and **efficient support for parallel multi-agent deployment** [2507.05321]. The appendix includes prompts for the agents and for qualitative evaluation, but the supplied text does not include their full wording [2507.05321]. Likewise, the paper does **not** specify a sandboxed execution environment, exact notebook runner implementation, token accounting, runtime, or latency numbers [2507.05321].

The principal limitations acknowledged are **dependence on observable in-submission evidence**, difficulty with **ambiguous or conflicting rubrics**, lack of validated generalization beyond **code- and visualization-centric assignments**, absence of **formal robustness analysis**, missing **cost/runtime accounting**, and the constrained dataset scope of **60 participants**, **360 submissions**, and a specific Korean curricular context [2507.05321]. Future directions proposed in the paper include **dynamic calibration or clarification mechanisms**, tighter **human-in-the-loop feedback loops**, extension to more **open-ended text assignments**, and study of **learner trust**, **perceived clarity**, and **behavioral impact** of the generated feedback [2507.05321].

Within the literature represented here, AGACCI is best understood as a **criteria-centric, modular grading system** for complex notebook-based assignments rather than as a general-purpose auto-grader or a mathematically formalized multi-agent learning framework [2507.05321]. Its central claim is empirical: distributing rubric interpretation, submission analysis, execution checking, quantitative result evaluation, visualization judgment, interpretive assessment, consistency validation, and final aggregation across affiliated agents can improve **rubric accuracy**, **feedback accuracy**, **consistency**, and **coherence** relative to a matched single-model baseline in educational coding contexts [2507.05321].

Source: https://www.emergentmind.com/topics/agacci