Roundtable Essay Scoring (RES)
- Roundtable Essay Scoring (RES) is a multi-agent framework that simulates human deliberation by using specialized evaluator personas and rubric-based trait evaluations.
- It organizes essay assessment in a rationale-first process where individual trait scores and reasoned debates are synthesized by a moderator for a final holistic score.
- RES improves scoring reliability and transparency over traditional AES methods by addressing supervised-data limitations and prompt sensitivity with structured, dialectical reasoning.
Roundtable Essay Scoring (RES) denotes a class of automated essay scoring procedures that organize evaluation as a structured exchange among multiple LLM agents rather than as a single-pass judgment. In its named formulation, RES is a zero-shot, multi-agent framework in which evaluator personas independently construct trait-based rubrics, perform rationale-first scoring, and then enter a moderator-led roundtable discussion whose dialectical synthesis yields a final holistic score within the task’s official range (Jang et al., 18 Sep 2025). A plausible broader interpretation is that RES also names a design pattern for essay assessment in which rubric-grounded, trait-specialized rationales are generated first and then aggregated by a downstream scorer or judge: RMTS realizes this through one LLM agent per trait followed by a fine-tuned encoder–decoder S-LLM, while MADRAG realizes it through an Advocate–Skeptic–Judge protocol with rubric-aligned exemplar retrieval (Chu et al., 2024, Keramati et al., 4 Jun 2026).
1. Conceptual emergence within automated essay scoring
AES has historically sought reliable score prediction from essay text, often through supervised regression or classification pipelines. The motivation for RES arises from two limitations emphasized in recent work. First, traditional supervised AES depends on labeled data, struggles with cross-prompt generalization, and may miss nuanced traits central to human raters. Second, single-agent zero-shot prompting with LLMs lacks robust consensus formation when perspectives conflict, and its outputs vary with prompt phrasing, topic domain, and rubric details (Jang et al., 18 Sep 2025).
Concurrently, adjacent lines of research shifted AES from pure score prediction toward explicit rationale production. RMTS argues that prior multi-trait scoring models largely used essay text alone and therefore often missed trait-specific signals that human raters extract with rubrics; it addresses this by introducing trait-specific rationales generated by separate LLM agents conditioned on rubric guidelines (Chu et al., 2024). RDBE similarly departs from score-only modeling by training a small LLM to generate rubric-grounded reasoning before outputting the score, thereby framing essay evaluation as reason-then-score generation rather than classification-like prediction (Mohammadkhani, 2024).
Within this trajectory, RES is best understood as an attempt to simulate how human raters work: multiple specialized evaluators independently construct topic-calibrated rubrics, produce rationale-first trait scores, and then engage in thesis–antithesis–synthesis discussion moderated to resolve disagreements and reach a balanced holistic score. This suggests that the roundtable metaphor is not merely stylistic; it is a procedural claim about how deliberation, perspective diversity, and explicit rationales can be operationalized in AES (Jang et al., 18 Sep 2025).
2. Canonical RES architecture
The canonical RES system takes as inputs the prompt text and essay text, with optional metadata such as grade level and essay type, together with the official score range for the prompt. The framework then creates multiple evaluator personas, with the default configuration using $4$ evaluator agents. These personas are designed to cover complementary foci, including content relevance and understanding of topic or argument, organization and coherence, style and voice appropriate for grade or genre, grammar and mechanics, and prompt fulfillment or genre conventions (Jang et al., 18 Sep 2025).
Each persona automatically constructs its own rubric. By default, each agent generates $3$ traits, yielding $12$ total traits. Each trait contains a name, a description, and discrete criteria mapped to the prompt’s score range. The rubrics are calibrated to grade level and essay type, and they blend general writing traits with prompt-specific expectations, such as argument and evidence for argumentative prompts, textual analysis for response prompts, and narrative structure for narrative prompts (Jang et al., 18 Sep 2025).
Evaluation is rationale-first. Each agent scores the essay across its rubric traits and produces detailed rationales per trait explaining how essay evidence meets rubric criteria. These outputs are structured as JSON, which the paper reports improves downstream parsing and consistency. The resulting trait scores and rationales are then passed to a moderator LLM, which orchestrates the roundtable discussion and produces the final holistic score. In this implementation, the moderator is the aggregation mechanism; no external training or hand-crafted rubrics are required (Jang et al., 18 Sep 2025).
A closely related architecture appears in RMTS, although its aggregation stage differs. RMTS assigns one LLM agent per trait, conditions each agent on trait-specific rubric guidelines, concatenates the resulting rationales in a standardized order, and then feeds the essay plus the combined rationale into a shared-encoder, encoder–decoder S-LLM. Special tokens such as <Essay>, <Rationale>, and trait tokens like <Content> delimit the input, and a linear projection layer, denoted RMTS_Linear, projects encoder representations to a unified feature vector for autoregressive generation of trait names and scores (Chu et al., 2024).
3. Dialectical reasoning, aggregation, and scoring logic
The distinctive procedural core of RES is its dialectical reasoning protocol. In the thesis phase, each agent proposes an initial holistic score grounded in its trait evaluations. In the antithesis phase, other agents challenge that proposal by citing textual evidence and rubric mismatches. In the synthesis phase, the moderator reconciles viewpoints, weighs trait importance implicitly through the discussion, and assigns the final holistic score. The system instructions explicitly require turn-taking, critique, identification of overlooked evidence, and revision of stances when justified by rubrics and textual evidence (Jang et al., 18 Sep 2025).
A common misconception is to treat RES as a voting system. In the reported implementation, consensus is not a numeric voting rule; it is the moderator’s synthesis. The paper does provide an optional numeric consensus augmentation in which post-discussion trait scores are normalized and aggregated with trait weights, but it presents this as a possible robustness or auditing layer rather than as the scoring rule actually used. The operational RES pipeline therefore remains moderator-centric (Jang et al., 18 Sep 2025).
MADRAG supplies a more explicitly formalized roundtable variant for analytic essay scoring. Its roles are Advocate, Skeptic, Judge, and Supervisor. For trait , the Judge conditions on the prompt , essay , trait , score range , retrieved exemplars, the debate transcript , and confidence proxies from the Advocate and Skeptic. The final trait score is defined as
where $3$0 denotes the full bundle of essay, prompt, trait, range, retrieval output, debate transcript, and confidence features (Keramati et al., 4 Jun 2026).
MADRAG also introduces an important procedural finding for roundtable-style scoring: one round per trait, with Advocate first, Skeptic second, and Judge last, is substantially more stable than multiple rounds, while skeptic-first configurations perform poorly. This suggests that not all forms of deliberation are equally useful; roundtable design depends on agent order, role fidelity, and constraints on verbosity and drift (Keramati et al., 4 Jun 2026).
4. Evaluation protocols and empirical results
The named RES framework was evaluated on the ASAP dataset, which contains $3$1 essays across $3$2 prompts for grades $3$3–$3$4, with genres ARG for prompts $3$5–$3$6, RES for prompts $3$7–$3$8, and NAR for prompts $3$9–$12$0. The experiments used $12$1 of the test split for evaluation, comprising $12$2 essays, following prior zero-shot AES setups. The LLM backbones were GPT-4.1-mini-2025-04-14 and Claude-3.5-haiku-20241022, both accessed through official APIs and used without fine-tuning (Jang et al., 18 Sep 2025).
The primary evaluation metric was Quadratic Weighted Kappa. In the RES paper, it is defined as
$12$3
where $12$4 is the observed agreement matrix, $12$5 is the expected agreement under independence, and $12$6 is the number of discrete score categories (Jang et al., 18 Sep 2025).
| Model | Vanilla avg. QWK | RES avg. QWK |
|---|---|---|
| ChatGPT | 0.364 | 0.483 |
| Claude | 0.370 | 0.499 |
For ChatGPT, the comparison against Vanilla and MTS was $12$7, with MTS at $12$8; this corresponds to a relative improvement over Vanilla of $12$9 and an absolute gain of 0. For Claude, the comparison was 1, with MTS at 2; this corresponds to a relative improvement over Vanilla of 3 and an absolute gain of 4, consistent with the reported “up to 5” improvement (Jang et al., 18 Sep 2025).
Ablations clarify where the gains arise. RES without dialectical reasoning reached an average QWK of 6 with ChatGPT, compared with Vanilla at 7, while RES with dialectical reasoning reached 8 and was best on all prompts except 9. Increasing the number of agents from 0 to 1 yielded approximately 2 average QWK, whereas increasing from 3 to 4 added 5, indicating diminishing returns. With agents fixed at 6, increasing the total number of traits from 7 to 8 gave 9 average QWK, while increasing from 0 to 1 added only 2, suggesting that over-fragmentation can add noise (Jang et al., 18 Sep 2025).
The performance gains come with computational cost. Averaged across prompts 3–4, Vanilla required approximately 5 seconds and cost about 60.002171.78\$e$9 per essay. Time and tokens scaled roughly linearly with the number of agents and traits, plus moderator discussion length (Jang et al., 18 Sep 2025).
5. Relation to rationale-based, distilled, and retrieval-grounded variants
RMTS provides the clearest blueprint for a roundtable-style multi-trait scorer that uses explicit trait-specialist agents. One LLM agent is assigned to each trait, each agent receives a trait-specific prompt containing rubric guidelines and the essay, and the final rationale sequence is ordered from more elementary constituent traits to the overall trait. The essay and rationale are concatenated and scored by one of five encoder–decoder S-LLMs—T5, Flan-T5, BART, Pegasus, or LED—fine-tuned for sequence-to-sequence score generation. On ASAP/ASAP++, average QWK across traits improved from $r_i$0 to $r_i$1 for T5, from $r_i$2 to $r_i$3 for Flan-T5, from $r_i$4 to $r_i$5 for BART, from $r_i$6 to $r_i$7 for Pegasus, and from $r_i$8 to $r_i$9 for LED when <a href="https://www.emergentmind.com/topics/generative-pretrained-transformer-gpt" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">GPT</a> rationales were added. The paper also reports that linear projection outperformed convolution, cross-attention, mean or sum pooling, and max pooling for representation fusion (<a href="/papers/2410.14202" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Chu et al., 2024</a>).</p> <p>RDBE addresses a related but distinct problem: how to distill rubric-grounded reasoning from a teacher LLM into a compact student model. Its teacher is Llama-3-70B, queried in zero-shot mode to generate short rationales conditioned on the rubric and the gold score; its student is LongT5-Base, trained to output the rationale followed by <code>" ---> "</code> and the score. On DREsS_New, RDBE reached QWK values of $[m_i, M_i]$0 for Content, $[m_i, M_i]$1 for Organization, $[m_i, M_i]$2 for Language, and $[m_i, M_i]$3 for Total, compared with $[m_i, M_i]$4, $[m_i, M_i]$5, $[m_i, M_i]$6, and $[m_i, M_i]$7 for the ArTS baseline using the same LongT5-Base backbone. This indicates that rubric-grounded reasoning can serve not only as explanation but also as an inductive bias for stronger scoring (Mohammadkhani, 2024).
MADRAG is the most explicit training-free analytic scoring extension of the roundtable idea. Its Judge is augmented with rubric-aligned exemplar retrieval based on all-MiniLM-L6-v2 embeddings and Chroma, while the Advocate and Skeptic debate strengths and weaknesses for each trait. On ASAP Set $[m_i, M_i]$8, MADRAG with GPT-5 reported QWK values of $[m_i, M_i]$9 for Ideas, $\tau_i(e) = (a_i, k_i)$0 for Organization, $\tau_i(e) = (a_i, k_i)$1 for Style, and $\tau_i(e) = (a_i, k_i)$2 for Conventions; on Set $\tau_i(e) = (a_i, k_i)$3, MADRAG with GPT-4o-mini reported $\tau_i(e) = (a_i, k_i)$4 for Ideas, $\tau_i(e) = (a_i, k_i)$5 for Organization, $\tau_i(e) = (a_i, k_i)$6 for Voice, $\tau_i(e) = (a_i, k_i)$7 for Word Choice, $\tau_i(e) = (a_i, k_i)$8 for Sentence Fluency, and $\tau_i(e) = (a_i, k_i)$9 for Conventions. The paper’s ablations attribute the largest single calibration gains to retrieval and report that debate improves higher-level reasoning, especially for Organization and Fluency, while sometimes adding noise on surface traits (Keramati et al., 4 Jun 2026).
Taken together, these systems show that RES can be instantiated in at least three operational forms: moderator-centered zero-shot holistic scoring, rationale-conditioned scoring with a fine-tuned S-LLM, and trait-wise debate with retrieval-grounded calibration. This suggests that “roundtable” is less a single architecture than a family of mechanisms for distributing evaluation across specialized perspectives and then constraining their aggregation through rubrics, rationales, or exemplars.
6. Reliability, limitations, and future directions
The literature presents RES as improving trust and transparency, but it also records several stability and validity constraints. In RMTS, rationales alone retained at least $s_i(e) = \arg\max_s \, p_J(s \mid I_i(e)),$0 of essay-only performance for most traits, and ROUGE-L similarity was higher within essays than between essays, which the authors interpret as evidence that the rationales carry meaningful trait-relevant signals and remain essay-specific. At the same time, the paper notes that LLMs can hallucinate, that autoregressive models can be sensitive to trait order, and that experiments focused on English multi-trait scoring rather than cross-lingual generalization (Chu et al., 2024).
The RES paper itself does not report inter-agent agreement or score variance across runs. It states that JSON-structured outputs and rationale-first steps reduce variance compared with free-form outputs, but it also identifies reproducibility challenges arising from evolving API models and from the absence of reported decoding parameters and seeds. The same paper notes that proprietary LLMs offered strong instruction-following, whereas open-source LLMs showed weaker adherence and output parsing reliability with the same prompts. Bias and fairness are treated as open concerns: LLMs may encode training-data biases, and consensus among agents may conceal systematic bias if their pretraining priors are similar (Jang et al., 18 Sep 2025).
MADRAG adds a different reliability perspective. By grounding the Judge in exemplars spanning the full trait score range, it reduces central tendency bias and improves agreement near extreme scores. Yet it also depends on access to scored essays for retrieval, and sparse exemplar coverage can reduce calibration and misplace essays near band boundaries. The paper identifies additional sensitivities to anonymization placeholders such as @PERSON, which can be misread as real errors and cause under-scoring in Conventions unless the prompts explicitly warn agents not to penalize them (Keramati et al., 4 Jun 2026).
The most frequently proposed extensions are dynamic agent creation and role allocation, improved consensus algorithms such as confidence-weighted averaging or structured voting, supervised or reinforcement learning for agents or moderators, human-in-the-loop calibration, and cross-topic or out-of-domain evaluation. RDBE suggests a complementary path in which a heavy multi-agent or rationale-producing teacher is distilled into a portable student model. A plausible implication is that future RES systems may combine zero-shot deliberation, explicit trait rationales, and lightweight deployed scorers rather than treating these design choices as mutually exclusive (Jang et al., 18 Sep 2025, Mohammadkhani, 2024).