iRULER: Rubric-Based LLM Feedback

• iRULER is a rubric-centered framework that leverages large language models to provide structured, transparent, and actionable writing feedback.
• It employs a recursive rubric-of-rubrics methodology to evaluate and refine both user-defined rubrics and writing through clear justifications and minimal-edit suggestions.
• The system integrates a serverless web application with interactive UIs and structured JSON outputs to ensure criterion-specific guidance and enhanced rubric reliability.

iRULER is a rubric-centered framework for evaluation and revision leveraging LLMs to deliver intelligible, actionable, and user-defined feedback on writing. It is distinguished by its recursive “rubric-of-rubrics” methodology, explicit operationalization of best practices from educational assessment and explainable AI, and a system architecture designed to produce criterion-specific justifications and minimal-edit revision suggestions. iRULER directly addresses persistent deficiencies in LLM-based feedback—namely, lack of structure, opacity, genericity, and misalignment with user goals—through structured design guidelines and an interactive, end-to-end revision loop (Bai et al., 13 Feb 2026).

1. Motivation and Design Principles

Contemporary LLM feedback on writing is frequently characterized by unstructured justifications, generic statements, and lapses in alignment with articulated criteria. Standard approaches, such as holistic scoring or basic rubric use, often fail to deliver transparent, criterion-grounded, and actionable guidance. iRULER systematically remediates these limitations by binding LLM evaluation to explicit, user-defined rubrics and introducing mechanisms for transparency, actionable advice, and rubric qualification.

Six design guidelines constitute the framework:

• Specific: Feedback targets user-defined criteria only.
• Scaffolded: Each criterion is delineated into 3–6 performance levels within a weighted tabular rubric.
• Justified: The model must generate “Why” (score justification) and “Why Not” (why higher scores were not assigned) explanations, consisting of global judgments and supporting evidence.
• Actionable: “How To” counterfactual suggestions recommend minimal edits that escalate writing to a target level, each edit rationalized.
• Qualified: The “rubric-of-rubrics” concept applies the framework recursively, evaluating and refining the quality of the user-authored rubric itself.
• Refinable: Users can iteratively revise rubrics in a closed loop, supporting long-term evolution of evaluation standards (Bai et al., 13 Feb 2026).

2. Rubric-of-Rubrics Methodology

Central to iRULER is the recursive rubric-of-rubrics process for qualifying rubrics before deployment. Users supply a draft rubric, which is then critiqued and scored by the LLM using a meta-rubric of three axes:

• Criteria Alignment (does each criterion measure what it intends)
• Level Distinction (are performance levels separable and clearly staged)
• Descriptive Language (is wording specific and actionable)

Each axis is rated 1–4; weighted aggregation yields an overall rubric quality metric. The system provides “Why/Why Not” explanations for each dimension, and on request, autogenerates a fully revised rubric using a minimal-modification principle to preserve user’s intent while enhancing clarity. This evaluation-revision cycle is repeatable, yielding stable, high-quality rubrics tailored to user needs.

Pseudocode for the revision loop: $L$4 This process is strictly recursive, allowing for continual rubric improvement (Bai et al., 13 Feb 2026).

3. System Architecture and Prompt Engineering

The iRULER implementation is a serverless web application. The front end (Vue.js + Quasar) offers two main interfaces:

• Writing Revision UI: Features an artifact edit pane and an interactive rubric table with cell-level actions for justification and revision guidance.
• Rubric Creation UI: Enables direct editing of rubric JSON and provides meta-rubric feedback in real time.

Both modalities employ a shared LLM scoring engine accessed via structured prompts. Artefacts and rubrics are serialized as JSON and sent with a role/task instruction. Outputs include per-criterion scores, “Why/Why Not” explanations in a structured JSON, and “How To” tracked-edit suggestions or rubric rewrites upon request.

Prompt templates follow a strict seven-step process: criteria parsing, level selection, constrained evaluation tone, justification generation for selected and non-selected levels, evidence structuring, and forcing correct JSON output (Bai et al., 13 Feb 2026).

4. Formal Model and Scoring Mechanics

Let $K$ denote the number of rubric criteria, each with $L$ levels and weight $w_k$ (summing to 100%). For a given artifact, the LLM selects a level $s_k \in \{1,\ldots,L\}$ for each criterion, with overall composite score:

$S = \sum_{k=1}^K w_k \cdot (s_k / L)$

For rubric evaluation (meta-rubric), with $M=3$ meta-criteria, weight $v_m$ and per-dimension scores $r_m \in \{1,2,3,4\}$:

$R = \sum_{m=1}^M v_m \cdot \frac{r_m}{4}$

Reliability is quantified via Krippendorff’s $\alpha$ (for intra-model consistency) and Quadratic Weighted Kappa (for LLM-vs-expert agreement). These metrics establish empirical validity and replicability of scoring (Bai et al., 13 Feb 2026).

5. Empirical Validation: Quantitative and Qualitative Evidence

Three controlled studies substantiate iRULER’s efficacy:

• Writing Revision (N=48): iRULER yielded the largest improvement in LLM-assessed writing scores (ΔOverallScore: iRULER +27/30.8, Rubric LLM +19/23.8, Text LLM +16; $L$0), maximized per-criterion gains (especially Content & Vocabulary), and required fewer iterations than competitors.
• Rubric Creation (N=36): Final rubric quality was highest for iRULER participants (83–86 vs 77–79 for baseline; $L$1), with marked increases in perceived helpfulness and confidence.
• Expert Agreement: Rubric LLM vs experts QWK=0.88, Text LLM QWK=0.76; both show strong alignment but rubric-grounded approaches are superior.

Qualitative analysis confirms coverage of all design guidelines (DG1–6), frequent and valued use of justification features, transparent rationale in scoring, efficiency via minimal-edit counterfactuals, and iterative rubric refinement. A plausible implication is that iRULER both enhances user agency and produces more reliable, criterion-anchored feedback in AI-supported writing workflows (Bai et al., 13 Feb 2026).

6. Example Use Cases and Best Practices

Writing Revision

Given a brief product description, iRULER will identify the rubric level achieved (e.g., Clarity=3/5 because “no sensory adjectives”), provide “Why Not” justifications, and output a model revision minimally editing one sentence to “room-filling sound…”—explicitly advancing to the next rubric level.

Rubric Creation

A vague criterion such as “Creativity” results in low meta-rubric scores for Descriptive Language. The system suggests an advanced rewrite (e.g., “Exhibits original metaphors and surprising organization…”) providing actionable scaffolding for rubric authors.

Recommended practices include: grounding feedback in user-controlled rubrics, qualifying rubrics prior to use, exposing explanations at per-cell resolution, enforcing minimal-change principle in revision, tuning prompt temperature for consistency vs. creativity, and validating LLM-as-judge pipelines on expert-rated samples (targeting Krippendorff’s $L$2 and QWK $L$3).

7. Applications, Limitations, and Outlook

iRULER is applicable wherever intelligible, criteria-based LLM evaluation is desired, including educational writing, form-based review, and structured peer evaluation. Robust gains over baseline approaches are empirically supported in both writing revision and rubric creation scenarios.

Limitations include potential cognitive load for users alternating between artifact and rubric author roles, and occasional verbosity or misalignment in LLM-generated explanation chains (mitigable by temperature tuning and usage analytics). A plausible implication is that further UI and prompt optimization could enhance accessibility for broader user cohorts.

By integrating explicit, intelligible scaffolds into the LLM review loop and recursively qualifying rubric structure, iRULER establishes a rigorous, extensible blueprint for user-aligned, actionable, and transparent feedback in AI-supported assessment ecosystems (Bai et al., 13 Feb 2026).