SAGEval: Reference-Free NLG Evaluation

• SAGEval is a reference-free evaluation framework for NLG systems, employing a two-stage process with an evaluator and a critiquing agent.
• It leverages chain-of-thought prompting and score calibration to iteratively adjust facet-level ratings, reducing the reliance on human labeling.
• Experimental results show a 20% improvement in alignment with human judgment over baseline models, highlighting its practical efficacy.

SAGE Evaluation Framework

SAGEval (“Satisfactory Agent-based NLG Evaluation”) is a reference-free, agentic framework for evaluating natural language generation (NLG) systems on open-ended, semi-structured outputs where no gold standard references exist and human labeling is prohibitively expensive. SAGEval introduces a two-stage, agent-based evaluation loop, leveraging a first-pass LLM evaluator and a second-pass, more “meta” critiquing agent to iteratively refine facet-level quality scores without reliance on reference outputs or ground-truth annotations (Ghosh et al., 2024).

1. Motivation and Problem Setting

Traditional NLG evaluation approaches presuppose the existence of ground-truth references (e.g., for translation, summarization, or Q&A). However, many applications require the generation of open-ended artifacts—such as surveys, quizzes, lists, or JSON-structured forms—that have no canonical target output. In such scenarios, existing evaluation protocols either default to ad hoc human judgment or fail to account for multiple dimensions of appropriateness, relevance, and style. SAGEval was developed to address this critical gap by enabling robust, reference-free, and facet-distributed evaluation in these unconstrained settings.

2. Formal Framework and Workflow

Let $U$ denote the space of generative outputs (e.g., JSON forms), and $A = \{a_1, \dots, a_m\}$ the set of evaluation aspects (examples: Accuracy, Coherence, Audience Engagement).

Components

• Evaluator Agent ($E$): A prompt-based LLM that, for each $(u, a)$, produces a raw score $S_\text{initial}(u,a) \in [1,5]$. $E$ is designed with chain-of-thought templates, in-context examples, and natural-language aspect definition.
• Critiquing Agent ($C$): A second LLM prompt receiving the output $u$, the aspect $a$, the initial score, and the evaluator’s reasoning. $C$ performs the following:
  • Computes a correction $\Delta F(u,a) \in \mathbb{R}$ (adjustment to $S_\text{initial}$),
  • Optionally suggests refinements to aspect definitions,
  • Optionally proposes new aspects for extended coverage.
• Feedback Loop: The corrected score is

$$S_\text{corrected}(u, a) = S_\text{initial}(u, a) + g\left(\Delta F(u, a)\right)$$

where $g(\cdot)$ is a calibration function (identity by default, potentially clipped to $[1,5]$). The critiquing agent can iteratively refine scores, but empirical results show one round typically suffices.

Pseudocode Summary

Input: Output u; aspects A = {a₁,…,a_m}; max_iters For each aspect a in A: S ← EVALUATOR_AGENT.evaluate(u, a) reasoning ← EVALUATOR_AGENT.reasoning(u, a) ΔF, aspect_updates, new_aspects ← CRITIQUING_AGENT.critiquer(u, a, S, reasoning) S_corr ← Clip(S + g(ΔF), 1, 5) SCORES[u][a] ← S_corr if aspect_updates ≠ ∅: A_definitions[a] ← A_definitions[a] ∪ aspect_updates For each a′ in new_aspects: Append a′ to A Return SCORES, updated A, updated definitions

3. Design Principles and Prompt Engineering

The SAGEval framework operates fully reference-free by leveraging:

• Few-shot, in-context exemplars of both high- and low-quality outputs per aspect.
• Explicit aspect definitions in natural language to anchor evaluator and critic.
• Chain-of-thought prompting to elicit explicit rationalizations for each quality score.
• Score-probability calibration: LLM token probabilities for $1,2,\ldots,5$ are converted into soft scores, discouraging defaulting to ties or edge values:

$\text{score} = \sum_{i} p(s_i) \cdot s_i$

4. Correction and Aspect Discovery Mechanisms

The critiquing agent serves both quantitative and qualitative purposes:

• Score Rectification: Systematically adjusts initial scores toward human-like calibration by correcting LLM over-optimism or under-penalization.
• Aspect Definition Refinement: In about 33% of cases, the agent suggests tightening, expanding, or clarifying definitions (e.g. disambiguating “Coherence” from “Semantic Diversity”).
• New Aspect Identification: Freeform suggestions are topic-modeled to empirically surface missing evaluation axes (e.g., “Creativity,” “Content-Quality”) in more than 40% of survey-type outputs.

5. Experimental Validation

Benchmark Construction

• Dataset: 96 reference-free surveys/forms (e.g., “List the top considerations for telemedicine adoption in rural areas”) generated by GPT-3.5-Turbo.
• Human Annotation: 4 expert linguists provided 1–5 facet ratings for 8 aspects: Accuracy, Semantic Diversity, Coherence, Relevancy, Audience Understandability, Audience Engagement, Fairness, Sentiment/Tone.

Baselines

• Automated: G-Eval (GPT-4 single-agent), CheckEval, FreeEval, MATEval, vanilla LLM-Eval (chain-of-thought without critique).

Metrics

• Correlation with Human Judgment: Spearman $\rho$ and Kendall $\tau$ between automatic and human-provided facet scores.

Results

Metric	Vanilla LLM-Eval	G-Eval	SAGEval (Corrected)
Spearman $\rho$ (mean)	—	baseline	$\approx$ 0.55
Kendall $\tau$ (mean)	—	baseline	$\approx$ 0.42

• SAGEval achieved a $\sim$20% improvement in correlation over G-Eval across aspects.
• The distribution of automatic scores shifted from excessive 4–5 ratings (80% floor effect) to a more human-aligned 2–3 mode after critique.
• SAGEval facilitated aspects’ redefinition and discovery for more comprehensive evaluation coverage (Ghosh et al., 2024).

6. Strengths, Limitations, and Future Directions

Strengths

• Reference-Free Evaluation: Eliminates the need for costly gold labels or references.
• Meta-Agentic Critique: Enables both score correction and rubric evolution via agent interaction.
• Alignment with Human Judgment: Empirical correlations approach those of expert annotators.

Limitations

• Inference Cost: Two LLM calls (evaluator and critic) per aspect incur computational overhead.
• Domain Scope: Validation to date focuses on semi-structured survey generation, with untested performance on fully unconstrained generation (e.g., conversational dialogues, code synthesis).
• Single Round Feedback: Empirical findings suggest deep multi-round critique may further enhance alignment at additional cost.

Future Work

• Adaptive Calibration Functions: Learning $g(\cdot)$ from small human-labeled seeds.
• Peer Debate: Multiple critiquing agents with diverse priors.
• Coverage Expansion: Application to dialogue, creative writing, and multimodal NLG (Ghosh et al., 2024).

7. Summary Table: Core SAGEval Components

Component	Functional Role	Output/Effect
Evaluator Agent	Aspect-wise initial scoring and reasoning	$S_\text{initial}(u,a) \in [1,5]$
Critique Agent	Score correction, aspect/rubric refinement	$\Delta F(u,a)$, and aspect updates
Feedback Loop	Aggregates, clips, and optionally iterates	$S_\text{corrected}(u, a)$

SAGEval offers a principled, agent-based protocol for high-granularity, reference-free evaluation of NLG systems. By tightly coupling evaluator and critic LLMs, the framework achieves robust score correction, emergent rubric refinement, and empirical alignment with expert judgment, substantially reducing the need for manually labeled data even in complex, open-ended generation scenarios (Ghosh et al., 2024).