G-Eval: Structured LLM Evaluation Framework

• G-Eval is a framework for automatic evaluation of natural language generation that uses GPT-4 as a structured judge instead of traditional reference metrics.
• It employs auto-generated chain-of-thought steps and a fixed form-filling prompt to standardize evaluations for tasks like summarization and dialogue generation.
• Its probability-based score aggregation refines integer ratings into continuous scores while addressing common biases and tie issues in LLM evaluations.

Searching arXiv for papers on G-Eval and closely related follow-up analyses. arXiv search: "G-Eval NLG Evaluation using GPT-4 with Better Human Alignment" G-Eval is a framework for automatic evaluation of natural language generation (NLG) that uses LLMs, especially GPT-4, as structured judges rather than as likelihood scorers. Proposed by Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu of Microsoft Cognitive Services Research, it combines criterion-specific prompting, auto-generated chain-of-thought (CoT) evaluation steps, a form-filling output format, and probability-based score aggregation to improve alignment with human judgments on summarization and dialogue evaluation (Liu et al., 2023).

1. Definition and historical position

G-Eval was introduced to address a central problem in NLG evaluation: conventional reference-based metrics such as BLEU and ROUGE often have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. The original paper also argues that earlier LLM-based evaluators had not clearly surpassed strong medium-sized neural evaluators in human correlation, leaving a gap between the flexibility of LLM prompting and the practical quality of automatic evaluation (Liu et al., 2023).

The framework occupies a specific methodological position within the LLM-as-a-judge literature. In contrast to lexical-overlap metrics, G-Eval can be reference-free and can use the source or input context directly. In contrast to trained evaluators such as UniEval, FactCC, or USR, it does not rely on a separately fine-tuned task-specific evaluator. In contrast to GPTScore, which treats evaluation as a conditional generation likelihood problem, G-Eval treats evaluation as a direct scoring task in which the model is instructed to judge a candidate output according to an explicit criterion and to return a constrained rating (Liu et al., 2023).

This positioning made G-Eval an influential baseline for later work. Subsequent papers repeatedly treat it as a representative prompting-based LLM evaluator, especially for summarization, and often use it as the comparison point when proposing more decomposed, rubric-driven, or human-grounded alternatives (Pereira et al., 2024).

2. Core methodology

G-Eval has three main components: a prompt defining the task and evaluation criterion, auto-generated CoT evaluation steps, and a scoring function based on the model output and token probabilities. The evaluator is first given a task description and a criterion definition, such as coherence for summarization. It then generates explicit evaluation steps automatically from the task introduction and criterion. These steps are inserted back into the final judging prompt together with the task inputs and a constrained evaluation form (Liu et al., 2023).

For summarization, the prompt structure includes a source document and a candidate summary. For dialogue generation, it can include conversation history, corresponding fact metadata, and the candidate response. The form-filling paradigm is central: the model is instructed to complete a fixed field such as Evaluation Form (scores ONLY): - Coherence:. This constrains the output space, standardizes parsing, and supports downstream score extraction (Liu et al., 2023).

The canonical coherence example in the paper illustrates the full workflow. The evaluation criterion defines coherence on a 1–5 scale and aligns it with the DUC quality question of structure and coherence. The auto-generated evaluation steps then instruct the model to read the news article, identify the main topic and key points, compare the summary against the article, and assign a score. This design makes G-Eval a structured judge rather than a free-form critic (Liu et al., 2023).

A key nuance is that CoT in G-Eval is primarily a prompt-construction mechanism. The model first generates the evaluation steps, and those steps are then reused in the actual evaluation prompt. Later analyses would question whether this auto-CoT is consistently helpful, but in the original formulation it is a defining element of the framework (Chiang et al., 2023).

3. Scoring function and inference procedure

G-Eval’s final score is not limited to the discrete integer emitted by the model. Given a predefined rating set $S = \{s_1, s_2, \ldots, s_n\}$ and model probabilities $p(s_i)$ over those labels, the framework computes the expected score as

$score = \sum_{i=1}^n p(s_i) \times s_i$

This probability-weighted score is motivated by two observed problems in direct discrete rating: score distribution collapse, often toward a middle label such as 3 on a 1–5 scale, and excessive ties caused by the tendency of models to emit integers even when decimal scores are requested. By using the expected value over rating labels, G-Eval obtains a finer-grained continuous score (Liu et al., 2023).

The implementation differs by backbone. For GPT-3.5 (text-davinci-003), the paper uses temperature $= 0$. For GPT-4, token probabilities were unavailable in the reported setup, so the authors estimate them by repeated sampling with $n = 20$, temperature $= 1$, and top_p $= 1$. This repeated-sampling design later became one of the most recognizable procedural details associated with G-Eval in follow-up work (Liu et al., 2023).

G-Eval therefore combines two forms of structure: symbolic structure at the prompt level, through explicit criteria and evaluation steps, and probabilistic structure at the score level, through expected-value aggregation. A plausible implication is that its influence derives not from a single prompt template but from this combination of constrained reasoning, constrained output, and uncertainty-aware scoring.

4. Benchmarks and empirical performance

The original paper evaluates G-Eval on three meta-evaluation benchmarks spanning two NLG task families: text summarization and dialogue response generation. For summarization, it uses SummEval and QAGS. For dialogue, it uses Topical-Chat. Human alignment is measured with Spearman $\rho$, Kendall $\tau$, and Pearson $r$, depending on the benchmark (Liu et al., 2023).

Benchmark	Task	G-Eval-4 result
SummEval	Summarization	average Spearman $p(s_i)$0, Kendall $p(s_i)$1
Topical-Chat	Dialogue	average Pearson $p(s_i)$2, Spearman $p(s_i)$3
QAGS	Factual consistency / hallucination	average $p(s_i)$4, $p(s_i)$5, $p(s_i)$6

On SummEval, the dimension-wise G-Eval-4 scores are coherence $p(s_i)$7, $p(s_i)$8; consistency $p(s_i)$9, $score = \sum_{i=1}^n p(s_i) \times s_i$0; fluency $score = \sum_{i=1}^n p(s_i) \times s_i$1, $score = \sum_{i=1}^n p(s_i) \times s_i$2; and relevance $score = \sum_{i=1}^n p(s_i) \times s_i$3, $score = \sum_{i=1}^n p(s_i) \times s_i$4. The paper presents the average Spearman score of 0.514 on summarization as its headline result and reports that this outperforms prior methods such as UniEval, GPTScore, and BARTScore on that benchmark (Liu et al., 2023).

On Topical-Chat, G-Eval-4 reports naturalness $score = \sum_{i=1}^n p(s_i) \times s_i$5, $score = \sum_{i=1}^n p(s_i) \times s_i$6; coherence $score = \sum_{i=1}^n p(s_i) \times s_i$7, $score = \sum_{i=1}^n p(s_i) \times s_i$8; engagingness $score = \sum_{i=1}^n p(s_i) \times s_i$9, $= 0$0; and groundedness $= 0$1, $= 0$2. The paper also notes that G-Eval-3.5 performs similarly to G-Eval-4 on this benchmark, suggesting that this setting is easier for the framework than summarization consistency or hallucination detection (Liu et al., 2023).

On QAGS, G-Eval-4 is especially strong on the more abstractive XSum subset, where it substantially outperforms BARTScore and UniEval in rank-based metrics. The paper uses this result to argue that structured LLM judging is particularly valuable in difficult factuality settings where lexical metrics and some learned evaluators degrade (Liu et al., 2023).

The original paper also includes limited ablation evidence. On SummEval, removing CoT lowers average Spearman from 0.514 to 0.500 and average Kendall from 0.418 to 0.407. Removing the probability-based score handling lowers average Spearman from 0.514 to 0.502, although the paper notes that tie-heavy direct scoring can artificially inflate Kendall-Tau. These results support the authors’ interpretation that both CoT and probability normalization contribute to performance, even if later work would dispute how robust those gains are (Liu et al., 2023).

5. Biases, limitations, and critical reassessments

A major limitation already acknowledged in the original paper is bias toward LLM-generated text. In a comparison between human-written summaries and GPT-3.5-generated summaries, G-Eval-4 always gives higher scores to GPT-3.5 summaries than to human-written summaries, even when human judges prefer the human-written summaries. The authors treat this as preliminary evidence of a systematic bias toward LLM-generated text and warn that such bias could create self-reinforcing optimization loops if LLM judges are used as reward models or training signals (Liu et al., 2023).

Subsequent re-analysis sharpened these concerns. "A Closer Look into Automatic Evaluation Using LLMs" reports that the auto-CoT used in G-Eval does not always make the method more aligned with human ratings and that forcing the model to output only a numeric rating is suboptimal. That paper further finds that asking the LLM to explain its own ratings consistently improves correlation with human ratings and can push state-of-the-art correlations on SummEval and Topical-Chat (Chiang et al., 2023).

Later checklist-based work reframed G-Eval’s strengths and weaknesses in more structural terms. "Check-Eval: A Checklist-based Approach for Evaluating Text Quality" characterizes G-Eval as a GPT-4-based evaluator that uses CoT and self-consistency with $= 0$3 samples to mitigate score dominance and integer-score tendencies. On SummEval, Check-Eval reports overall average correlations of $= 0$4 against G-Eval’s $= 0$5, while G-Eval remains stronger on relevance and slightly stronger on coherence Spearman. The critique is therefore not that G-Eval is uniformly weak, but that its direct scalar judging is less structured and less interpretable than checklist decomposition (Pereira et al., 2024).

"HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation" places G-Eval in the class of prompting-based, zero-shot evaluators. It argues that such evaluators impose an unnecessary restriction by not consulting human evaluations or groundings and reports that HypoEval, using only 30 human evaluations, outperforms G-Eval by 11.86% on average. This positioning turns G-Eval into a baseline for two later research directions: decomposition into finer rubrics and injection of small-sample human grounding (Li et al., 9 Apr 2025).

Taken together, these critiques establish a stable picture. G-Eval is widely regarded as a strong and important baseline, but not as a final solution. The most persistent concerns involve interpretability, prompt sensitivity, numeric-only output constraints, computational overhead from repeated sampling, and bias toward LLM-style outputs.

6. Influence and extensions

G-Eval has been influential less as a fixed prompt than as a general design pattern for LLM-based evaluation. Later work repeatedly preserves four ingredients: explicit criteria, stepwise judging instructions, structured output formats, and score extraction that accounts for output uncertainty. This suggests that G-Eval’s durable contribution is methodological rather than merely benchmark-specific.

The clearest multimodal extension is "G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o," which is explicitly described as inspired by G-Eval. G-VEval adopts the LLM-as-a-judge paradigm, criteria-based evaluation, chain-of-thought-style evaluation steps, structured scoring, and expected-value score handling, while extending the framework to image and video caption evaluation with reference-free, reference-only, and combined modes. At the same time, that paper criticizes the original implementation of G-Eval for only weakly realizing CoT, arguing that it often produces single-digit outputs and lacks a genuine in-context reasoning process (Tong et al., 2024).

Within text evaluation, G-Eval’s legacy is visible in systems that preserve its judge architecture while replacing its holistic scalar decision with finer-grained decompositions. Check-Eval uses binary yes/no checklist items; HypoEval uses decomposed rubric dimensions scored on a Likert scale. A plausible implication is that later work has tended to accept G-Eval’s premise—that instruction-following LLMs can serve as credible evaluators—while modifying its internal representation of judgment from monolithic scoring toward explicit intermediate structure.

G-Eval therefore marks an inflection point in automatic evaluation research. It demonstrated that GPT-4 could function as a practical NLG evaluator with stronger human alignment than prominent contemporaneous baselines on summarization, dialogue, and factual consistency benchmarks (Liu et al., 2023). It also exposed a methodological fault line that has remained central ever since: whether high-performing LLM judges should rely on direct holistic scoring, as in the original framework, or on more decomposed, interpretable, and human-grounded evaluation pipelines (Pereira et al., 2024).