Model-Generated Evaluation

• Model-generated evaluation is an automated assessment approach that uses learned models to generate scores, checklists, and rubrics for evaluating generative outputs.
• It decomposes evaluations into structured criteria and multi-dimensional metrics, improving interpretability and aligning better with human judgment than traditional overlap methods.
• Recent advances leverage LLM-based judges and domain-specific frameworks across text, images, video, and code to yield statistically robust and efficient evaluations.

Model-generated evaluation denotes the use of learned models to produce evaluative artifacts for generative systems, including scalar scores, pairwise preferences, checklists, rubrics, natural-language rationales, uncertainty estimates, and discovered failure cases. In recent work, such evaluators appear as LLM-as-a-Judge protocols for text, vision-language evaluators for images, video, and 3D objects, and agentic or execution-based evaluators for code. The central motivation is consistent across these settings: manual evaluation is slow and costly, conventional overlap metrics are often misaligned with human judgment, and many generative tasks require assessments that are decomposed, interpretable, and statistically valid rather than a single opaque score (Liu et al., 2023, Cook et al., 2024, Fisch et al., 6 Jul 2026).

1. Historical emergence and conceptual scope

Earlier work on automatic evaluation exposed a basic tension between human judgment, lexical overlap, and discriminative “judge” models. In online review generation, large-scale comparison of human evaluators, discriminative evaluators, and word-overlap metrics found that human evaluators did not correlate well with discriminative evaluators, while human decisions correlated better with lexical overlaps; the same study reported Fleiss’ kappa $k = 0.2748$, individual human accuracy of 66.61%, and majority-vote accuracy of 72.63% in real-versus-generated review identification (Garbacea et al., 2019). This result challenged the assumption that fooling an adversarial discriminator is equivalent to producing text judged as human-like.

Human evaluation itself was then formalized as a resource-allocation problem. A dynamic framework for relative model comparisons used Hoeffding-style stopping rules to determine when enough human labels had been collected, and its simulations and crowdsourcing study indicated that assigning a single random worker per task required the least overall labeling effort (Thorleiksdóttir et al., 2021). The emphasis here was not yet on model-generated judgment, but on the statistical design problem that later model-based evaluators would inherit.

The transition toward model-generated evaluation accelerated with learned evaluators trained directly on human judgments. Instruction-tuned evaluation on the HEAP dataset unified 229,104 pairwise comparisons from 8 NLG datasets and 22 criteria, showing that evaluator behavior itself could be learned from natural-language instructions, context, and candidate outputs. In that setting, a BART-base evaluator improved from a base accuracy of about 50.6% to about 63.2% in single-task training and about 68.8% in multi-task training (Mehri et al., 2023). In parallel, LLM-based judges such as G-Eval combined chain-of-thought and form-filling to reach stronger human alignment in summarization and dialogue than prior automatic metrics (Liu et al., 2023).

The scope of model-generated evaluation then expanded beyond text. Domain-agnostic hierarchical frameworks such as MPEGO structured evaluation from sub-feature to global feature levels, while later systems specialized for Chinese academic generation, text-to-3D, AI-generated images and video, geospatial code, cryptographic Rust, and PETSc scientific code (Tadesse et al., 2023, Zeng et al., 2023, Maiti et al., 10 Apr 2025, Hou et al., 12 Jun 2025, Zhang et al., 16 Mar 2026). Taken together, these developments indicate that model-generated evaluation is not a single method but a family of automated assessment protocols that differ in how they decompose quality, how they ground judgments, and how they quantify uncertainty.

2. Decomposition into forms, checklists, and rubrics

A defining methodological trend is decomposition: instead of asking for a monolithic score, the evaluator first generates or applies structured criteria. G-Eval exemplifies the form-filling approach. It prompts the evaluator with a task introduction and explicit criteria such as coherence or relevance, asks the model to generate intermediate evaluation steps, and then collects discrete scores through a form. To reduce ties, it computes a probability-weighted score,

$\text{score} = \sum_{i=1}^n p(s_i)\, s_i,$

which yielded a Spearman correlation of 0.514 with humans on summarization and strong results on dialogue generation (Liu et al., 2023).

TICK pushes decomposition further by converting an instruction into a checklist of instruction-specific YES/NO questions. Each question targets one explicit or important implicit requirement, and the evaluator answers all checklist items before aggregation. The paper defines Checklist Pass Rate for instruction $i$ as

$PR_i = \frac{1}{n_i}\sum_{j=1}^{n_i} a_{i,j},$

and the dataset-level Decomposed Requirements Following Ratio as

$DRFR = \frac{\sum_{i,j} a_{i,j}}{\sum_i n_i}.$

This formulation makes the evaluation interpretable at the requirement level and supports both judging and self-improvement workflows (Cook et al., 2024).

Rubric generation generalizes the same idea from binary requirements to full evaluation dimensions. GER-Eval studies whether LLMs can generate sets of criteria, each with a name, description, and scale, and then apply them consistently to candidate outputs. Its findings indicate that models can generate interpretable and task-aware evaluation dimensions and apply them consistently within models, although reliability degrades in factual and knowledge-intensive settings (Siro et al., 9 Feb 2026). A related line of work replaces human-authored rubrics altogether: dynamic rubric generation creates dataset-specific and instance-specific rubrics without human annotation, then refines the rubric generator through meta-judge preference signals, Bradley–Terry ranking, and Direct Preference Optimization (Wang et al., 28 May 2026).

These systems collectively shift evaluation from post hoc scalar scoring toward criterion generation. A plausible implication is that interpretability is no longer merely a reporting feature; it becomes part of the evaluator’s internal computational structure.

3. Composite metrics, prediction-powered estimation, and uncertainty

Model-generated evaluation does not eliminate metric design; rather, it moves metric construction into composite or statistically corrected estimators. CG-Eval illustrates this in Chinese open-ended generation. It evaluates 11,000 questions across six domains and defines Gscore for standard questions as

$$\text{Gscore} = 0.2 \times \text{BLEU-4} + 0.25 \times \text{ROUGE-2} + 0.25 \times \text{CHRF} + 0.3 \times \text{Semantic Similarity}.$$

For multi-step mathematical problems, it uses

$$\text{Gscore} = \text{Accuracy} + (1 - \text{Accuracy}) \times 0.3 \times \text{StepCHRF},$$

so exact correctness dominates while partially crediting the solution process when the final answer is wrong. On 1,100 QA pairs annotated by experts, Gscore reached a Kendall Tau of about 0.61 against human assessment (Zeng et al., 2023).

Another development is the explicit treatment of evaluation as statistical estimation under annotation scarcity. CollabEval formulates model evaluation as matrix completion over an $M \times N$ score matrix, where rows are models and columns are prompts. Low-rank reconstructions of missing scores are then used as control variates within a prediction-powered estimator,

$$\hat{\theta}_i = \frac{1}{N}\sum_{j=1}^N \hat{Y}_{ij} + \frac{1}{|\mathcal{J}_i|}\sum_{j \in \mathcal{J}_i}(S_{ij}-\hat{Y}_{ij}),$$

which is constructed to remain unbiased and to support valid confidence intervals (Fisch et al., 6 Jul 2026). Empirically, when the labeled fraction $p$ is small, the method reduced mean confidence-interval width by over 20% on average, with some settings showing reductions above 30%.

ProEval addresses a related problem from an active-testing perspective. It models the unknown performance score function with pre-trained Gaussian Processes, estimates global performance by Bayesian quadrature, and treats failure discovery as superlevel set sampling. The method is theoretically unbiased and bounded, and empirically required 8–65 times fewer samples to achieve estimates within 1% of ground truth while also finding more diverse failures under a stricter budget (Huang et al., 25 Apr 2026). In both cases, the evaluator is not merely producing scores; it is also allocating scarce evaluation effort.

4. Multimodal and domain-specific instantiations

Model-generated evaluation has diversified rapidly across modalities, and the strongest systems are often domain-specific rather than task-agnostic.

System	Target domain	Evaluative output
CG-Eval (Zeng et al., 2023)	Chinese academic/professional generation	Gscore over six domains
Gen3DEval (Maiti et al., 10 Apr 2025)	Generated 3D objects	Pairwise judgments on text fidelity, appearance, surface quality
Q-Save (Wu et al., 24 Nov 2025)	AI-generated video	MOS-linked ratings and attributions
AIGVE-MACS (Liu et al., 2 Jul 2025)	AI-generated video	Nine aspect-wise scores and comments
Q-REAL (Wang et al., 21 Nov 2025)	AI-generated images	Judgment, grounding, reasoning on realism and plausibility
AutoGEEval++ (Hou et al., 12 Jun 2025)	Google Earth Engine code	Accuracy, resource, runtime, and error analysis
petscagent-bench (Zhang et al., 16 Mar 2026)	PETSc scientific code	14 evaluators across five categories

Gen3DEval shows how specialized training changes evaluator behavior. It uses a Llama3-based LLaVA-style vLLM that jointly processes multi-view RGB renders and surface normals, and it evaluates text fidelity, appearance, and surface quality without requiring ground-truth 3D references. On human-evaluated preferences, Gen3DEval variants reported 0.9 on appearance, 0.99 on surface, and 0.98–1.0 on text fidelity, outperforming CLIP and generalist vLLMs (Maiti et al., 10 Apr 2025).

Video evaluation has also moved beyond scalar-only quality assessment. Q-Save introduces a benchmark of nearly 10,000 videos with mean opinion scores and fine-grained attributions across visual quality, dynamic quality, and text-video alignment, and trains a unified model based on Qwen3-VL-8B-Instruct with SlowFast encoding and a multi-stage SFT $\text{score} = \sum_{i=1}^n p(s_i)\, s_i,$0 GRPO $\text{score} = \sum_{i=1}^n p(s_i)\, s_i,$1 SFT regimen (Wu et al., 24 Nov 2025). AIGVE-MACS, by contrast, fine-tunes Qwen2.5-VL-7B on 2,500 videos and 22,500 human-annotated comments and scores, producing JSON-formatted aspect-wise scores and comments across nine dimensions such as technical quality, dynamic quality, consistency, physics, and overall quality (Liu et al., 2 Jul 2025).

Image evaluation has developed similar attribution-rich structures. Q-REAL contains 3,088 images from 10 text-to-image models and 17,879 localized entities, each annotated with realism and plausibility judgments plus reasoning descriptions. Its benchmark separates ObjectQA from ImageQA, thereby coupling classification, localization, and explanation in a single evaluation setting (Wang et al., 21 Nov 2025).

Code evaluation illustrates the strongest case for domain specificity. AutoGEEval++ provides 6,365 Google Earth Engine test cases across 26 data types and three task categories, with execution-based validation, pass@n, stability-adjusted accuracy, resource measures, and detailed error types (Hou et al., 12 Jun 2025). petscagent-bench goes further by using an agents-evaluating-agents architecture in which an evaluator agent communicates with a model-under-test agent through A2A and MCP protocols and runs a 14-evaluator pipeline over correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions (Zhang et al., 16 Mar 2026). These systems indicate that model-generated evaluation increasingly incorporates the operational environment of the target artifact rather than only its textual surface.

5. Human alignment, reliability, and systematic bias

The central empirical question is whether model-generated judgments align with human judgments. TICK provides one of the clearest gains from structured decomposition: exact agreement between LLM judgments and human preferences rose from 46.4% with direct scoring to 52.2% with checklists, while direct preference achieved 29.3%; weighted pairwise label distance also improved, and providing generated checklists to human annotators increased Krippendorff’s alpha from 0.194 to 0.256 on WildBench (Cook et al., 2024). Here, checklists improved both machine judgment and human consistency.

CG-Eval likewise validated its automated metric against expert annotation, with Gscore reaching a Kendall Tau of about 0.61 on a sample spanning 55 sub-disciplines and ten models (Zeng et al., 2023). GER-Eval found a more qualified pattern: LLMs generate coherent, task-aware rubrics and apply them consistently within models, but reliability declines in factual and knowledge-intensive settings, and closed-source models such as GPT-4o generalize better across models than open-weight systems such as Llama (Siro et al., 9 Feb 2026). This suggests that evaluator competence is uneven across domains even when rubric generation appears fluent and interpretable.

At the same time, bias remains a persistent concern. G-Eval explicitly reported that GPT-4-based evaluation can favor LLM-generated texts over human-written ones, even when human annotators prefer the human-written summaries; the study also noted very low human agreement in that comparison setting, with Krippendorff’s alpha of 0.07 (Liu et al., 2023). This does not invalidate model-generated evaluation, but it limits the interpretation of high evaluator-human correlation as a universal property.

A separate limitation concerns domain invariants that generic evaluators do not encode. In cryptographic Rust generation, only 23.3% of 240 generated samples compiled, CodeQL produced only two detections and both were false positives, whereas a crypto-specific rule-based analyzer identified vulnerabilities in 57% of compiled samples with zero false positives (Elsayed et al., 29 Apr 2026). The reported failures included nonce reuse, hardcoded secrets, unsafe error handling, and API hallucinations. This indicates that black-box judge quality is strongly contingent on whether the evaluator has access to domain-specific structure rather than only general code-analysis heuristics.

6. Efficiency, closed-loop improvement, and open research problems

A major driver of model-generated evaluation is compute efficiency. Reformulating generative benchmarks as cheaper NLU alternatives showed strong correlation between the original NLG tasks and multiple-choice or log-likelihood variants across mathematical reasoning, code generation, factual knowledge, and reading comprehension, while reducing evaluation time by over 35 times on average and by more than 170 times for code generation (Hangya et al., 4 Jun 2025). The same work also reported that a simple linear regressor using NLU scores and past NLG scores could predict NLG benchmark scores within 5% absolute error.

Efficiency gains increasingly feed into closed-loop generation. STICK uses TICK-style checklists for self-refinement and Best-of-$\text{score} = \sum_{i=1}^n p(s_i)\, s_i,$2 selection, yielding a +7.8% absolute gain on LiveBench reasoning and a +6.3 absolute improvement on WildBench in Best-of-$\text{score} = \sum_{i=1}^n p(s_i)\, s_i,$3 selection (Cook et al., 2024). AIGVE-MACS extends this principle to video through a multi-agent refinement pipeline in which evaluator feedback is used to revise instructions and regenerate videos, producing up to a 53.5% increase in average score on low-quality cases (Liu et al., 2 Jul 2025). Q-Save similarly binds score prediction to explanation generation, using chain-of-thought-style outputs to produce both ratings and attributions in a single pass (Wu et al., 24 Nov 2025). This suggests that evaluation is no longer purely post hoc; it is becoming a control signal in iterative generation systems.

Several open problems recur across the literature. Coverage remains incomplete: CG-Eval does not yet cover role-play, multi-turn dialogue, creative writing, or casual chat (Zeng et al., 2023). Dynamic rubric generation still treats oracle human-crafted rubrics as an upper bound, even when meta-judge-guided fine-tuning improves performance (Wang et al., 28 May 2026). Multimodal systems report rare failure modes and out-of-domain weaknesses, such as “janus” artifacts and remaining OOD surface-evaluation gaps in Gen3DEval (Maiti et al., 10 Apr 2025). More broadly, GER-Eval characterizes evaluation as a learned linguistic capability that is internally coherent within models but fragmented across them (Siro et al., 9 Feb 2026). A plausible implication is that future progress will require joint modeling of human evaluative language, domain-specific structure, and model-internal evaluative representations rather than a universal scalar judge.