---
title: 'EvalAgent: Automated Evaluation & Criteria'
url: https://www.emergentmind.com/topics/evalagent
type: topic
---

# EvalAgent: Automated Evaluation & Criteria

“EvalAgent” has been used for two distinct systems in recent arXiv literature. One usage refers to a retrieval-grounded framework for discovering implicit, task-specific evaluation criteria for structured writing prompts; the other refers to an AI assistant that automates the generation of evaluation artifacts for AI agents, including metrics, executable code, and reports. In both cases, the common theme is that evaluation is treated as a first-class technical problem rather than as a thin wrapper around a single judge prompt, but the two systems operate at different layers of the evaluation stack [2504.15219] [2605.11378].

## 1. Scope and disambiguation

The earlier “EvalAgent” is a framework for **criterion discovery**. Its central question is not how to score a response once a rubric already exists, but how to discover the criteria that should be used in the first place, especially when high-quality responses depend on implicit conventions that are not stated in the prompt. The later “EvalAgent” is a framework for **evaluation generation**. Its central question is how to automatically build an evaluation workspace for an agent, including test scenarios, instrumentation, metric code, and reports, from agent code, execution traces, and user requirements [2504.15219] [2605.11378].

| Usage | Main object | Primary output |
|---|---|---|
| EvalAgent [2504.15219] | Structured writing evaluation | Ranked evaluation criteria |
| EvalAgent [2605.11378] | Agent evaluation automation | Plans, test cases, code, reports |

This naming overlap is substantive rather than accidental. Both systems reject the assumption that evaluation can be reduced to a single scalar judgment over a final output. The earlier system expands the criterion set; the later system expands the evaluation pipeline itself. This suggests that “EvalAgent” has become associated with evaluation as an active, decomposed, and tool-augmented process rather than a static metric.

## 2. EvalAgent as retrieval-grounded criterion discovery

In “EvalAgent: Discovering Implicit Evaluation Criteria from the Web” [2504.15219], the problem setting is structured writing evaluation. The paper argues that conventional setups typically capture explicit properties such as accuracy, coherence, formatting, or direct instruction following, but often miss **implicit criteria**: omitted-but-desirable properties that are still specific to the task and genuinely useful for judging quality. The motivating examples are prompts such as drafting an academic talk, where humans also expect features like a compelling opening, clear research questions, and a strong takeaway even when those are not written into the instruction.

The system formalizes evaluation around a criterion set $\mathcal{C}$ for each instruction $\mathbf{x}$, with criterion-based scoring performed by a function $f(\mathbf{x}, \mathbf{y}, c)$ over prompt, response, and criterion. Its main contribution is therefore upstream of judge design: it aims to discover a better task-specific criterion set rather than proposing a new scalar evaluator. The paper characterizes the target criteria as grounded in retrieved evidence, diverse, long-tail, implicit, and lexically specific. That position distinguishes it from instruction decomposition systems that restate explicit constraints and from prompted LLM baselines that generate generic checklists from parametric knowledge alone [2504.15219].

Two variants are defined. **EvalAgent-WebOnly (EA-Web)** uses only web-derived criteria. **EvalAgent-Full (EA-Full)** merges those web-derived criteria with an LLM-generated criterion set, specifically the stronger LLM-$n$ baseline, and reranks the union. The rationale is that web retrieval is particularly good at surfacing implicit, domain-specific criteria, whereas directly prompted LLM generation is better at recovering criteria already implied by the instruction [2504.15219].

## 3. Pipeline, metrics, and empirical profile of the criterion-discovery system

The criterion-discovery pipeline has five stages: **query generation, retrieval, URL filtering, evidence extraction and summarization, then criterion synthesis and ranking**. From an instruction $\mathbf{x}$, the Query Generator produces three conceptual Google queries oriented toward “how to …” or “what are …” advice rather than topical background. For each query, the system retrieves 30 URLs through the Google Search API, scores each URL with an LLM on a **goodness/expertise score on a 1–5 scale** and an **instruction relevance score on a 0–2 scale**, keeps the top 5 URLs, answers the query from each retained page, summarizes the answers conservatively, rewrites the resulting advice into yes/no-evaluable criteria beginning with “the response should …”, filters out inapplicable criteria, and ranks the remainder by relevance to the original instruction. The implementation uses **gpt-4o-mini-2024-07-18** throughout the pipeline [2504.15219].

The paper evaluates discovered criteria using several automatic and human-judged properties. **Specificity** is operationalized with normalized inverse word frequency. **Implicitness** is measured as $1-\text{WO}$, where
$$
\text{WO}(x, c) = \frac{|\text{W}(x) \cap \text{W}(c)|}{|\text{W}(c)| + \epsilon}.
$$
**Actionability** is defined through whether a model can revise a response to satisfy a criterion it initially failed. **Recall of human criteria** measures entailment coverage of human-written criteria. Human evaluation uses **Utility** and **Obviousness**; the paper reports annotation over about **700 criteria across 54 unique instructions**, with Fleiss’ kappa averages of $O=0.88, U=0.64$ for instruction decomposition and around $O=0.25, U=0.26$ for EvalAgent, similar to human-written criteria [2504.15219].

Empirically, EA-Web is more implicit and more specific than LLM-only baselines. In the aggregate comparison table, **EA-Web** has $S=0.14$, $I=0.88$, $O=0.58$, and $U=2.46$, while **LLM-$n$** has $S=0.09$, $I=0.82$, $O=0.78$, and $U=2.67$. On actionability, EA-Web yields lower initial pass rates but larger post-edit improvement; for **Claude-3.5-Sonnet**, the paper reports **Initial 0.51, Edited 0.90, $\Delta=0.39$, $A=0.79$**. On human-criterion recall, **EA-Full** reaches **0.74**, exceeding **LLM-60 at 0.69** and **LLM-30+30 at 0.68**. The reported interpretation is that web-grounded criteria expose harder, less obvious deficiencies while remaining actionable for revision [2504.15219].

## 4. EvalAgent as automated generation of agent evaluations

In “An Empirical Study of Automating Agent Evaluation” [2605.11378], “EvalAgent” denotes a different artifact: an AI assistant that automates the **end-to-end agent evaluation pipeline**. The paper’s argument is that agent evaluation is structurally harder than ordinary LLM evaluation because it must assess multi-step behavior involving tool use and intermediate reasoning. Simply prompting frontier coding assistants is reported to be insufficient: without evaluation-specific knowledge, they achieve only about a **30% execution success rate** and produce over-engineered evaluations averaging **12+ metrics per agent**.

The system addresses this by encoding domain expertise as **evaluation skills**, defined as self-contained packages of **procedural instructions, reusable code and templates, and dynamically retrieved API documentation**. These skills are loaded stage by stage through progressive disclosure. The resulting pipeline has six stages: **evaluation planning, test case generation, agent instrumentation, trace collection and processing, evaluation code generation, and reporting**. The input consists of agent source code, user requirements, and execution traces, generally in **OTEL/OpenTelemetry-compatible format**; the output is a complete evaluation workspace including a plan, test scenarios, instrumentation, executable metric code, stored results, and a report [2605.11378].

The emphasis on traces is central. The paper argues that evaluating only final answers misses robust intermediate behavior such as tool use, state changes, and error recovery. EvalAgent therefore operates over real traces, not just agent source code or one final artifact. Its reusable code patterns include OTEL trace parsing, conversion of traces into DeepEval `LLMTestCase` objects, and scaffolding for both deterministic metrics and **GEval / BaseMetric** custom metrics. The system also uses **Context7** for live API documentation retrieval, which the paper treats as necessary because evaluation libraries such as DeepEval, LiteLLM, and Bedrock APIs change faster than static model knowledge [2605.11378].

## 5. AgentEvalBench, meta-evaluation, and reported results

To evaluate evaluation generators, the paper introduces **AgentEvalBench**, a benchmark comprising **20 real-world agents** spanning **9 frameworks**, **14 domains**, and **3 complexity tiers**, each paired with **5 test scenarios** and **2 requirement types**, generic and specific. It also introduces a meta-evaluation framework based on pairwise comparison across five dimensions: **User Requirement Fulfillment (15%)**, **Metric Relevance (30%)**, **Code Quality & Complexity (25%)**, **Plan Quality (15%)**, and **Plan-Code Alignment (15%)**. If a method has no planning stage, the weights are renormalized to **25%**, **40%**, and **35%** for requirement fulfillment, metric relevance, and code quality respectively [2605.11378].

The central executability metric is **Eval@1**, defined operationally as the fraction of evaluation-generation attempts in which the produced evaluation code both executes successfully on the first run and yields meaningful, non-vacuous results. This is stricter than ordinary code-generation pass rates because constant, all-zero, synthetic, mocked, or otherwise behaviorally meaningless evaluations are counted as failures. The paper also defines win-tie rate as
$$
\text{Win-Tie Rate} = \frac{\text{wins} + 0.5 \times \text{ties}}{\text{total}} \times 100.
$$
These metrics are used to compare EvalAgent against four baselines: a single-turn LLM, an agentic assistant operating on source code only, a one-stage trace-based generator, and a two-stage planner-then-implement baseline without evaluation skills [2605.11378].

The reported gains are large. With Sonnet 4.5, **Eval@1 improves from 17.5% to 65.0%** relative to the weakest trace-based baseline, and removing evaluation skills drops it from **65% to 30%**. Human experts prefer EvalAgent over the strongest no-skills two-stage baseline in **79.5%** of comparisons, with **10.5% ties**, and the paper reports **Fleiss’ $\kappa = 0.923$**. On generic requirements, EvalAgent produces far smaller artifacts than the two-stage baseline: **289 ± 67 LOC**, **2.0 ± 0.2 files**, and **1.4 ± 1.2 metric classes**, compared with **1,902 ± 622 LOC**, **8.6 ± 5.4 files**, and **12.0 ± 5.6 metric classes** for the baseline. A Context7 ablation is especially strong: for Sonnet, Eval@1 falls from **65.0% with** Context7 to **20.0% without** it [2605.11378].

## 6. Conceptual significance of the two EvalAgent lineages

The two EvalAgent systems address different bottlenecks in evaluation. The criterion-discovery system operates at the level of **what should be judged**: it expands the criterion set beyond explicit instruction constraints by mining expert-authored web guidance. The evaluation-generation system operates at the level of **how evaluation should be built**: it automates the production of plans, traces, code, and reports for assessing agent behavior. One system enriches the semantic target of evaluation; the other automates the engineering substrate of evaluation [2504.15219] [2605.11378].

Taken together, the two usages point toward a broader research pattern. Evaluation is no longer treated merely as a scalar output of a judge model. Instead, it is decomposed into criterion induction, evidence gathering, trace processing, code synthesis, metric design, and reporting. This suggests that future “evaluation agents” will likely combine both layers: discovering richer task-specific criteria and constructing executable, trace-aware evaluation pipelines that can apply those criteria to complex multi-step systems.

Source: https://www.emergentmind.com/topics/evalagent