Realistic Agent Evaluation

• Realistic agent evaluation is defined as a rigorous, task-grounded assessment of autonomous systems over long-horizon tasks, dynamic conditions, and comprehensive error metrics.
• It employs structured methodologies including tree-based rubrics, task-specific judge agents, and intermediate feedback to monitor detailed progress.
• It benchmarks agents across diverse domains such as search, software engineering, mobile applications, and multilingual workflows to guide robust, safe agent design.

Realistic agent evaluation is the rigorous, task-grounded assessment of autonomous systems that mirrors the complexity, variability, and requirements of real-world deployments. Such evaluation transcends simple and static test cases by incorporating long-horizon workflows, dynamically updated environments, multi-modal interactions, comprehensive error analysis, and reliable automated judgment. These methodologies have become essential as LLM-based agents increasingly operate in open-ended, operational settings—requiring detailed benchmarking across domains such as search, safety, alignment, privacy, health, mobile applications, software engineering, and multilingual or multi-user workflows.

1. Principles of Realistic Agent Evaluation

Realistic evaluation addresses two central challenges: designing tasks that faithfully capture genuine human information-seeking and operational complexity, and reliably judging agent outputs that are long, structured, and depend on time-varying, external content. The benchmarks that exemplify realism feature:

• Long-horizon, multi-step tasks. Realistic agentic benchmarks require dozens to hundreds of primitive actions (e.g., Mind2Web 2: median 110 webpages/task, search horizons exceeding 375 pages in human studies) (Gou et al., 26 Jun 2025).
• Real-time, dynamic environments. Evaluation accounts for rapidly changing online data (product prices, news, time-sensitive details), going beyond static ground-truth assessments.
• Holistic, structured judgment. Outputs are assessed for both correctness and attribution—answers must be factually correct, satisfy task constraints, and be properly cited to live web sources or environment states (Gou et al., 26 Jun 2025).
• Error taxonomy. Evaluation must categorize incompleteness, violated criteria, invalid/missing attribution, and synthesis/retrieval failures.
• Automated, agentic judging frameworks. Tools such as Agent-as-a-Judge use modular agents with rich file/workspace traversal, hierarchical rubric trees, and step-wise evidence aggregation to rival or surpass human annotators in consistency and coverage (Zhuge et al., 2024, Gou et al., 26 Jun 2025).

These features stand in contrast to prior benchmarks which restrict evaluation to <10 steps, static data, or coarse binary accuracy, systematically overestimating real-world agent capability (Garg et al., 10 Oct 2025).

2. Architectural and Methodological Advances

Recent agentic evaluation frameworks operationalize realism in both benchmark construction and judgment:

• Tree-structured rubrics. Mind2Web 2 assigns each task an explicit tree rubric separating leaf criteria (e.g., budget, URL evidence) and gates/averages non-critical branches. Quantitative metrics include partial completion (average rubric root score) and success rate (fraction of fully correct answers), mathematically formalized per task (Gou et al., 26 Jun 2025).
• Task-specific judge agents. Rubric instantiation is largely automated via LLMs, with Python-based Extractor/Verifier modules and self-reflection (Gou et al., 26 Jun 2025, Zhuge et al., 2024).
• Intermediate feedback. Agent-as-a-Judge aligns requirement-level progress with dependency graphs, producing dense intermediate supervisory signals instead of endpoint measurements (Zhuge et al., 2024). In DevAI, this enables precise diagnosis of process failures and supports process-supervised reward modeling.
• Automated substate tracking for mobile/GUI agents. AutoEval decomposes each mobile UI task to a chain/tree of substates mapped to screen elements, using vision-LLMs for screenshot analysis with 93% coverage of human-labeled reward elements and 94% judge accuracy (Sun et al., 4 Mar 2025).
• Scenario-driven adversarial risk assessment. Realistic safety and self-replication risk evaluations (e.g., OpenAgentSafety, Agent Matrix) simulate genuine production environments (Kubernetes clusters, live shell/coding/browsing) and apply operational stressors to elicit behaviors only emergent in real workflows (Zhang et al., 29 Sep 2025, Vijayvargiya et al., 8 Jul 2025).

3. Domain-Specific Realism: Benchmarking Across Fields

Research has instantiated realistic agent evaluation in various domains:

• Agentic search: Mind2Web 2's 130 tasks across 24 subdomains require extensive synthesis, up-to-date citations, and complex answer structures. Systems like OpenAI Deep Research achieve 54% partial completion and 28% success rate in 8.4 min/task, compared to human 79%/54% in 18.4 min (Gou et al., 26 Jun 2025).
• Software engineering: Mutation-based evaluation (e.g., Saving SWE-Bench) transforms overspecified GitHub issues to terse, realistic user queries derived from IDE telemetry, correcting success rate overestimations by up to 50% (Garg et al., 10 Oct 2025). End-to-end development frameworks (E2EDevBench) combine migrated tests and fine-grained, LLM-based requirement verification, revealing that workflow design governs implementation rates and planning/requirement omission are dominant failure modes (Zeng et al., 6 Nov 2025).
• Mobile agents: AutoEval replaces handcrafted reward code with structured substate representations and vision-language judging, achieving automation unmatched by manual methods (Sun et al., 4 Mar 2025).
• Multilingual agents: Ticket-Bench localizes all entities, user profiles, and evaluation criteria across six languages, uncovering cross-lingual disparities and rewarding consistency over single-run maxima (Almeida et al., 17 Sep 2025).
• Privacy and safety: OA-Safety and PrivacyLens-Live complement rule-based outcome analyses with LLM-as-Judge trajectory scoring to surface both overt and subtle unsafe behaviors at scale (Vijayvargiya et al., 8 Jul 2025, Wang et al., 22 Sep 2025).
• Clinical/health interaction: MedAgentSim models realistic doctor-patient-measurement workflows, requiring explicit test orders and multi-turn, memory-augmented dialog in diagnostic reasoning (Almansoori et al., 28 Mar 2025).
• Adaptive user simulation: SAGE leverages both top-down persona profiles and bottom-up business/infra knowledge, producing multi-turn, realistic user simulations and surfacing 33% more bugs than ablations (Shea et al., 13 Oct 2025).
• Mental health: AnnaAgent simulates seekers with dynamic emotion modulator and multi-session memory, showing higher anthropomorphism and personality fidelity in counseling simulations than prior baselines (Wang et al., 31 May 2025).
• Data science automation: DSAEval presents a 641-task, multimodal, multi-query benchmark with reasoning/code/result scoring; image modality increases CV task performance up to 11.3% (Sun et al., 20 Jan 2026).

4. Metric Suites and Result Interpretation

Realistic agent evaluation depends on multi-dimensional, formally defined metrics:

• Partial completion: Average rubric root score, capturing incremental progress (Gou et al., 26 Jun 2025).
• Success rate: Fraction of tasks with all key criteria satisfied (Gou et al., 26 Jun 2025).
• Pass@k and consistency: Probability of success in at least one of k retries; variance across languages or trials for stability (Mohammadi et al., 29 Jul 2025, Almeida et al., 17 Sep 2025).
• Edge-case/attack break-rate: Share of security probing sessions producing agent failures (Wang et al., 19 Jul 2025).
• Risk scores (safety/replication): Composite scores blending frequency/severity of unsafe actions (OA-Safety, Agent Matrix φ_R risk score) (Zhang et al., 29 Sep 2025, Vijayvargiya et al., 8 Jul 2025).
• Tool-use metrics: Function selection accuracy, parameter extraction F1, and end-to-end invocation rates (Mohammadi et al., 29 Jul 2025).
• Efficiency/cost-effectiveness: Task performance per token and per USD, revealing trade-offs between raw accuracy and resource consumption (Sun et al., 20 Jan 2026).
• Human-centric metrics: PULSE framework leverages augmented statistical inference to combine sparse human labels and ML predictions, yielding robust confidence intervals on user satisfaction and surfacing benchmark-human gaps (Chen et al., 10 Oct 2025).
• Emulator/refiner-driven failure discovery: ALI-Agent automates the emulation and refinement of adversarial scenarios to probe long-tail alignment breakdowns, with ablative analysis revealing 20–15% drops in detected misalignments when memory or refinement is removed (Zheng et al., 2024).

5. Error Analysis and Forward Directions

Realistic agent evaluation mandates detailed error taxonomies and iterative improvement:

• Error types: Incompleteness, explicit criteria violations, hallucinated/missing attribution, retrieval and synthesis errors are systematically catalogued (Mind2Web 2, OA-Safety) (Gou et al., 26 Jun 2025, Vijayvargiya et al., 8 Jul 2025).
• Scenario-driven risk amplification: Real benchmarks expose time-varying, survival-driven risks undetectable in synthetic tests; e.g., model safety drops by 10–20% on time-sensitive tasks without live browsing (Gou et al., 26 Jun 2025, Zhang et al., 29 Sep 2025).
• Planning and memory bottlenecks: Omissions in requirements, plan drift, or premature self-verification drive >50% of failures in end-to-end tasks (Zeng et al., 6 Nov 2025).
• Adaptive benchmarking: Live or continuously refreshed benchmarks (BFCL, WebArena, DSAEval) prevent saturation and enable timely analysis of evolving agent behaviors (Sun et al., 20 Jan 2026, Yehudai et al., 20 Mar 2025).
• Best practices: Employ dynamic, hybrid evaluation (agent-judge + migrated tests), validate judge reliability against human consensus, track environment and data contamination, and integrate cost, safety, and compliance as first-class metrics (Zhuge et al., 2024, Mohammadi et al., 29 Jul 2025, Sun et al., 4 Mar 2025).

6. Implications for Agent Design and Research

The deployment of agents evaluated under realistic protocols has widespread methodological and operational consequences:

• Benchmark realism corrects performance optimism—mutation of overly verbose formal queries reduces success rates by 10–50% (Garg et al., 10 Oct 2025).
• **Automated judge agents rival human experts at 3% of the cost and time, supporting scalable, reproducible pipeline integration (Zhuge et al., 2024, Gou et al., 26 Jun 2025).
• Scenario-driven safety and alignment testing yields actionable safeguards—structured reasoning, explicit resource constraints, and red-team threat modeling are effective at reducing uncontrolled replication and safety violations (Zhang et al., 29 Sep 2025, Vijayvargiya et al., 8 Jul 2025, Zheng et al., 2024).
• Multilingual, culturally grounded evaluation is essential for globally deployable agents, with task-localization, multi-run metrics, and cross-lingual analysis exposing nontrivial gaps even among top-tier models (Almeida et al., 17 Sep 2025).
• User-centric approaches (e.g., PULSE) provide robust human-in-the-loop satisfaction assessment, surfacing discrepancies between in-the-wild usage and static benchmarks and guiding design optimization (Chen et al., 10 Oct 2025).
• Memory and planning modules require targeted research—errors attributable to planning and requirement comprehension point toward the need for improved, robust agent reasoning and dynamic goal tracking (Zeng et al., 6 Nov 2025).

7. Summary Table of Key Benchmarks and Features

Benchmark/Framework	Domain	Realism Features	Primary Metrics
Mind2Web 2 (Gou et al., 26 Jun 2025)	Web search	Long horizon, time-varying, agent-judge	Partial completion, success rate
DevAI (Zhuge et al., 2024)	AI code tasks	Hierarchical requirements, modular judge	M_I, M_D, alignment rate
AutoEval (Sun et al., 4 Mar 2025)	Mobile agents	SSR-based, VLM/LMM judge, no manual reward	Substate/task completion rates
OA-Safety (Vijayvargiya et al., 8 Jul 2025)	Safety	Multi-tool, real sandboxes, adversarial NPCs	Unsafe rate, rule vs. LLM judge
Ticket-Bench (Almeida et al., 17 Sep 2025)	Multilingual FC	Localized entities/profiles, multi-run	Accuracy, pass@k, cross-lingual
MedAgentSim (Almansoori et al., 28 Mar 2025)	Clinical	Multi-agent sim, imaging, self-improve	Diagnostic accuracy, F1
DSAEval (Sun et al., 20 Jan 2026)	Data science	Multimodal, multi-query, multi-metric	Performance, efficiency, cost-effectiveness

Each of these frameworks integrates features that jointly define realistic agent evaluation: dynamically curated environments, exhaustive and structured rubrics, automated and reliable judgment, error taxonomy and interpretability, reporting across multiple axes (correctness, attribution, safety, efficiency), and extensibility toward new domains and protocols. The convergence toward such methodologies is essential for accurately assessing, benchmarking, and improving the next generation of autonomous agentic systems.