HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

Abstract: As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward real-world healthcare applications. We introduce HealthAgentBench, a suite of 54 agentic healthcare tasks across 7 categories each with its unique environment. The benchmark suite spans diverse workflows throughout the patient journey and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given minimal instructions, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting. A final task success rate is reported to provide a single, interpretable metric for HealthAgentBench overall performance for each agent. Evaluating frontier agents on HealthAgentBench, we find that overall task success rate remains low, underscoring the difficulty of the suite. The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate. Beyond aggregate performance, HealthAgentBench reveals nuanced strengths and weaknesses across task categories. Frontier agents show promise in automatically developing research modeling pipelines over EHR data, but medical imaging remains especially challenging, particularly for Claude Code models, while Codex GPT-5.5 shows emerging capability. Tasks that combine large search spaces with compositional reasoning requirements remain difficult for all current agents. Together, these results suggest that HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark at https://github.com/microsoft/HealthAgentBench.

• The paper presents HealthAgentBench, a unified benchmark that rigorously evaluates AI agents through complex, realistic clinical workflows.
• It employs seven diverse task categories requiring autonomous tool use, multi-step planning, and compositional reasoning over heterogeneous data.
• Empirical results show top agents like GPT-5.5 achieve only 42% success, especially struggling with imaging tasks, highlighting significant potential for improvement.

HealthAgentBench: A Unified Evaluation Suite for Agentic Healthcare AI

Introduction and Motivation

HealthAgentBench is introduced as the first unified, agent-focused benchmark to rigorously assess AI agents in realistic, executable healthcare environments spanning the full patient journey. The paper positions HealthAgentBench as a remedy to limitations in existing benchmarks, which are either saturated, static in their evaluation (often single-shot QA), or limited to narrow clinical tasks, modalities, or workflows. As healthcare applications transition toward complex end-to-end agentic systems that need to operate over heterogeneous, multimodal clinical data, benchmarks must evolve beyond static, prompt-based paradigms to reflect this complexity and realism.

HealthAgentBench comprises seven diverse task categories, each emulating a different clinical workflow and data modality. Tasks demand autonomous exploration, tool use, planning, and compositional reasoning over raw healthcare data and environments. They are scored with binary success/failure metrics against human/expert labels or clinical baselines, providing a clear, interpretable cross-agent comparison.

Figure 1: HealthAgentBench is a unified evaluation framework for agentic healthcare AI, supporting realistic patient data environments and multi-step agent interactions.

Benchmark Design and Workflow

HealthAgentBench tasks are systematically selected and designed via a principled workflow emphasizing:

• Multi-step, agentic execution: Tasks are unsolvable with naive single-shot prompting.
• Realistic clinical workflows: Coverage spans data management, diagnostics, event modeling, and treatment planning.
• Modality and environment diversity: Tasks utilize 2D and 3D medical images, whole-slide pathology, free-text documents, and structured EHRs.
• Verifiability with low-chance success: Success is highly unlikely through random guessing, with aggregate success rates typically below 10% for chance.

Tasks are sourced both by transforming established benchmarks into agent-native workflows and by generating new tasks with synthetic or rule-injected clinical challenges. Each task is delivered in a package centered on a terminal environment, facilitating interaction with raw clinical data and domain-specific tools. Instructions are deliberately minimal, enforcing agent autonomy in strategy formation and solution execution.

A post-creation quality process eliminates trivial, non-informative, or ambiguous tasks, resulting in a robust, leak-proof evaluation pipeline.

Coverage and Positioning Among Benchmarks

HealthAgentBench is shown to occupy a unique position among healthcare agent benchmarks, providing the highest interaction realism and the broadest modality and workflow coverage (Figure 2). It substantially surpasses both question-answering and simulated-conversation benchmarks in both agent autonomy and raw clinical complexity. The suite’s coverage matrix demonstrates full span across text, EHR, imaging, and longitudinal data, with tasks ranging from micro-scale (single-record) to cohort-level, and cross-sectional to longitudinal timelines.

Figure 2: Comparative landscape of healthcare agent benchmarks; HealthAgentBench uniquely combines high interaction realism with broad, multi-modal and workflow coverage.

Empirical Evaluation of Frontier Agents

Ten current best-in-class agents—spanning the GPT-5 series and Anthropic’s Claude family—are evaluated across HealthAgentBench, utilizing three different harnesses (Codex, Claude Code, Copilot-CLI). All agents operate under stringent, security-tight settings (no browsing; opaque identifiers).

Aggregate findings:

• The best agent, Codex GPT-5.5, achieves a mean task success rate of approximately 42%—well below clinical deployment requirements and indicating substantial benchmark difficulty and headroom.
• Other high performers include Copilot-CLI Opus-4.8 and Copilot GPT-5.5 (36%, 35% respectively); the weakest, Codex GPT-5.4-mini, achieves only 16% success.

  Figure 3: Pooled task success rates for ten frontier agents on HealthAgentBench, highlighting the persistent difficulty of agentic healthcare benchmarks.

Notably, the cost and wall-clock execution time of agents do not strictly correlate with performance. Claude Code agents are both slower and more expensive (e.g. Sonnet-4.6 at 24 min/task) without corresponding gains in accuracy. The Pareto frontier for cost-efficiency is traced exclusively by GPT-5 agents.

Figure 4: Cost versus success rate; GPT-5 family traces the cost-optimal frontier, outstripping Claude Code agents in both accuracy and efficiency.

Task Category Breakdown and Model Bottlenecks

A detailed per-task breakdown reveals:

• Agent strength: Agents show robust performance in (1) EHR Event Modelling (auto-research over clinical data pipelines) and (2) EHR Format Conversion (ETL tasks); most agents reach 100% pass rates in the latter.
• Major bottlenecks:
  • Medical Imaging: CT, X-ray, and pathology slide tasks are substantially harder (mean agent success 17% vs. 49% for text tasks). Codex GPT-5.5 is superior, with up to 40% success in Pathology Tumor Area Selection, a marked improvement relative to Claude Code agents.
  • Large Search Spaces and Compositional Reasoning: Tasks such as EHR Data Quality Auditing and Clinical Trial Matching (searching for clinical trial eligibility among hundreds of database entries and protocols) highlight current agent struggles with combinatorial complexity and retrieval.
  • 

  • Figure 5: Cross-category agent performance profiles, showing nuanced task difficulties, effort (time), and cost tiers.

  • 

  • Figure 6: Imaging versus text task performance by model family. Codex GPT models lead in imaging; text tasks show more parity.

Imaging tasks frequently require multi-resolution, multi-step approaches, necessitating domain-specific perceptual strategies. Codex GPT-5.5 demonstrates emerging, pathologist-like reasoning in successful pathology trials, with nuanced hierarchical navigation and avoidance of naive pixel counting.

Search bottlenecks are quantitatively exposed in the EHR Data Quality Auditing task, where providing partial search hints significantly increases recall for all agents, suggesting that current agents do not yet employ efficient decomposition or intelligent search heuristics at scale.

Figure 7: Impact of narrowing the search space in EHR Data Quality Auditing; recall sharply increases when specific clues are provided.

Quantitative and Task-Specific Evaluation

Task-specific metric analyses confirm that, while sometimes matching or exceeding deep learning or human-engineered baselines in EHR event prediction (Figure 8), overall agent performance is heterogeneous and incomplete. The binary pass criteria in sensitive workflows (e.g., perfect recall or F1 thresholds) create a high bar, but nonetheless, models like GPT-5.5 exhibit capacity for state-of-the-art performance in certain machine-learning-centric tasks.

Figure 8: On EHR Event Modelling tasks, best agent matches or surpasses deep-learning baselines for all but the most challenging predictions.

Implications and Future Directions

The results highlight the substantial gap between current frontier agent capabilities and the requirements of real-world, high-stakes clinical settings. Specific implications include:

• Agent Robustness and Planning: Success in HealthAgentBench requires agents to autonomously construct and verify complex workflows, emphasizing the need for improvements in autonomous tool use, search heuristics, and compositional reasoning.
• Modality Integration: Multimodal reasoning remains a key unsolved challenge. Imaging-based tasks, especially those involving hierarchical, high-volume visual inspection, require novel strategies, potentially benefiting from tool augmentation, vision-model backends, or more specialized architectures.
• Evaluation Methodology: Binary, expert-benchmarked pass/fail gates provide high-fidelity signals for agent comparison, but future benchmarks could refine reward granularity to trace partial progress toward clinical utility.
• Agent Harness Choices: Significant differences in performance between agent harnesses (e.g., Codex vs Copilot vs Claude Code) suggest both model back-end and orchestration layer are independently relevant for practical deployment.

The benchmark framework can be readily extended—both in breadth (new modalities, new tasks, extended clinical pathways) and in integration (support for more agent harnesses, grounding in live EHR systems)—to track the rapid evolution of foundation model agents in healthcare. As such, HealthAgentBench functions as both a progress bar for field-wide agent capability and as a diagnostic instrument for bottlenecks.

Conclusion

HealthAgentBench establishes a high-standard, unified evaluation framework for agentic healthcare AI, spanning the diversity and complexity of real-world clinical tasks and data. Despite rapid progress, even the best current agents achieve low absolute task success (under 50%), especially in settings requiring perceptual grounding and large-scale search, highlighting major unsolved problems. The benchmark's design enables nuanced identification of strengths, weaknesses, and research priorities as agentic AI progresses toward deployment-ready, clinically reliable performance. The suite is positioned to become a foundational community standard for empirical evaluation and longitudinal tracking of frontier agents in healthcare.