HealthAdminBench: Healthcare Admin Benchmark

• HealthAdminBench is a domain-specific evaluation framework that assesses CUAs automating complex, multi-portal healthcare administrative workflows.
• It decomposes 135 multi-step tasks into granular subtasks with metrics like SSR and TSR to measure performance across legacy systems.
• Experimental results highlight reliability gaps in memory, document handling, and clinical reasoning, guiding future improvements in agents.

HealthAdminBench is a benchmark designed to evaluate computer-use agents (CUAs) powered by LLMs on realistic, high-stakes healthcare administrative workflows common in the United States. The benchmark systematically assesses the end-to-end automation challenges posed by administrative tasks that constitute over $1 trillion in annual spending. HealthAdminBench fills a critical gap by providing a domain-specific evaluation framework that goes beyond clinical LLM tasks, targeting multi-portal, multi-step workflows that span legacy systems such as EHRs, payer portals, and fax platforms (Bedi et al., 10 Apr 2026).

1. Motivation and Core Challenges

Healthcare administration involves complex and heterogeneous processes, including prior authorization (PA), claims appeals and denials management, and durable medical equipment (DME) order processing. These workflows entail cross-system navigation (e.g., EHR to payer portal to fax), rigid data validation, extensive document handling, and frequent requirements for clinical and policy-grounded reasoning. Traditional automation approaches are hampered by the lack of unified APIs, policy variation, and the need for precise user-level interaction.

CUAs based on LLMs offer the prospect of bridging these gaps by operating over graphical interfaces and executing user-level actions—such as clicks, data entry, uploads/downloads—within real or simulated legacy environments. However, key challenges persist:

• Cross-system coordination without unified APIs
• Schema validation and atomic “commit” actions
• Reliable document download, upload, and attachment
• Long-horizon memory and reasoning across 10–20+ subtasks
• Subtasks requiring clinical reasoning (e.g., dose calculation, eligibility evaluation)

HealthAdminBench was developed to quantify how well state-of-the-art CUAs handle these requirements and to illuminate persistent reliability gaps in end-to-end administrative workload automation (Bedi et al., 10 Apr 2026).

2. Benchmark Design and Environment Architecture

HealthAdminBench comprises four deterministic, web-based environments built using Playwright and the REAL framework:

• Electronic Health Record (EHR-like)
  • Worklists for PA, denials, and DME
  • Tabbed patient information (diagnoses, services, referrals, documents, insurance, notes)
  • Typical operations: navigating lists, extracting codes, downloading documents, updating records
• Payer Portal A (Anthem-inspired)
  • Tabs for eligibility, PA, appeals, documents, attachments
  • Structured form input, strict data validation, confirmation dialogs
• Payer Portal B (Availity-inspired)
  • Alternative UI structure and naming, unique test-ids, policy logic variations
• Fax System (RightFax-inspired)
  • File upload, fax number entry, cover sheets, delivery status feedback

Agents observe either full pixel-rendered screenshots or a structured accessibility tree indexed by stable data-testid identifiers, facilitating symbolic interaction. Actions are encoded as natural language strings (e.g., click([id]), fill([id], value)) which map to low-level browser operations (Bedi et al., 10 Apr 2026).

3. Task Suite and Subtask Decomposition

The benchmark’s task suite consists of 135 expert-defined, multi-step administrative tasks across three main domains:

• Prior Authorization (60 tasks)
• Appeals and Denials Management (60 tasks)
• DME Order Processing (15 tasks)

Each task is decomposed into fine-grained, verifiable subtasks for robust, objective assessment, totaling 1,698 evaluation points (1,177 deterministic, 521 LLM-judged). The subtask taxonomy comprises:

1. Information Retrieval (419)
2. Documentation (515)
3. Form Completion (292)
4. Task Resolution (200)
5. Document Handling (149)
6. Clinical Reasoning (123)

Tasks scale in complexity, from simple eligibility checks and documentation updates to full-cycle submissions, cross-system document transfer, and terminal workflow closure (e.g., successful submission and confirmation recording) (Bedi et al., 10 Apr 2026).

4. Evaluation Methodology and Metrics

Success criteria reflect both atomic (subtask) and holistic (task) reliability:

• Subtask Success Rate (SSR): $$\text{SSR} = \frac{\#~\text{successful subtasks}}{\#~\text{total subtasks}}$$
• Task Success Rate (TSR):

$$\text{TSR} = \begin{cases} 1 & \text{if all subtasks succeed} \ 0 & \text{otherwise} \end{cases}$$

Aggregate results report mean SSR and TSR with 95% bootstrap confidence intervals. Head-to-head agent comparisons and ablation studies (e.g., on observation modality or prompting regime) use bootstrap resampling for significance estimates (Bedi et al., 10 Apr 2026).

5. Agent Configurations, Prompting, and Observation Paradigms

Seven agent configurations were benchmarked, spanning both “native” CUAs (Claude Opus 4.6 CUA, GPT-5.4 CUA) and harness-based agents (Kimi K2.5, Qwen-3.5-27B, Gemini 3.1 Pro):

• Prompting Settings:
  • Task Description only: concise, high-level goal
  • Task Description + Portal Guidance: adds form-filling rules, document transfer protocols, navigation best practices
  • Task-Specific Step-by-Step: granular procedural instructions, used for fine-tuning but not for main evaluation
• Observation Modes:
  • Screenshot-only: pixel-level rendering, element inference required
  • Accessibility-tree: structured semantic UI
  • Performance varies significantly by both agent architecture and prompting/observation modality. For instance, accessibility-tree inputs with guidance can boost TSR by up to 15 percentage points relative to raw screenshots alone (Bedi et al., 10 Apr 2026).

6. Experimental Findings and Agent Performance

Despite high subtask success, end-to-end reliability is limited, underscoring brittle dependencies across workflow chains.

Overall agent results (screenshot+guidance):

• Claude Opus 4.6 CUA: TSR=36.3% (highest), SSR=79.2%
• GPT-5.4 CUA: TSR=26.7%, SSR=82.8% (highest)
• Harnessed agents typically yield much lower TSRs (e.g., GPT-5.4 harness: 6.7%)

By task type:

Agent	PA TSR	Appeals/Denials TSR	DME TSR
Claude Opus 4.6 CUA	38.3%	25.0%	73.3%
GPT-5.4 CUA	26.7%	18.3%	60.0%
Kimi K2.5	18.3%	13.3%	13.3%

By subtask category: Information Retrieval and Document Handling approach ceiling level performance for top agents (≈96%), while Clinical Reasoning (Claude: 61.3%, GPT-5.4: 77.0%) and Task Resolution remain the major failure points.

Fine-tuning impact: Domain-specific supervised fine-tuning on 100 step-by-step demonstrations enabled Qwen-3.5-Kinetic-SFT to jump from 17.1% to 40.0% TSR (SSR=83.3%) on a held-out set, suggesting substantial gains from targeted in-domain supervision (Bedi et al., 10 Apr 2026).

7. Failure Analysis and Future Directions

Key Failure Modes:

1. Hidden long-term dependencies: Agents fail to carry needed information from EHR to portal, or across subtasks.
2. Document operation avoidance: File upload/download steps are often omitted, leading to incomplete workflows.
3. Memory loss over long horizons: Agents forget earlier values or decisions, underutilizing available scratch memory.

These lead to a pronounced gap between SSR (atomic correctness) and TSR (holistic reliability): a single step failure in a 10–20 subtask chain leads to end-to-end task failure. “Clinical Reasoning” and “Task Resolution” subtasks are especially vulnerable to these gaps (Bedi et al., 10 Apr 2026).

Recommendations for Next Steps:

• Explicit memory/state mechanisms: Systems that enable agents to persist and retrieve information across long workflows
• Dedicated document-handling tools: High-level primitives for reliable download, upload, and PDF parsing
• Robust UI grounding: Multimodal (vision-language) models to mitigate UI drift and instability
• Interactive error recovery: Agents equipped to self-correct and resume after intermediate failures
• Human-in-the-loop oversight: Tools for seamless exception handover to human staff during automation
• Enhanced realism: Incorporation of MFA/CAPTCHA, session expiry, and dynamic business rules for greater production fidelity

Through open-sourcing of environments, code, and a leaderboard (https://healthadminbench.stanford.edu), HealthAdminBench aims to anchor systematic progress toward safe, robust automation of healthcare administration in real-world settings (Bedi et al., 10 Apr 2026).

8. Related Benchmarks and Positioning

HealthAdminBench builds on and extends the landscape of agent-oriented healthcare benchmarks:

• CHI-Bench: Focuses on policy-rich, multi-role, dialog-centric healthcare workflows across PA, utilization management, and care management, featuring a high-fidelity simulator and a managed-care operations handbook. CHI-Bench introduces the concepts of “policy density,” modular tool surfaces, multi-agent role composition, and explicit dialogue modeling. Its findings (e.g., best pass@1=28.0%, strict pass^3 < 20%) suggest similar reliability gaps for enterprise-scale AI agents (Chen et al., 15 May 2026).
• MedAgentBench: Targets agent performance on FHIR-compliant, patient-centered clinical EHR tasks (e.g., information retrieval, order entry, documentation), featuring 300 tasks and 100 real-patient records. While MedAgentBench assesses single-app, EMR API-based workflows, it notes that high-level administrative flows such as prior authorization and billing remain out of scope, with action-oriented task performance (e.g., 70% overall, query SR 84%, action SR 56% for Claude 3.5 Sonnet v2), still trailing real-world requirements (Jiang et al., 24 Jan 2025).

A plausible implication is that HealthAdminBench, by providing GUI-based, multi-system administrative tasks with fine-grained observable outputs, sets the stage for closing the technical and reliability gaps exposed by both prior GUI and API-based benchmarks.