CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

Abstract: End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.

• The paper introduces a high-fidelity simulation environment (X-World) to evaluate AI agents managing complex, multi-role, policy-rich healthcare workflows.
• It details quantitative evaluations showing low pass rates, non-determinism, and significant performance drops during repeated trials.
• The study highlights operational risks in policy compliance and coordination, urging further innovations in agent memory and hybrid system designs.

X-Bench: Evaluating Agentic Automation for Policy-Dense Healthcare Workflows

Motivation and Context

The paper "CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?" (2605.16679) explores the formidable challenges in automating U.S. healthcare workflows with agentic AI. Administrative workflows such as prior authorization (PA), utilization management (UM), and care management (CM) are characterized by structural complexity, dense policy requirements, multi-role compositions, and multilateral interactions—elements largely unaddressed in conventional benchmarks. The operational inefficiencies in these domains stem from irreducible coordination problems, intricate regulatory constraints, and workflow handoffs that create numerous failure points. Current frontier agents show impressive long-horizon capabilities on coding and web benchmarks but fail to generalize to enterprise settings with high policy density and irreversible state transitions.

Benchmark Construction and Architecture

X-Bench introduces a high-fidelity simulation environment (X-World engine) with 20 healthcare apps exposed through 87 MCP tools and 151 APIs, supporting realistic multi-role workflows. The benchmark covers three domains:

• Provider Prior Authorization: From clinical evidence assembly to submission, response handling, peer-to-peer negotiations, and appeals.
• Payer Utilization Management: Intake normalization, triage, nurse/MD review, peer-to-peer clinical review, determination, correspondence, and appeals.
• Care Management: Intake, chart review, multi-turn patient outreach, structured assessment, and care plan authoring within the NANDA-I/NOC/NIC framework.

Each agent operates through domain-scoped tool surfaces and is equipped with a 1,279-document Managed-Care Operations Handbook skill, encompassing workflow playbooks, medical libraries, and platform usage tutorials. Task construction is grounded in clinical realism and policy alignment, with staged human review for annotation and rubric validation.

Evaluation Protocol

Agents are evaluated across 30 harness/model configurations, including proprietary stacks (e.g., Claude Code, Codex, Gemini CLI) and open-source agents (Hermes, OpenClaw, DeepAgents) paired with frontier (GPT-5.5, Claude Opus 4.6/4.7, Gemini 3.1 Pro) and open-weight models (GLM-5.1, DeepSeek V4 Pro, Kimi K2.6, Qwen 3.6 Max, Grok 4.3). Metrics include pass@1, pass@3, and strict pass^3 for reliability. Additionally, agent performance is probed in end-to-end provider-payer arena runs, marathon sessions (multi-case, long-horizon continuity), skill ablation studies, and MCP-to-CLI surface transformation experiments.

Empirical Results

Key quantitative findings demonstrate that frontier agents are far from operationally competent in policy-dense healthcare workflows:

• Best agent performance: 28.0% pass@1 (Claude Code + Opus 4.6); no agent exceeds 20% at strict pass^3.
• Domain breakdown: UM peaks at 41.3% for Opus 4.6; PA at 29.3% for GPT-5.5; CM at 32.0% for Opus 4.7 but collapses under hard refusal consent scenarios.
• Reliability collapse: Repeat trials produce significant drops, e.g., Opus 4.6 (28.0% → 18.7%) and GPT-5.5 (20.9% → 9.3%), exposing non-determinism problematic for deployment.
• End-to-end arena: Two-agent PA provider-payer runs produced 0% successful automation, highlighting cross-role protocol and information asymmetry bottlenecks.
• Marathon runs: Agents fail to maintain working memory and drive cases to terminal states in multi-case sessions, slumping far below isolated-task baselines.

Skill ablation experiments reveal that access to structured operational policy documents is critical for certain domains (e.g., UM), but can induce over-verification and decision refusal in PA. The interplay between skill complexity and cognitive overload remains unresolved.

Failure Mode Analysis

A comprehensive taxonomy exposes agent-side failure patterns:

• Clinical-Reasoning errors (35.4%): Erroneous medical/protocol judgment despite correct tool use and evidence access, notably concern-mined consent in CM hard-refusal scenarios.
• Workflow-Completion errors (23.3%): Incomplete execution of terminal steps despite narrative reasoning, reflecting "completion theater."
• Policy-Compliance errors (13.2%): Literal misreading of cited criteria and inadequate policy citation in structured artifacts.
• Tool-Use-Error (10.7%): Malformed tool calls or misrouted function invocation, concentrated in DeepAgents rows.
• Abstain-or-Stuck (15.6%): Timeouts, looping, or explicit refusal to act.
• Hallucination (rare, 0.8%): Fabrication of facts or tool results contradictory to input.

Policy-read recall analysis demonstrates that agents access only 29.3% of ground-truth cited policies; high recall correlates with success, but low recall is linked to policy-compliance failures. The dominant second-level modes are criteria misapplication, skipped required steps, and misread policy criteria.

Comparative Benchmarking and Related Work

X-Bench’s task surface uniquely combines long-horizon tool invocation, explicit policy density, multi-role composition, and in-situ verification against simulator state—a confluence not found in existing healthcare or agentic benchmarks. It substantially outpaces HealthAdminBench (Bedi et al., 10 Apr 2026), MedQA (Jin et al., 2020), MedHELM (Bedi et al., 26 May 2025), MedAgentBench (Tang et al., 10 Mar 2025), and others in coverage of multilateral interaction and structured policy reasoning.

Theoretical and Practical Implications

The results present a strong negative signal for end-to-end healthcare workflow automation by current agentic AI. The inability to consistently resolve multi-role, policy-dense tasks, or to reliably navigate irreversible procedural states, implies that domain adaptation and operational context grounding are unresolved challenges. Failure patterns such as consent mining, criteria misapplication, and workflow completion gaps have direct clinical, financial, and regulatory consequences. The finding that completion alone is not a safety criterion is critical; agentic systems must not only finish workflows but must also adhere to protocol boundaries and preserve patient autonomy.

Practically, X-Bench provides an empirical barrier to unsupervised agent deployment in healthcare administration, demanding caution given the clinical risks. Theoretically, the benchmark reveals that state-of-the-art LLMs and agents are brittle when confronted with compound operational policies and irreversible coordination, highlighting future research axes in memory management, policy interpretation, and human-in-the-loop design.

Future Directions

Immediate next steps include the integration of multimodal reasoning (imaging, speech), expansion to additional long-tail healthcare workflows, and evaluation under alternate judge models to mitigate potential rubric bias. Advances in agent memory, policy retrieval, and adaptive dialog strategies are necessary for increasing pass rates and closing reliability gaps. The benchmark also motivates development of hybrid systems (combining LLMs with rule-based engines) and ongoing assessment of agentic safety in regulated environments.

Conclusion

X-Bench establishes a rigorous, high-fidelity stress test for agentic automation in healthcare operations, with robust empirical and diagnostic methodology. The 28% pass@1 ceiling underscores the non-trivial complexity of policy-dense, multi-role enterprise workflows. The dominant failure modes are agent reasoning and protocol navigation, not tool literacy or harness faults. The benchmark exposes operational gaps that, if unaddressed, translate directly to patient harm, regulatory violations, and financial risk. It serves as both a cautionary milestone and a roadmap for future agentic research in high-risk domains.