PhysicianBench: EHR Workflow Benchmark

• PhysicianBench is a benchmark that tests large language model agents on realistic, long-horizon clinical workflows within an EHR environment.
• It utilizes real de-identified records, 100 tasks across 21 subspecialties, and 670 checkpoints to systematically verify clinical data retrieval, reasoning, and action execution.
• The benchmark highlights challenges in clinical reasoning and documentation, demonstrating that even top models require improved fidelity in complex medical workflows.

PhysicianBench is a benchmark for evaluating LLM agents on realistic physician workflows executed inside an electronic health record system, with full, execution-grounded verification. It was introduced to address the gap between benchmarks centered on static knowledge recall, free-text conversation quality, or single atomic actions and the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench uses real de-identified EHR records, instantiates 100 long-horizon tasks adapted from real asynchronous e-consults between primary care and subspecialty physicians, spans 21 subspecialties, and evaluates completion through 670 structured checkpoints that directly inspect post-execution EHR state and generated documentation (Liu et al., 4 May 2026).

1. Scope and design objectives

PhysicianBench targets physician work as it is carried out inside a clinical information system. In this setting, success requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. The benchmark is therefore not organized around isolated question answering, action intent without execution, or one-step chart interactions; it is organized around end-to-end workflows grounded in a FHIR EHR environment (Liu et al., 4 May 2026).

The benchmark’s design emphasizes four properties that the paper uses to distinguish it from prior systems: an EHR environment, long-horizon tasks, verifiable execution, and clinician validation. This framing is central to the benchmark’s identity. A task is not considered solved because an agent states the right plan in prose; it must retrieve the relevant chart data, create the appropriate FHIR resources when needed, and write the requested deliverable to the specified file path. This makes PhysicianBench a benchmark of clinical agency rather than of clinical rhetoric.

The task set comprises 100 long-horizon tasks adapted from real consultation cases. Tasks span diagnosis interpretation, medication prescribing, treatment planning, and workup or risk stratification. The benchmark states that solving a task requires an average of 27 tool calls per task. This scale is large enough that failures propagate across stages: an incomplete chart review can induce a reasoning error, which can then produce an incorrect order or an incomplete management note. That chained dependency is one of the benchmark’s main methodological commitments.

2. Task construction and clinical coverage

Tasks are adapted from real electronic consultation cases in which a primary care physician submits a structured question in the EHR and a subspecialist reviews the full chart and responds with workup, treatment, and follow-up recommendations. The benchmark generalizes these cases into broader physician workflows while preserving the original clinical complexity and reasoning structure. Each task includes four artifacts: a natural-language instruction, a patient record, an evaluation test suite, and a reference solution summary and checklist (Liu et al., 4 May 2026).

Patient data come from the STAnford Research Repository. The released benchmark applies additional de-identification and perturbation: demographics are replaced with synthetic values via Faker, dates are shifted, numeric values are perturbed within plausible ranges while preserving clinical meaning, some codes are added or dropped, and free-text notes are de-identified and paraphrased via an internal SecureGPT system that removes PHI and rewrites notes while preserving clinical content. These steps are intended to preserve workflow realism while permitting public evaluation.

The task inventory covers 4 workflow types and 14 subtypes.

Workflow type	Subtypes
Diagnosis & Interpretation (13)	Result interpretation: 9; Imaging interpretation: 4
Medication Prescribing (26)	New therapy initiation: 9; Regimen adjustment: 9; Acute treatment: 8
Treatment Planning (27)	Medication optimization: 8; Empiric treatment: 7; Diagnostic planning: 5; Substance use management: 4; Risk–benefit analysis: 3
Workup & Risk Stratification (34)	Disease workup: 14; Abnormal lab evaluation: 11; Incidental finding followup: 5; Infection evaluation: 4

Subspecialty coverage includes endocrinology, hematology, family medicine, gastroenterology, infectious disease, cardiology, internal medicine, psychiatry, pulmonology, nephrology, neurology, and urology, among others, for a total of 21 subspecialties. The paper also groups these into 8 specialty groups for reporting. This breadth matters because the benchmark is not restricted to one narrow workflow family such as medication reconciliation or lab retrieval; it samples the consultation surface of outpatient specialist medicine more broadly.

A plausible implication is that PhysicianBench operationalizes specialist chart review as a compositional task: the initial instruction specifies clinical role, trigger event, patient identifier, current date, required steps, and expected deliverable, and the benchmark then checks whether the agent reconstructs the clinical workflow rather than merely echoing the instruction.

3. EHR environment and action space

PhysicianBench runs each task inside an isolated Docker container with a dedicated FHIR R4 server implemented with HAPI FHIR JPA and backed by an H2 database. The patient record is pre-loaded into that server, and each run begins from a clean state. This design gives the benchmark a production-style API surface while preserving determinism and preventing cross-task leakage (Liu et al., 4 May 2026).

Agents interact with the chart through 14 tools. Search tools expose structured FHIR GET access for Patient, Condition, Observation for labs, Observation for vitals, Observation for social history, MedicationRequest, Procedure, DocumentReference, and ServiceRequest. Create tools support MedicationRequest, ServiceRequest, Appointment, and Communication. A utility tool, write_file, is used to save deliverables such as management plans, letters, or reports to the required file path.

This action space is consequential. A correct answer can require not only identifying a diagnosis or interpreting a lab pattern but also creating a MedicationRequest with the correct drug, dose, and frequency, or creating a ServiceRequest with the appropriate code, priority, and reason. Only resources authored after the task’s current date and created during the agent run are counted, which isolates agent behavior from pre-existing chart history.

The environment models longitudinal chart state rather than a single encounter summary. The FHIR snapshot can include Conditions, Observations, MedicationRequests, Procedures, DocumentReferences, and existing ServiceRequests distributed over time. Agents must therefore reason temporally: they must determine what is current, what is historical, what changed across visits, and which values are relevant relative to the task’s current date.

4. Checkpoints, graders, and success metrics

Each task is formalized as a POMDP,

$$(\mathcal{S}, \mathcal{A}, T, \mathcal{O}, \Omega, I, C),$$

where the full EHR state and workspace files form the hidden state, the 14 tools form the action space, observations are FHIR JSON bundles or file/tool confirmations, the natural-language instruction provides task context, and $C = \{c_1, \ldots, c_K\}$ is the checkpoint set. Across the benchmark there are 670 structured checkpoints, or roughly 6–7 per task (Liu et al., 4 May 2026).

Checkpoints are organized around four workflow components. Data-retrieval checkpoints verify that the agent surfaced critical chart facts. Clinical-reasoning checkpoints verify calculations, interpretations, diagnoses, and treatment judgments. Action-execution checkpoints verify created FHIR resources. Documentation checkpoints verify the completeness and soundness of written plans and notes. This decomposition is important because task success is conjunctive: a task is fully completed only if all required checkpoints are passed.

The benchmark uses three grader types. Code graders are deterministic and directly inspect the FHIR server for expected resources and attributes. Hybrid graders compute a ground-truth value from the EHR programmatically and use an LLM only to extract the agent’s stated value from free text before deterministic comparison. LLM-judge graders are used where correctness cannot be fully encoded in logic, especially for multi-paragraph assessment and plan documents. Each LLM-judge checkpoint has a rubric with required elements, ground-truth references, and three-level grading: pass, partial, or fail.

Headline results are reported with task-level pass metrics rather than checkpoint averages. The paper reports pass@1, pass@3, and Pass$^3$. Pass@3 estimates the probability that at least one out of three independent runs succeeds. Pass$^3$ measures the probability that all 3 runs succeed. The distinction is substantive: pass@1 captures first-try competence, whereas Pass$^3$ captures consistency, which is a more demanding property for clinical deployment.

The agent framework itself is intentionally minimal. All models use the same loop, the same 100-tool-calling-turn limit, the same tool schemas, and provider-default temperature; reasoning_effort is set to high where supported. This isolates performance differences to the model rather than to bespoke orchestration.

5. Empirical results and failure structure

PhysicianBench is difficult for current frontier systems. The best-performing model, GPT-5.5, achieves 46.3 ± 1.2% pass@1, 57.4% pass@3, and 28.0% Pass$^3$, with 41.9 average tool calls per task. Claude Opus 4.6 reaches 31.7% pass@1 and 18.0% Pass$^3$; Claude Opus 4.7 reaches 29.3% pass@1 and 18.0% Pass$^3$; GPT-5.4 reaches 27.7% pass@1 and 13.0% Pass$^3$. The best open-source model, DeepSeek V4-Pro, reaches 18.7% pass@1 and 6.0% Pass$^3$ (Liu et al., 4 May 2026).

Performance varies by specialty group. GPT-5.5 is best in 7 of 8 groups, with pass@1 scores of 59.0% in Endocrinology, 55.6% in Cardiology, 57.1% in GI/Hep, 48.7% in Hem/Onc, 48.1% in Pulm/Other, 38.9% in Immunol/ID, 33.3% in Psych/Neuro, and 29.2% in Neph/Urol. It also varies by workflow type: for GPT-5.5, Diagnosis/Interpretation is 46.2%, Medication Prescribing 41.0%, Treatment Planning 40.7%, and Workup/Risk Stratification 54.9%. Treatment Planning is generally hardest for many models, whereas Workup/Risk Stratification is relatively easier for strong models.

Error analysis shows that failed checkpoints are dominated by clinical reasoning. Averaged over models, Clinical Reasoning accounts for 50.4% of failed checkpoints, Action Execution for 18.8%, Documentation for 18.4%, and Data Retrieval for 12.4%. For GPT-5.5 specifically, failed checkpoints are 42.5% Clinical Reasoning, 19.4% Action Execution, 28.1% Documentation, and 10.0% Data Retrieval. For DeepSeek V4-Pro, failed checkpoints are 58.3% Clinical Reasoning, 15.3% Action Execution, 14.0% Documentation, and 12.4% Data Retrieval. This distribution indicates that structured tool access does not eliminate the central bottleneck: chart-grounded clinical judgment remains the hardest component.

A representative failure trajectory in the appendix involves an adrenal insufficiency management task completed by Claude Opus 4.6. The agent correctly retrieves demographics, Conditions including E27.40 Adrenal insufficiency, unspecified, current hydrocortisone dosing, and multiple endocrine notes describing secondary adrenal insufficiency due to chronic ophthalmic steroids and opioids. It then fails in Clinical Reasoning by reclassifying the case as primary adrenal insufficiency and recommending fludrocortisone, fails in Action Execution by mentioning a Cardiology referral without creating the corresponding ServiceRequest, fails in Data Retrieval by not properly surfacing or interpreting the coded problem list, and fails in Documentation because the final management plan inherits the misclassification and unsafe recommendation. The example illustrates how a single upstream reasoning error can contaminate every downstream benchmark component.

The paper also reports a head-to-head contrast between GPT-5.5 and Claude Opus 4.6. GPT-5.5 systematically uses more tool calls on tasks where it outperforms Claude, suggesting deeper EHR exploration. The authors state that gains are not from radically superior medical knowledge; rather, GPT-5.5 is better at literal fidelity to chart data and at enumerating all rubric-required items.

6. Position within physician-centered benchmarking

PhysicianBench sits within a broader shift from exam-style medical evaluation toward physician-centered, workflow-grounded evaluation. MedAgentBench introduced a FHIR-compliant interactive environment with physician-written tasks for EMR work, but its tasks are described as single-step atomic operations rather than long-horizon, composite workflows (Jiang et al., 24 Jan 2025). HealthBench and HealthBench Professional evaluate open-ended clinical conversations through physician-authored rubrics, including real clinician chats in the professional setting, but they do not evaluate execution-grounded FHIR state mutation (Arora et al., 13 May 2025, Hicks et al., 30 Apr 2026). LiveClin evaluates full clinical pathways on contemporary, peer-reviewed case reports and compares models directly with Attending and Chief Physicians, but it is organized around sequential multimodal MCQs and Case Accuracy rather than EHR-native action execution (Wang et al., 18 Feb 2026). PhysAssistBench moves closer to integrated clinical assistance by combining implicit physician queries, patient interaction, and FHIR-based tools, yet it is explicitly framed as physician assistance across short sessions rather than end-to-end physician task completion inside an EHR workflow (Du et al., 17 Jun 2026). Doctorina MedBench and DR.BENCH, in turn, emphasize simulated doctor–patient dialogue and EHR-grounded diagnostic reasoning over clinical text, respectively, rather than execution-grounded EHR workflows (Kozlova et al., 26 Mar 2026, Gao et al., 2022).

Within that landscape, PhysicianBench is best understood as an execution-grounded benchmark for autonomous or semi-autonomous clinical agents. It complements conversational and reasoning benchmarks by asking whether an agent can actually do the work of a physician inside a production-style chart environment. A common misconception is to treat strong performance on medical QA or clinician-chat benchmarks as evidence of readiness for EHR autonomy. PhysicianBench directly tests that assumption and finds a substantial capability–demand gap.

The benchmark also has explicit limitations. The current task set focuses on EHR-mediated, outpatient e-consult–style physician tasks. It does not cover inpatient workflows such as discharge summaries or inpatient medication reconciliation, nursing workflows, triage, patient-facing tasks, real-time team interactions, multimodal inputs, imaging viewers, waveforms, or external knowledge bases. The underlying data come from Stanford Health Care, so demographics, disease prevalence, coding patterns, and local practice patterns may not generalize. De-identification perturbations may also introduce artifacts, although the benchmark reports physician validation that clinical meaning is preserved (Liu et al., 4 May 2026).

A plausible implication is that future extensions of PhysicianBench will need not only broader workflow coverage but also stronger attention to physician-centered evaluation methodology. Related work has shown that physician disagreement in rubric-based medical AI evaluation is largely structural and case-specific, with reducible uncertainty such as missing context and ambiguous phrasing materially increasing disagreement odds (Borgohain et al., 26 Feb 2026). Other physician-in-the-loop benchmark audits have found that benchmarks used as reward signals can contain substantial label noise, and that correcting those labels can materially change downstream reinforcement learning outcomes (Ye et al., 22 Dec 2025). These findings do not modify PhysicianBench’s execution-grounded design, but they suggest that benchmark maintenance, physician adjudication, and uncertainty reporting remain core issues even when evaluation is tied directly to EHR state.