FHIR-AgentBench: EHR QA Benchmark

• FHIR-AgentBench is a benchmark that assesses LLM agents’ ability to navigate graph-structured FHIR EHR data for clinical question answering.
• It leverages 2,931 real-world clinical questions from de-identified MIMIC-IV-FHIR records, with explicit gold FHIR resource IDs to differentiate retrieval from reasoning failures.
• The benchmark supports diverse agent strategies—including retrieval tools, code interpretation, and reinforcement learning—to improve answer correctness over complex, nested resources.

FHIR-AgentBench is a benchmark for evaluating LLM agents on realistic, interoperable electronic health record question answering in the HL7 FHIR standard. It grounds 2,931 real-world clinical questions in de-identified MIMIC-IV-FHIR records, records the full set of ground-truth FHIR resource IDs needed to support each answer, and evaluates whether an agent can retrieve the right resources and reason over nested, graph-structured clinical data rather than over flat, site-specific schemas. The benchmark was introduced to measure an agent’s ability to navigate FHIR’s graph structure, retrieve the precise set of resources, and reason over nested fields to produce correct answers, and it is released with a dataset and evaluation suite to support reproducible research (Lee et al., 12 Sep 2025).

1. Scope, motivation, and benchmark identity

FHIR-AgentBench is situated at the intersection of clinical QA, tool-using LLM agents, and interoperability research. Its central premise is that modern hospitals increasingly adopt the HL7 FHIR standard to share and query EHR data as a graph of nested resources such as Patient, Observation, and Condition, so evaluating clinical AI on SQL-only or synthetic benchmarks underestimates the practical difficulty of resource-based retrieval and reasoning (Lee et al., 12 Sep 2025).

The benchmark is defined against several prior baselines. Existing QA benchmarks such as MIMICSQL and EHRSQL assume flat, site-specific schemas and SQL access; synthetic-data FHIR benchmarks use Synthea; and broader agent evaluations such as MedAgentBench emphasize interactive medical-record tasks rather than multi-step question answering over large FHIR graphs. By contrast, FHIR-AgentBench focuses specifically on authentic clinical questions over real, de-identified MIMIC-IV-FHIR records, including cases that require retrieval over hundreds to thousands of FHIR resources per patient (Lee et al., 12 Sep 2025).

A common misconception is that any FHIR-based clinical benchmark is interchangeable with any other. The surrounding literature shows otherwise. MedAgentBench is a broad evaluation suite with 300 patient-specific clinically-derived tasks from 10 categories in a FHIR-compliant interactive environment, designed around agent capabilities such as retrieval and resource creation; FHIR-AgentBench instead centers on interoperable EHR question answering and the retrieval-reasoning interface over existing resources (Jiang et al., 24 Jan 2025).

2. Dataset construction and representational structure

FHIR-AgentBench begins from 24,411 clinician-sourced, single-patient question–SQL pairs in EHRSQL-2024. From these, the authors retained 2,931 questions that reference only one patient, can be answered from FHIR resources, and demand non-trivial retrieval, with an average of 14.5 resources per question and a maximum of more than 2,000. Approximately 24% of questions correctly yield empty answers, which functions as an explicit realism check (Lee et al., 12 Sep 2025).

Each question is translated back into natural language compatible with raw MIMIC-IV-FHIR, restoring de-identification anchor years and original term casing. For every instance, the benchmark stores the set of gold FHIR resource IDs $R_i$ needed to support the answer $y_i$. This design is important because it separates two failure modes that are often conflated in end-to-end evaluation: retrieval failure and reasoning failure. An agent may retrieve the wrong resources and therefore reason correctly over the wrong evidence, or it may retrieve the correct resources and still fail during filtering, aggregation, or temporal interpretation.

The resource-type distribution further clarifies the benchmark’s emphasis. Observation accounts for 50% of questions, Condition 15%, Encounter 12%, Medication/Request 10%, Procedure 8%, and Patient/Empty/Other 5% (Lee et al., 12 Sep 2025). This distribution makes the benchmark clinically heterogeneous while still reflecting the operational centrality of observations, encounters, medications, and diagnoses in longitudinal EHR reasoning.

The RL reformulation of the benchmark further characterizes the underlying data model as a directed, typed, heterogeneous graph: nodes are JSON resources and edges are Reference fields such as Observation.subject → Patient and MedicationRequest.medication → Medication. In that formulation, the benchmark is instantiated over a real-world, de-identified FHIR store built from 100 demo patients in MIMIC-IV-FHIR, with a 424-question validation split (Knorr et al., 13 May 2026).

3. Task decomposition and evaluation methodology

FHIR-AgentBench decomposes each question $q_i$ into two subproblems. First, the agent must retrieve a set $\hat R_i$ of FHIR resource IDs via API or tool calls. Second, it must generate a final answer $\hat y_i$ based solely on the retrieved data (Lee et al., 12 Sep 2025).

The core retrieval metrics are defined as:

$P_i = \frac{|\hat R_i \cap R_i|}{|\hat R_i|}$

$R_i = \frac{|\hat R_i \cap R_i|}{|R_i|}$

An optional retrieval $F1$ score is also defined as:

$F1_i = 2 \cdot \frac{P_i \cdot R_i}{P_i + R_i}$

Final-answer quality is evaluated through answer correctness,

$AC_i = 1(\hat y_i = y_i)$

with special handling for empty-answer cases. If $y_i$0 and $y_i$1, then $y_i$2; if $y_i$3 and $y_i$4, then $y_i$5 and precision is undefined and dropped; if $y_i$6 and $y_i$7, then $y_i$8 and recall is dropped (Lee et al., 12 Sep 2025).

Answer evaluation is not restricted to exact string matching. Free-text answers are judged by an LLM evaluator, OpenAI o4-mini, which achieved 97% agreement with human labels on a 500-sample audit. In the RL extension, answer correctness is also treated as a binary reward $y_i$9 and judged by Qwen2.5-Instruct, reported at 96.2% agreement versus a physician. These two evaluation regimes point in the same direction: FHIR-AgentBench is not only a dataset of questions, but also a methodological framework for separating data access, evidence selection, and final-answer generation (Lee et al., 12 Sep 2025).

4. Agentic frameworks, tools, and baseline empirical results

The benchmark evaluates several agentic strategies. Its core tool components are a FHIR Query Generator, a Retriever, and a Code Interpreter. The FHIR Query Generator composes raw FHIR API queries such as Observation?patient=XXX&[code](https://www.emergentmind.com/topics/karpathy-agent-code)=9279-1&date=ge2133-12-31; the Retriever is a high-recall tool fetching all resources of a type for a patient, or specific IDs; and the Code Interpreter is a Python REPL allowing agents to programmatically filter, link, and compute over retrieved JSON (Lee et al., 12 Sep 2025).

Two interaction patterns are considered. Single-turn agents retrieve and answer in one pass, with variants based on the FHIR Query Generator, the Retriever, or the Retriever plus Code Interpreter. Multi-turn, ReAct-style agents iteratively issue retrieval calls and reasoning steps, again with or without Code Interpreter support. The comparison isolates three axes that matter in practice: direct FHIR API calls versus specialized tools, single-turn versus multi-turn interaction, and natural-language reasoning versus code generation.

Agentic approach	Precision / Recall	Answer correctness
Single-Turn + FHIR Query Generator	0.46 / 0.43	0.25
Single-Turn + Retriever	0.41 / 0.58	0.22
Single-Turn + Retriever + Code	0.41 / 0.58	0.33
Multi-Turn + Retriever	0.33 / 0.71	0.20
Multi-Turn + Retriever + Code	0.35 / 0.68	0.50

Several empirical regularities emerge from these baselines. Multi-turn interaction boosts recall from 58% to 71%. The Code Interpreter markedly improves final answer correctness from 33% to 50%, indicating that agents often struggle to parse FHIR correctly without procedural logic. Retrieval precision remains low, approximately 0.35–0.46, because the Retriever tool trades precision for high recall and floods the context with irrelevant resources. Among top architectures, different LLMs such as Gemini-2.5, Qwen3, and Llama-3 vary only $q_i$0 in answer accuracy, suggesting that the architecture of the agent and the difficulty of the benchmark dominate model choice (Lee et al., 12 Sep 2025).

The error analysis sharpens this picture. Retrieval failures include misidentifying the correct FHIR resource type, overly narrow or overly broad FHIR Query Generator calls, and single-turn agents being unable to chain multiple resource-type fetches. Reasoning failures include not following reference links such as MedicationRequest→Medication, filtering on human-readable display strings instead of canonical codes, mistaking encounter classes, and generating Python with syntax or logic errors. The benchmark therefore exposes an “interpretation gap” between obtaining JSON and deriving clinically correct answers from it (Lee et al., 12 Sep 2025).

5. Reinforcement-learning reformulation and post-training results

Subsequent work reformulates FHIR-AgentBench as a sequential decision-making problem over a queryable structured graph. In that environment, the agent observes a fixed question and context, maintains a workspace of retrieved resources, and acts through three executable tools: fhir_query(resource_type, patient_fhir_id), python(code), and finish(answer). Traversal constraints are enforced by the environment: malformed or semantically invalid FHIR calls return empty lists or errors, and Python execution is sandboxed so that runtime errors are exposed to the agent as observations rather than silently ignored (Knorr et al., 13 May 2026).

The agent architecture in this setting is a multi-turn CodeAct agent with a Qwen3 instruction-tuned backbone and a ReAct-style loop. Post-training uses SkyRL with a high-throughput rollout engine, an environment wrapper exposing the tools, and a trainer implementing Group Relative Policy Optimization. The reward is binary and execution-grounded:

$q_i$1

and the objective is

$q_i$2

with group-normalized advantages and a GRPO surrogate objective including clipping and a KL penalty (Knorr et al., 13 May 2026).

Empirically, this post-training substantially raises benchmark performance. In the reported multi-turn code-agent baselines with a 12-turn budget, o4-mini reaches 46.9%, Gemini-3-Flash 51.9%, GLM-5 58.7%, and zero-shot Qwen3 variants range from 11.1% for 4B to 39.3% for 32B. After 3 K steps, Qwen3-8B with GRPO and dynamic sampling reaches $q_i$3 accuracy and pass@5=79.7, improving on the 50% answer correctness baseline associated with the strongest prompt-based multi-turn Retriever + Code setting, and exceeding the closed-model baselines with a smaller and cheaper Qwen3-8B model (Knorr et al., 13 May 2026).

These results do not eliminate the benchmark’s difficulty. The same study reports that early gains appear on “Empty” questions, then on Encounter/Observation, and finally on MedicationRequest→Medication reference resolution, explicitly identifying reference traversal as the hardest regime. This suggests that FHIR-AgentBench is not merely a retrieval benchmark and not merely a reasoning benchmark; it is a benchmark of graph-constrained retrieval, code-level filtering, and schema-faithful traversal under clinical semantics.

6. Relations to adjacent benchmarks, misconceptions, and likely extensions

FHIR-AgentBench is often grouped with all FHIR-centered medical-agent benchmarks, but the neighboring literature makes the distinctions substantive rather than terminological. MedAgentBench is a realistic virtual EHR environment with 300 patient-specific clinically-derived tasks spanning ten categories such as patient information retrieval, lab result retrieval, documentation, referral ordering, medication ordering, and procedure ordering; it evaluates whether agents can retrieve and manipulate resources in a live FHIR environment. FHIR-AgentBench, by contrast, is centered on question answering over authentic FHIR records, with explicit gold resource IDs and retrieval metrics (Jiang et al., 24 Jan 2025).

A second misconception is that structured FHIR inputs automatically make reasoning more faithful or easier than unstructured clinical text. MedCase-Structured reports the opposite tendency: across all evaluated models and settings, FHIR input yields 5–23 percentage points lower diagnostic accuracy than plain text, and the paper attributes robust generation of structured bundles to multi-stage generation, terminology-grounded validation, repair loops, and diagnosis-hiding safeguards. This suggests that FHIR-AgentBench’s difficulty is not incidental; it is partly a consequence of deployment-aligned data representation itself (Muti et al., 28 May 2026).

A third source of confusion concerns administrative versus clinical scope. H-AdminSim is a multi-agent simulator for realistic hospital administrative workflows with FHIR integration, including patient intake, appointment scheduling, rescheduling, and cancellation, with quantitative rubrics such as task success rate, workflow completion rate, policy adherence, and interoperability compliance. It is not the same benchmark. However, its design notes explicitly identify components reusable in a “FHIR-AgentBench,” and its roadmap proposes extending such a benchmark toward billing and claims, insurance pre-authorization, referral management, lab and imaging orders, pharmacy orders, multiple FHIR versions, security and auditability through Provenance and AuditEvent, and customizable ImplementationGuide resources (Lee et al., 5 Feb 2026).

Within its present scope, the benchmark’s own recommendations are narrower and more immediate: multi-step planning frameworks, enhanced retrieval tools with fine-grained filtering by code systems, date ranges, and reference traversal, prompt-tuned or few-shot examples that explicitly outline FHIR’s JSON schema and common pitfalls, and incorporation of real-world constraints such as latency, cost, and failure-handling (Lee et al., 12 Sep 2025). Taken together, these proposals indicate two complementary trajectories. One is vertical: improving agents on the existing task of interoperable EHR question answering. The other is horizontal: broadening the benchmark family from QA over existing resources toward the wider space of administrative, operational, and conformance-sensitive workflows that contemporary FHIR deployments expose.