ClinEnv: Inpatient Simulation Benchmark

• ClinEnv is a process-aware benchmark that simulates inpatient care by evaluating language models acting as attending physicians on real EHR admissions.
• It employs a Longitudinal Inpatient Simulation framework with multi-turn interactions, where models sequentially gather information from role-specialized agents before committing to clinical decisions.
• The system uses deterministic, ontology-based scoring to assess decision quality across diagnoses, medications, and procedures while also measuring process efficiency and cost.

ClinEnv is an interactive benchmark for evaluating LLMs as inpatient attending physicians over real MIMIC-IV admissions under a paradigm termed Longitudinal Inpatient Simulation (LIS) (Lu et al., 1 Jun 2026). It is designed around the premise that clinical practice is not the selection of an answer from enumerated options, but a process in which a physician gathers heterogeneous information incrementally and commits to sequential, irreversible decisions under uncertainty. In this formulation, the model must actively query specialized agents, commit stagewise diagnoses, medications, procedures, and plans, and is evaluated not only on decision quality through deterministic ontology-grounded matching, but also on how information is gathered, including efficiency and financial waste (Lu et al., 1 Jun 2026). In a broader research sense, the label “ClinEnv” has also been used to denote clinical environments for named entity recognition, trial approval estimation, benchmarked clinical NLP, dynamic diagnostic dialogue evaluation, and task-focused clinical planning systems, which situates the 2026 benchmark within a wider lineage of process-aware clinical evaluation frameworks (Zhou et al., 2021).

1. Definition and conceptual scope

ClinEnv is defined as “an interactive, process-aware, multi-stage benchmark” that evaluates a model acting as the inpatient attending over real EHR admissions, forcing active information gathering from four role-specialized agents before the model commits structured clinical actions that are verified deterministically against EHR-derived ground truth (Lu et al., 1 Jun 2026). Its central contribution is to make the “information-acquisition gap” directly measurable: a dissociation between what models ultimately decide and how they obtain the information required to decide.

The benchmark is motivated by limits of both static medical QA and prior interactive medical benchmarks. Static medical QA probes factual recall but not the realities of inpatient care, where information must be actively elicited and actions are sequential and irreversible. Prior interactive medical benchmarks are described as compromising on at least one of the following dimensions: real EHR trajectories, multi-stage sequential decisions, active information acquisition across heterogeneous sources, deterministic ontology-grounded scoring, and process metrics for efficiency and cost (Lu et al., 1 Jun 2026). ClinEnv is therefore positioned as a unified environment in which real longitudinal records, sequential action commitment, and process evaluation are combined.

A plausible implication is that ClinEnv shifts the evaluation target from isolated correctness to clinically structured decision-making under partial observability. This interpretation is supported by the paper’s formulation of the environment as a finite-horizon partially observable Markov decision process with stage boundaries, but the benchmark itself is described in outcome- and process-based terms rather than with explicit reward shaping (Lu et al., 1 Jun 2026).

2. Longitudinal Inpatient Simulation

The underlying paradigm of ClinEnv is Longitudinal Inpatient Simulation. LIS is defined over an admission-level EHR timeline segmented into ordered decision stages, with each stage requiring interactive information gathering before structured clinical commitments are made (Lu et al., 1 Jun 2026).

Stages are indexed by $t = 1, \ldots, T$, where $T$ is case-specific. The reported benchmark average is 2.65 stages per case, and 49.8% of cases are described as long-horizon with $T \ge 3$ (Lu et al., 1 Jun 2026). Each stage $S_k$ is represented as

$S_k = (C_k, G_k, A_k),$

where $C_k = [start_k, end_k]$ is the visible event window, $G_k$ is the structured ground-truth decision set for that stage, and $$A_k \subseteq \{\text{patient, nurse, lab, history}\}$$ is the set of activated information agents (Lu et al., 1 Jun 2026). Medication decisions in $G_k$ include an action type in $\{start, stop, switch, adjust\}$.

Interaction proceeds through tool calls $T$0 at stage $T$1, turn $T$2. The tools include information-gathering actions—ask_patient, ask_nurse, order_lab, get_history_summary, get_history_detail—and structured submission actions—submit_medication, submit_diagnosis, submit_procedure, submit_plan, finalize_decision (Lu et al., 1 Jun 2026). Observations accumulate as

$T$3

with exactly one tool call per turn and up to 60 turns per stage (Lu et al., 1 Jun 2026).

After decisions are committed, the environment advances to the next stage, injecting the previous stage’s realized ground truth into context while retaining dialogue trace but removing submit and finalize calls. The paper states that this is intended to keep longitudinal state clinically coherent and to prevent error compounding from earlier model submissions (Lu et al., 1 Jun 2026). This suggests that ClinEnv is designed to isolate stagewise reasoning quality rather than to simulate cumulative downstream harm from earlier mistakes.

3. Construction from real admissions

ClinEnv is built from MIMIC-IV v3.1 and MIMIC-IV-Note v2.2 (Lu et al., 1 Jun 2026). Structured sources include admissions, laboratory and microbiology tables, medication orders and administration records, ICU and chart events, fluid inputs and outputs, procedure events, and ICD-coded diagnoses and procedures. Notes include discharge summaries and radiology reports. Every retained row is turned into a provenance-bearing event with canonical event_time and source_table, and events are sorted into an admission timeline (Lu et al., 1 Jun 2026).

Case construction is automated and is organized into five phases. In Phase A, Claude-Sonnet-4.6 extracts attending-level decisions from discharge summaries, with fallback sections such as Chief Complaint, HPI, Brief Hospital Course, and Discharge Diagnosis stitched when needed. In Phase B, a forward sliding-window agent temporally anchors each decision $T$4 at $T$5 using mark_single and mark_range tools; decisions anchored only to pre-admission events are discarded. In Phase C, stages are constructed and validated using context windows $T$6, continuation medications already active in $T$7 are skipped, and a final diagnosis stage is appended to every case. In Phase D, a diagnostic groundability scan partitions the timeline into windows

$T$8

with stride $T$9, retaining only discharge ICD codes that are evidenced or managed in-record. In Phase E, raw decisions are classified as medication, procedure, diagnosis, or plan, and medication actions are assigned to $T \ge 3$0 with optional increase or decrease directionality (Lu et al., 1 Jun 2026).

The resulting benchmark comprises 3,509 admissions from 1,809 unique patients, 9,297 decision stages, and 26,043 ground-truth decisions (Lu et al., 1 Jun 2026). Decision types are distributed as 71.7% diagnoses (18,664), 21.4% medications (5,571), and 6.9% procedures (1,808). Medication action types are distributed as 79.1% start, 7.5% stop, 3.8% switch, and 9.5% adjust. The benchmark also contains 2,128 distinct ICD codes and 488 distinct drugs (Lu et al., 1 Jun 2026).

These design choices distinguish ClinEnv from simulated-vignette frameworks. Related work on dynamic evaluation, such as ClinDEF, also emphasizes multi-turn interaction and process-sensitive evaluation, but its cases are dynamically generated from a disease knowledge graph and medical encyclopedia rather than extracted from longitudinal inpatient EHR trajectories (Tang et al., 29 Dec 2025).

4. Multi-source agents and interaction protocol

ClinEnv exposes four role-specialized information agents, each powered by GPT-5.4-mini and restricted to a role-specific readview (Lu et al., 1 Jun 2026). The patient agent has access to demographics, chief complaint, HPI, and PMH and answers in lay language without objective measurements. The nurse agent exposes vitals, fluid balance, administered medications, and bedside observations. The lab agent exposes laboratory and microbiology results and resolves natural-language test queries against stage-available results via semantic matching. The history agent exposes prior discharge summaries for the same patient and is activated only when prior admissions exist (Lu et al., 1 Jun 2026).

Interaction is constrained deliberately. Exactly one tool call is allowed per turn, and only submit tools corresponding to ground-truth types in the current stage are exposed. The environment also reveals the exact count $T \ge 3$1 expected per decision type $T \ge 3$2, so the model submits exactly the required number of items of each type (Lu et al., 1 Jun 2026). The paper states that this eliminates format-guessing and keeps matching sets equal-sized, allowing Hungarian assignment to focus purely on match quality.

The benchmark further distinguishes effective from redundant querying. One well-formed query per activated agent is often sufficient to surface most relevant facts, whereas repeatedly ordering absent labs or re-asking the same agent without refining intent increases waste and suppresses efficiency without improving decisions (Lu et al., 1 Jun 2026). This operationalization is central to ClinEnv’s process-aware design, because the environment is not evaluating only whether facts are retrieved, but whether information acquisition is targeted.

A broader pattern appears across related clinical environments. ClinDEF restricts the doctor agent to one action per turn from $T \ge 3$3 and uses deterministic patient and examiner simulators tied to a latent case profile (Tang et al., 29 Dec 2025). ClinicalVis, although not an agent benchmark, similarly constrains available information and time so that task-focused clinical reasoning becomes observable in a controlled setting (Ghassemi et al., 2018). This suggests a shared methodological commitment across these systems: making clinical process measurable by limiting interaction pathways and externalizing sequential evidence gathering.

5. Deterministic scoring and process metrics

ClinEnv evaluates both outcome quality and process quality (Lu et al., 1 Jun 2026). Outcome scoring $T \ge 3$4 computes per-type matches via Hungarian assignment on equal-sized predicted and ground-truth sets, yielding continuous precision, recall, and stage-level $T \ge 3$5, where

$T \ge 3$6

Medication matching is action-gated and uses ATC partial credit. If action type mismatches, the score is zero regardless of drug similarity. When actions match, predicted and ground-truth names are resolved to ATC codes $T \ge 3$7 and $T \ge 3$8, and similarity is determined by longest common prefix length $T \ge 3$9: $S_k$0 if $S_k$1, $S_k$2 if $S_k$3, $S_k$4 if $S_k$5, $S_k$6 if $S_k$7, $S_k$8 if $S_k$9, and $S_k = (C_k, G_k, A_k),$0 otherwise. For adjust decisions with explicit directions, conflicting increase versus decrease directions reduce the score to $S_k = (C_k, G_k, A_k),$1 (Lu et al., 1 Jun 2026).

Diagnoses and procedures are scored through ICD hierarchical matching. Free-text submissions are mapped to ICD through a two-step retrieval pipeline: top-15 candidates by cosine similarity over the full ICD vocabulary, followed by LLM reranking. Matching then uses hierarchical F1 over ancestor sets: $S_k = (C_k, G_k, A_k),$2

$S_k = (C_k, G_k, A_k),$3

This gives partial credit for ontological proximity rather than requiring exact string identity (Lu et al., 1 Jun 2026).

Process scoring $S_k = (C_k, G_k, A_k),$4 evaluates information coverage, efficiency, laboratory waste, and estimated per-day drug acquisition cost. For each activated role $S_k = (C_k, G_k, A_k),$5, a judge enumerates key facts $S_k = (C_k, G_k, A_k),$6 from readview $S_k = (C_k, G_k, A_k),$7 and marks each fact as covered if it appears in the dialogue. Coverage per role is

$S_k = (C_k, G_k, A_k),$8

and stage-level coverage is

$S_k = (C_k, G_k, A_k),$9

Coverage efficiency discounts over-querying: $C_k = [start_k, end_k]$0 where $C_k = [start_k, end_k]$1 is the number of information-gathering queries (Lu et al., 1 Jun 2026).

Laboratory waste is measured by mapping ordered tests to the CMS Clinical Laboratory Fee Schedule; tests not evidenced in the ground-truth clinical picture are counted as excess. The waste ratio is

$C_k = [start_k, end_k]$2

Drug acquisition cost for newly initiated pharmacotherapy resolves submitted drugs through RxNorm, NDC, NADAC, and WHO DDD, with single-drug daily cost

$C_k = [start_k, end_k]$3

and stage total

$C_k = [start_k, end_k]$4

(Lu et al., 1 Jun 2026).

This deterministic, ontology-grounded evaluation contrasts with other clinical environments that rely on task-specific metrics or rubric-based judging. CLUE standardizes MAE, macro-F1, micro-F1, and aggregate scoring across benchmark tasks (Goodwin et al., 2022), while ClinDEF augments diagnostic accuracy with turns, positive and negative findings, Positive Hit Rate, and a rubric-based Diagnostic Quality Score assessed by an LLM-as-judge ensemble (Tang et al., 29 Dec 2025). ClinEnv’s distinctive feature is that both action matching and process cost accounting are tied directly to EHR-derived and ontology-linked artifacts (Lu et al., 1 Jun 2026).

6. Empirical results and the information-acquisition gap

Across seven evaluated models, the strongest reported stage macro-average decision F1 is approximately 0.306 for GPT-5.4, with diagnosis $C_k = [start_k, end_k]$5, medication $C_k = [start_k, end_k]$6, and procedure $C_k = [start_k, end_k]$7 (Lu et al., 1 Jun 2026). The paper emphasizes a sharp diagnosis-versus-management gap. Diagnosis-only stages score around $C_k = [start_k, end_k]$8 pooled, whereas management stages score approximately $C_k = [start_k, end_k]$9 F1; for GPT-5.4 specifically, diagnosis is $G_k$0 and management is $G_k$1 (Lu et al., 1 Jun 2026).

Medication is reported as the hardest decision type. The best medication score is only $G_k$2, achieved by Llama-3.1-70B. Although action-type selection can be correct 53–73% of the time, drug identity still mismatches frequently, and the ATC-based scorer captures class-level proximity without awarding full credit for an incorrect specific agent (Lu et al., 1 Jun 2026). This is a central empirical finding because it indicates that models can often infer the type of intervention required without selecting the precise pharmacologic entity.

Process metrics reveal variation not captured by outcome-only evaluation. GPT-5.4 leads in coverage at $G_k$3 with lab waste $G_k$4, while GPT-5.4-nano achieves competitive efficiency $G_k$5 with the fewest turns, 4.4, and the lowest waste, $G_k$6 (Lu et al., 1 Jun 2026). By contrast, Llama-3.1-70B averages 14.1 turns, $G_k$7 coverage, and $G_k$8 lab waste. Long-horizon cases with $G_k$9 are uniformly harder; for GPT-5.4, decision F1 drops from $$A_k \subseteq \{\text{patient, nurse, lab, history}\}$$0 overall to $$A_k \subseteq \{\text{patient, nurse, lab, history}\}$$1 in long-horizon cases, while turn counts increase (Lu et al., 1 Jun 2026).

The benchmark’s named “information-acquisition gap” is supported by several reported patterns. Decision F1 declines monotonically with stage index in management stages; GPT-5.4 falls from approximately $$A_k \subseteq \{\text{patient, nurse, lab, history}\}$$2 in the first management stage to below $$A_k \subseteq \{\text{patient, nurse, lab, history}\}$$3 by the fourth, even though coverage holds or rises and queries surge from approximately 4 to more than 50 per stage (Lu et al., 1 Jun 2026). Across models, stage-level coverage and decision F1 are described as essentially uncorrelated, with Pearson $$A_k \subseteq \{\text{patient, nurse, lab, history}\}$$4 overall. The paper interprets this as evidence that more information alone does not yield better decisions; targeted, efficient information seeking matters (Lu et al., 1 Jun 2026).

A plausible implication is that ClinEnv exposes a failure mode distinct from ordinary factual error: models can continue to gather information, even extensively, without converting that information into effective management action. This interpretation is consistent with the reported decoupling between process quality and outcome quality, but the benchmark itself reports the phenomenon descriptively rather than as a mechanistic causal analysis.

7. Relation to adjacent clinical environments, limitations, and extensions

ClinEnv belongs to a broader family of clinical environments that make process visible rather than evaluating only terminal outputs. CLUE provides fixed, patient-level stratified splits over MIMIC-III for disease staging, phenotyping, mortality prediction, and remaining length-of-stay, with a unified evaluation script and aggregate scoring to improve comparability and reproducibility in clinical NLP (Goodwin et al., 2022). ClinDEF constructs a dynamic diagnostic dialogue environment grounded in a disease knowledge graph and medical encyclopedia, with multi-turn interaction among doctor, patient, and examiner agents and evaluation spanning accuracy, efficiency, and rubric-based diagnostic quality (Tang et al., 29 Dec 2025). ClinicalVis evaluates task-focused HCP–EHR interactions through a visualization-based prototype that measures decision accuracy, confidence, time-to-decision, and NASA TLX under realistic ICU planning tasks (Ghassemi et al., 2018). ClinicLens provides an interactive visual analytics system for exploring and optimizing clinic testing capacities under uncertainty, using aggregated COVID-19 testing data, regression models, and iterative human-in-the-loop scenario editing (Dong et al., 2023). In a different operational sense, “ClinEnv” also appears in work on clinical named entity recognition, where domain-specific contextualized embeddings are injected into a BiLSTM-CRF sequence tagger to support robust entity extraction in clinical environments (Zhou et al., 2021), and in trial approval estimation, where LINT uses registry free text and ICD codes for clinical trial outcome prediction as decision support in a clinical environment (Gao et al., 2024).

Within this broader landscape, ClinEnv’s distinctive combination is real EHR longitudinal trajectories, multi-stage sequential action commitment, active querying across four specialized agents, deterministic ontology-based scoring, and explicit process and cost metrics (Lu et al., 1 Jun 2026). It does not rely on simulated vignette scripts for ground truth, and it does not reduce evaluation to atomic data-access tasks or single-endpoint diagnostic conversation.

The benchmark also has stated limitations. Scores measure concordance with what actually happened during the admission rather than clinical optimality, so clinically reasonable alternatives may be scored as misses. The benchmark is single-center and US-centric in its coding and pricing schemes, relying on ICD, ATC, CMS lab fees, NADAC, and WHO ATC/DDD. LLMs are used in benchmark construction for decision extraction, anchoring assistance, classification, and candidate reranking, although the structured ground truth used for evaluation is deterministically copied from EHRs (Lu et al., 1 Jun 2026).

Future directions identified in the paper include outpatient and cross-hospital expansions, non-US coding schemas, multilingual notes, stronger process-aware metrics such as redundancy rates and query diversity indexes, richer management evaluation through guideline-concordance and contraindication handling, and training-time reward shaping to improve outcome-process balance while reducing waste (Lu et al., 1 Jun 2026). This suggests that ClinEnv is intended not only as a benchmark for measurement, but also as a substrate for studying how clinical agents might learn to move toward the high-coverage, low-waste frontier without sacrificing management accuracy.