SYNUR: Synthetic Nursing Benchmark

• SYNUR is an open-source benchmark for schema-constrained extraction from nurse dictations into structured, flowsheet-ready observations.
• It uses synthetic, nurse-validated data and predefined ontologies to simulate real EHR workflows while addressing privacy challenges.
• SYNUR underpins retrieval-augmented generation pipelines that enhance precision in mapping conversational transcripts to clinical schema.

SYNUR is an open-source SYnthetic NURsing benchmark and associated clinical information extraction task in which free-form nurse dictations, or more generally conversational nurse–patient transcripts, are converted into structured observations aligned with a predefined ontology. Introduced to address the scarcity and sensitivity of real nurse documentation data, SYNUR targets schema-constrained normalization into flowsheet-ready JSON rather than free-text summarization. In later work it becomes the basis for MEDIQA-SYNUR, where the same core problem is framed as automated extraction of ontology-grounded nursing observations from clinically dense, disfluent narrative transcripts (Corbeil et al., 7 Jul 2025, Karim et al., 14 May 2026).

1. Origin, motivation, and benchmark scope

SYNUR was introduced as a realistic but non-sensitive benchmark for a specific EHR workflow problem: nurses often document patient status into flowsheets, described as “large, multi-tab tables with hundreds of data rows,” while the upstream source may be free-form spoken remarks or dictation. The intended automation target is therefore not generic note generation, but direct support for structured, flowsheet-compatible charting (Corbeil et al., 7 Jul 2025).

Later shared-task papers retain that core formulation while emphasizing schema-constrained extraction from conversational transcripts. In that framing, the system must identify clinically salient findings and normalize them to a fixed schema of nursing assessment concepts rather than merely recognize mentions in text (Karim et al., 14 May 2026).

Aspect	Reported value	Source
Seed dictations	80 fabricated seed dictations reviewed by practicing nurses	(Corbeil et al., 7 Jul 2025)
Final corpus	223 dictations; 3000 observations	(Corbeil et al., 7 Jul 2025)
Ontology size	193 concepts	(Corbeil et al., 7 Jul 2025)
Shared-task schema mix	130 SINGLE_SELECT; 12 MULTI_SELECT; 20 NUMERIC; 31 [STRING](https://www.emergentmind.com/topics/string)	(Karim et al., 14 May 2026)
Transcript statistics	Average 192 words; range 59–343	(Karim et al., 14 May 2026)
Mean observations per instance	About 13.45	(Karim et al., 14 May 2026)

A central implication is that SYNUR occupies a space between benchmark design and deployment-oriented clinical NLP. It was created as a public research artifact, but its motivating use case is integration with existing nursing charting practice rather than replacement of those routines (Corbeil et al., 7 Jul 2025).

2. Dataset construction and ontology

The original SYNUR dataset was built from 80 proprietary, fabricated seed dictations that were reviewed by practicing nurses for realism. From those seeds, the authors used a six-stage agentic pipeline alternating domain experts and GPT-4o-0806: observation mining, concept consolidation, ontology expansion, scenario generation, dictation synthesis, and gold-standard labeling (Corbeil et al., 7 Jul 2025).

The reported ontology-building trajectory is explicit. Observation mining yielded 547 unique observations. These were distilled into 97 clinical concepts, each assigned a type among boolean, integer, string, single-choice, or multi-choice. Ontology expansion then grew the concept inventory to 193 concepts, with an experienced nurse validating and correcting the result. Final synthetic transcripts were verified by expert nurse annotators, who also produced the reference labels (Corbeil et al., 7 Jul 2025).

The type system is described slightly differently across papers. The original SYNUR construction paper uses the terms boolean, integer, string, single-choice, or multi-choice, while its appendix also uses boolean, numeric, multiple_selection, single_selection, and string. Later MEDIQA-SYNUR papers formalize the schema more rigidly as SINGLE_SELECT, MULTI_SELECT, NUMERIC, and STRING, and additionally note 15 explicit unit concepts, so some measurements are represented as value–unit pairs across separate concept ids (Corbeil et al., 7 Jul 2025, Karim et al., 14 May 2026).

Shared-task analyses also quantify the schema’s long tail. Among the 193 concepts, 55 appear at most 5 times and 23 appear only once or twice. This sparsity is operationally important because systems must choose among many rare, semantically adjacent concepts under exact normalization constraints (Karim et al., 14 May 2026).

3. Task formulation and output representation

SYNUR is a schema-constrained information extraction problem. The input is a nurse transcript plus schema context; the output is a structured observation list aligned with ontology entries. The original extraction prompt is stated as: “Below is a TRANSCRIPT from a nurse dictation along with a flowsheet SCHEMA. Please extract the clinical observations from the TRANSCRIPT in strict, parsable JSON adhering to SCHEMA” (Corbeil et al., 7 Jul 2025).

In the MEDIQA-SYNUR formalization, for an input transcript $x$, the model predicts

$\hat{O} = [o_1, \ldots, o_n],$

where each observation is normalized to one of $M=193$ concepts $S=\{c_1,\ldots,c_M\}$. Each concept $c_m$ has an identifier $id_m$, a canonical name $N_m$, a value type

$$T_m \in \{\texttt{SINGLE\_SELECT}, \texttt{MULTI\_SELECT}, \texttt{NUMERIC}, \texttt{STRING}\},$$

and, for categorical concepts, an allowed value set $V_m$ (Karim et al., 14 May 2026).

A representative expanded observation object has the form shown in the shared-task literature:

$\hat{O} = [o_1, \ldots, o_n],$8

The papers describe two closely related generation interfaces. One asks the model to output the full structured object with fields such as id, name, value_type, and value (Hwang et al., 27 Mar 2026). Another constrains the LLM to emit only compact (id, value) pairs in JSON, after which name and value_type are deterministically recovered from the schema table rather than trusted to the generator (Karim et al., 14 May 2026).

Type constraints are strict. For SINGLE_SELECT and MULTI_SELECT, values must come from the allowed enumerations; for MULTI_SELECT, the value must be a JSON list even if only one item is present; for NUMERIC, the output must be numeric only, with no units or extra text; and for STRING, the value should be copied verbatim from the transcript, without paraphrase or abbreviation expansion (Karim et al., 14 May 2026).

A further distinction concerns processing granularity. One system paper formulates extraction segment by segment: full dictations are first split into contiguous clinical-fact segments, each segment is paired with retrieved schema items and few-shot exemplars, and the model outputs a JSON list or [] if no schema match exists (Hwang et al., 27 Mar 2026). Another system performs transcript-level retrieval and prompting, using exemplars from the training set and either the full schema or a pruned candidate schema (Karim et al., 14 May 2026).

4. Modeling paradigms on SYNUR

The initial SYNUR study proposes a three-stage nursing extraction pipeline: segmentation, RAG, and extraction. Long transcripts are split into medically coherent, continuous, non-overlapping segments; retrieval then reduces a large schema to the top $N$ candidate rows for the current segment; and an LLM extracts and canonicalizes the relevant observations into strict JSON. Few-shot inference uses retrieved transcript–gold examples organized in a schema-specific database (Corbeil et al., 7 Jul 2025).

A later MEDIQA-SYNUR system paper develops a fully training-free retrieval-augmented generation pipeline. Its stages are: segment the nurse dictation, augment ontology items with layperson-friendly natural-language descriptions, retrieve relevant ontology schemas, retrieve similar segment–observation examples from training data, and generate observations for each segment. Ontology retrieval uses BlueBERT and TF-IDF to select the top 10 schemas, while exemplar retrieval uses BlueBERT and BM25 to select the top 15 training examples. In development ablations, adding schema retrieval to few-shot examples improves F1 from 0.739 to 0.829, and among the tested dense retrievers, BlueBERT yields the best development F1 at 0.833 (Hwang et al., 27 Mar 2026).

Another MEDIQA-SYNUR paper presents a more modular RAG architecture. The entire training split is used as the exemplar corpus; each training case is represented both by the original transcript and by a concept-name summary constructed from its gold labels. Candidate exemplars are reranked with

$\hat{O} = [o_1, \ldots, o_n],$0

where $\hat{O} = [o_1, \ldots, o_n],$1 is transcript-embedding cosine similarity, $\hat{O} = [o_1, \ldots, o_n],$2 is concept-summary cosine similarity, $\hat{O} = [o_1, \ldots, o_n],$3 is lexical Jaccard overlap, and the reported weights are

$\hat{O} = [o_1, \ldots, o_n],$4

The paper varies $\hat{O} = [o_1, \ldots, o_n],$5 and reports that the best GPT-5.2 configuration uses top-$\hat{O} = [o_1, \ldots, o_n],$6 exemplars (Karim et al., 14 May 2026).

That same work compares two schema-conditioning strategies. In full-schema prompting, the model receives the entire concept inventory with ids, names, value types, and categorical options. In pruned candidate-schema prompting, it receives only a per-instance subset $\hat{O} = [o_1, \ldots, o_n],$7, built from lexical matching, semantic schema matching, retrieval-driven label expansion, common-pattern injection, and size budgeting. The study finds that the optimal strategy is model-dependent: Llama-4 benefits from the constrained decision space of the pruned schema, whereas GPT-5.2 performs best with the full schema context (Karim et al., 14 May 2026).

Both shared-task systems also emphasize deterministic control layers around generation. One enforces schema-constrained JSON at segment level but leaves several merge and deduplication details unspecified (Hwang et al., 27 Mar 2026). The other adds explicit deterministic postprocessing—JSON cleaning, id validation, categorical normalization, duplicate-id removal, numeric conversion when possible, and deterministic expansion of name and value_type from the schema—plus an optional second-pass audit that removes unsupported items, fixes enum or type issues, and adds only high-confidence omissions (Karim et al., 14 May 2026).

5. Reported performance and benchmark findings

In the original SYNUR dataset paper, closed-weight LLMs are evaluated in zero-shot and few-shot settings. The reported F1 scores on SYNUR are 88.4 / 90.6 for GPT-4o, 81.8 / 89.1 for GPT-4o-mini, 89.6 / 92.4 for GPT-4.1, and 88.4 / 90.8 for GPT-4.1-mini, where each pair denotes zero-shot and few-shot performance. The paper states that GPT-4o achieved the best performance in the synthetic experiments, but the table itself shows GPT-4.1 with the highest zero-shot and few-shot values, so the textual summary and table are not fully aligned (Corbeil et al., 7 Jul 2025).

MEDIQA-SYNUR papers report lower but more tightly specified shared-task scores under fixed test conditions. A training-free RAG system using GPT-5.1 for generation and GPT-5-mini for segmentation reports Precision 0.786, Recall 0.807, and F1 0.796 on the MEDIQA-SYNUR test set (Hwang et al., 27 Mar 2026). A later modular RAG system with GPT-5.2, full schema, RAG, and second-pass audit reports 78.62 precision, 82.18 recall, and 80.36 F1, the best score in that paper (Karim et al., 14 May 2026).

Several comparative findings recur across these studies. Few-shot prompting improves all tested models in the original SYNUR experiments, with the largest gain reported for GPT-4o-mini, from 81.8 to 89.1 F1 (Corbeil et al., 7 Jul 2025). In MEDIQA-SYNUR, RAG consistently improves performance relative to prompt-only baselines for both Llama-4 and GPT-5.2. For GPT-5.2, full-schema prompting rises from 77.32 to 80.27 F1 with RAG, and the second-pass audit adds a further 0.09 F1 to reach 80.36 (Karim et al., 14 May 2026).

The most detailed error analysis comes from the 80.36-F1 system. On a 199-case test set it reports 2236 true positives, 608 false positives, and 485 false negatives. Among false positives, 498/608 = 81.9\% are spurious concept predictions and 110/608 = 18.1\% are wrong values for otherwise correct ids. Among false negatives, 382/485 = 78.8\% are missed concepts and 103/485 = 21.2\% are value mismatches. Identifier-format problems are a major source of error: leading-zero mismatches such as 03 versus 3 account for 147/485 false negatives and 159/608 false positives. The paper also highlights brittle exact-match behavior for string slots; for example, Pain description appears in gold 12 times and is often predicted, but none match exactly at string level (Karim et al., 14 May 2026).

A further qualitative pattern is overprediction of clinically plausible but sparsely annotated fields. The same analysis reports that Secondary diagnosis appears 3 times in the gold data but is predicted 61 times, while Patient identification appears 3 times in gold and is predicted 26 times. This indicates that part of the residual error is not purely clinical misunderstanding but disagreement between model extraction behavior and the dataset’s annotation regime (Karim et al., 14 May 2026).

6. Limitations, reproducibility, and open directions

The most fundamental limitation is that SYNUR is synthetic. The dataset was explicitly designed to mitigate privacy barriers, but the original paper also states that synthetic data may not fully capture the complexity of real clinical language. Additional caveats in that study include incomplete disclosure of some nursing-method details, evaluation limited to closed-weight models for the nursing task, and the possibility that use of the same model family in synthesis may confer an advantage in synthetic evaluations. The paper also notes that frequent updates to closed models reduce reproducibility (Corbeil et al., 7 Jul 2025).

Reproducibility gaps persist in later shared-task systems. The training-free RAG paper does not specify how BlueBERT similarity is computed, how dense and lexical retrieval scores are fused, what ANN or vector indexing infrastructure is used, or how final transcript-level outputs are deduplicated or reconciled across segments. It also mentions a “refined prompt” in some test configurations without fully defining the modification (Hwang et al., 27 Mar 2026). The modular RAG paper is more explicit about retrieval and postprocessing, but it still identifies annotation inconsistencies—especially leading-zero ids and temperature-unit encodings—as significant obstacles to stable evaluation (Karim et al., 14 May 2026).

The remaining technical difficulties are characteristic of schema-constrained normalization under exact matching. The output space is large, heterogeneous, and long-tailed; conversational evidence is diffuse; units may be encoded as separate concepts; and string-valued slots are highly brittle. Error correlations in one shared-task paper reinforce that difficulty profile: errors correlate more with number of gold items (0.45) and number of predicted items (0.53) than with transcript length (0.24), suggesting that content density is a stronger challenge than sequence length alone (Karim et al., 14 May 2026).

The research directions proposed in the literature are correspondingly pragmatic. One paper suggests a preliminary classifier to filter out segments without extractable medical content before invoking retrieval and generation, primarily to reduce unnecessary computation (Hwang et al., 27 Mar 2026). Across the shared-task studies, other implied priorities are better value and unit standardization, improved handling of rare concepts, concept-specific calibration for frequently confused schema items, and more robust span-grounded treatment of exact-string fields (Hwang et al., 27 Mar 2026, Karim et al., 14 May 2026).

Taken together, the literature positions SYNUR as a benchmark for a narrowly defined but operationally consequential clinical NLP problem: converting nurse-centered narrative documentation into ontology-compatible structured observations. Its importance lies less in raw extraction alone than in forcing systems to combine retrieval, schema adherence, value normalization, and workflow compatibility under clinically realistic constraints (Corbeil et al., 7 Jul 2025, Karim et al., 14 May 2026).