---
title: 'SYNUR: Synthetic Nursing Benchmark'
url: https://www.emergentmind.com/topics/synur
type: topic
---

# SYNUR: Synthetic Nursing Benchmark

SYNUR is an open-source **SYnthetic NURsing** benchmark and associated clinical information extraction task in which free-form nurse dictations, or more generally conversational nurse–patient transcripts, are converted into structured observations aligned with a predefined ontology. Introduced to address the scarcity and sensitivity of real nurse documentation data, SYNUR targets **schema-constrained normalization** into flowsheet-ready JSON rather than free-text summarization. In later work it becomes the basis for **MEDIQA-SYNUR**, where the same core problem is framed as automated extraction of ontology-grounded nursing observations from clinically dense, disfluent narrative transcripts [2507.05517][2605.15467].

## 1. Origin, motivation, and benchmark scope

SYNUR was introduced as a realistic but non-sensitive benchmark for a specific EHR workflow problem: nurses often document patient status into **flowsheets**, described as “large, multi-tab tables with hundreds of data rows,” while the upstream source may be free-form spoken remarks or dictation. The intended automation target is therefore not generic note generation, but direct support for **structured, flowsheet-compatible charting** [2507.05517].

Later shared-task papers retain that core formulation while emphasizing schema-constrained extraction from conversational transcripts. In that framing, the system must identify clinically salient findings and normalize them to a fixed schema of nursing assessment concepts rather than merely recognize mentions in text [2605.15467].

| Aspect | Reported value | Source |
|---|---:|---|
| Seed dictations | 80 fabricated seed dictations reviewed by practicing nurses | [2507.05517] |
| Final corpus | 223 dictations; 3000 observations | [2507.05517] |
| Ontology size | 193 concepts | [2507.05517] |
| Shared-task schema mix | 130 `SINGLE_SELECT`; 12 `MULTI_SELECT`; 20 `NUMERIC`; 31 `STRING` | [2605.15467] |
| Transcript statistics | Average 192 words; range 59–343 | [2605.15467] |
| Mean observations per instance | About 13.45 | [2605.15467] |

A central implication is that SYNUR occupies a space between benchmark design and deployment-oriented clinical NLP. It was created as a public research artifact, but its motivating use case is integration with existing nursing charting practice rather than replacement of those routines [2507.05517].

## 2. Dataset construction and ontology

The original SYNUR dataset was built from **80 proprietary, fabricated seed dictations** that were reviewed by practicing nurses for realism. From those seeds, the authors used a six-stage agentic pipeline alternating domain experts and GPT-4o-0806: **observation mining**, **concept consolidation**, **ontology expansion**, **scenario generation**, **dictation synthesis**, and **gold-standard labeling** [2507.05517].

The reported ontology-building trajectory is explicit. Observation mining yielded **547 unique observations**. These were distilled into **97 clinical concepts**, each assigned a type among **boolean, integer, string, single-choice, or multi-choice**. Ontology expansion then grew the concept inventory to **193 concepts**, with an experienced nurse validating and correcting the result. Final synthetic transcripts were verified by expert nurse annotators, who also produced the reference labels [2507.05517].

The type system is described slightly differently across papers. The original SYNUR construction paper uses the terms **boolean, integer, string, single-choice, or multi-choice**, while its appendix also uses **boolean, numeric, multiple_selection, single_selection, and string**. Later MEDIQA-SYNUR papers formalize the schema more rigidly as `SINGLE_SELECT`, `MULTI_SELECT`, `NUMERIC`, and `STRING`, and additionally note **15 explicit unit concepts**, so some measurements are represented as value–unit pairs across separate concept ids [2507.05517][2605.15467].

Shared-task analyses also quantify the schema’s long tail. Among the 193 concepts, **55 appear at most 5 times** and **23 appear only once or twice**. This sparsity is operationally important because systems must choose among many rare, semantically adjacent concepts under exact normalization constraints [2605.15467].

## 3. Task formulation and output representation

SYNUR is a **schema-constrained information extraction** problem. The input is a nurse transcript plus schema context; the output is a structured observation list aligned with ontology entries. The original extraction prompt is stated as: “Below is a TRANSCRIPT from a nurse dictation along with a flowsheet SCHEMA. Please extract the clinical observations from the TRANSCRIPT in strict, parsable JSON adhering to SCHEMA” [2507.05517].

In the MEDIQA-SYNUR formalization, for an input transcript \(x\), the model predicts
\[
\hat{O} = [o_1, \ldots, o_n],
\]
where each observation is normalized to one of \(M=193\) concepts \(S=\{c_1,\ldots,c_M\}\). Each concept \(c_m\) has an identifier \(id_m\), a canonical name \(N_m\), a value type
\[
T_m \in \{\texttt{SINGLE\_SELECT}, \texttt{MULTI\_SELECT}, \texttt{NUMERIC}, \texttt{STRING}\},
\]
and, for categorical concepts, an allowed value set \(V_m\) [2605.15467].

A representative expanded observation object has the form shown in the shared-task literature:

```json
[
  {
    "id": "67",
    "value_type": "SINGLE_SELECT",
    "name": "Dyspnea",
    "value": "Moderate"
  }
]
```

The papers describe two closely related generation interfaces. One asks the model to output the full structured object with fields such as `id`, `name`, `value_type`, and `value` [2603.26046]. Another constrains the LLM to emit only compact `(id, value)` pairs in JSON, after which `name` and `value_type` are deterministically recovered from the schema table rather than trusted to the generator [2605.15467].

Type constraints are strict. For `SINGLE_SELECT` and `MULTI_SELECT`, values must come from the allowed enumerations; for `MULTI_SELECT`, the value must be a JSON list even if only one item is present; for `NUMERIC`, the output must be numeric only, with no units or extra text; and for `STRING`, the value should be copied verbatim from the transcript, without paraphrase or abbreviation expansion [2605.15467].

A further distinction concerns processing granularity. One system paper formulates extraction **segment by segment**: full dictations are first split into contiguous clinical-fact segments, each segment is paired with retrieved schema items and few-shot exemplars, and the model outputs a JSON list or `[]` if no schema match exists [2603.26046]. Another system performs transcript-level retrieval and prompting, using exemplars from the training set and either the full schema or a pruned candidate schema [2605.15467].

## 4. Modeling paradigms on SYNUR

The initial SYNUR study proposes a three-stage nursing extraction pipeline: **segmentation**, **RAG**, and **extraction**. Long transcripts are split into medically coherent, continuous, non-overlapping segments; retrieval then reduces a large schema to the top \(N\) candidate rows for the current segment; and an LLM extracts and canonicalizes the relevant observations into strict JSON. Few-shot inference uses retrieved transcript–gold examples organized in a schema-specific database [2507.05517].

A later MEDIQA-SYNUR system paper develops a fully **training-free retrieval-augmented generation pipeline**. Its stages are: segment the nurse dictation, augment ontology items with layperson-friendly natural-language descriptions, retrieve relevant ontology schemas, retrieve similar segment–observation examples from training data, and generate observations for each segment. Ontology retrieval uses **BlueBERT** and **TF-IDF** to select the top **10** schemas, while exemplar retrieval uses **BlueBERT** and **BM25** to select the top **15** training examples. In development ablations, adding schema retrieval to few-shot examples improves F1 from **0.739** to **0.829**, and among the tested dense retrievers, **BlueBERT** yields the best development F1 at **0.833** [2603.26046].

Another MEDIQA-SYNUR paper presents a more modular RAG architecture. The **entire training split** is used as the exemplar corpus; each training case is represented both by the original transcript and by a **concept-name summary** constructed from its gold labels. Candidate exemplars are reranked with
\[
\text{score(candidate pool)} = w_t.s_t + w_c.s_c + w_l.s_l,
\]
where \(s_t\) is transcript-embedding cosine similarity, \(s_c\) is concept-summary cosine similarity, \(s_l\) is lexical Jaccard overlap, and the reported weights are
\[
(w_t, w_c, w_l) = (0.70, 0.25, 0.05).
\]
The paper varies \(k \in \{3,5,10,20,30,50\}\) and reports that the best GPT-5.2 configuration uses **top-\(k=30\)** exemplars [2605.15467].

That same work compares two schema-conditioning strategies. In **full-schema prompting**, the model receives the entire concept inventory with ids, names, value types, and categorical options. In **pruned candidate-schema prompting**, it receives only a per-instance subset \(C(x)\subset S\), built from lexical matching, semantic schema matching, retrieval-driven label expansion, common-pattern injection, and size budgeting. The study finds that the optimal strategy is **model-dependent**: Llama-4 benefits from the constrained decision space of the pruned schema, whereas GPT-5.2 performs best with the full schema context [2605.15467].

Both shared-task systems also emphasize deterministic control layers around generation. One enforces schema-constrained JSON at segment level but leaves several merge and deduplication details unspecified [2603.26046]. The other adds explicit deterministic postprocessing—JSON cleaning, id validation, categorical normalization, duplicate-id removal, numeric conversion when possible, and deterministic expansion of `name` and `value_type` from the schema—plus an optional **second-pass audit** that removes unsupported items, fixes enum or type issues, and adds only high-confidence omissions [2605.15467].

## 5. Reported performance and benchmark findings

In the original SYNUR dataset paper, closed-weight LLMs are evaluated in zero-shot and few-shot settings. The reported F1 scores on SYNUR are **88.4 / 90.6** for GPT-4o, **81.8 / 89.1** for GPT-4o-mini, **89.6 / 92.4** for GPT-4.1, and **88.4 / 90.8** for GPT-4.1-mini, where each pair denotes zero-shot and few-shot performance. The paper states that GPT-4o achieved the best performance in the synthetic experiments, but the table itself shows **GPT-4.1** with the highest zero-shot and few-shot values, so the textual summary and table are not fully aligned [2507.05517].

MEDIQA-SYNUR papers report lower but more tightly specified shared-task scores under fixed test conditions. A training-free RAG system using **GPT-5.1** for generation and **GPT-5-mini** for segmentation reports **Precision 0.786**, **Recall 0.807**, and **F1 0.796** on the MEDIQA-SYNUR test set [2603.26046]. A later modular RAG system with **GPT-5.2**, **full schema**, **RAG**, and **second-pass audit** reports **78.62 precision**, **82.18 recall**, and **80.36 F1**, the best score in that paper [2605.15467].

Several comparative findings recur across these studies. Few-shot prompting improves all tested models in the original SYNUR experiments, with the largest gain reported for GPT-4o-mini, from **81.8** to **89.1** F1 [2507.05517]. In MEDIQA-SYNUR, **RAG consistently improves performance** relative to prompt-only baselines for both Llama-4 and GPT-5.2. For GPT-5.2, full-schema prompting rises from **77.32** to **80.27** F1 with RAG, and the second-pass audit adds a further **0.09** F1 to reach **80.36** [2605.15467].

The most detailed error analysis comes from the 80.36-F1 system. On a **199-case** test set it reports **2236 true positives**, **608 false positives**, and **485 false negatives**. Among false positives, **498/608 = 81.9\%** are spurious concept predictions and **110/608 = 18.1\%** are wrong values for otherwise correct ids. Among false negatives, **382/485 = 78.8\%** are missed concepts and **103/485 = 21.2\%** are value mismatches. Identifier-format problems are a major source of error: leading-zero mismatches such as `03` versus `3` account for **147/485** false negatives and **159/608** false positives. The paper also highlights brittle exact-match behavior for string slots; for example, `Pain description` appears in gold **12** times and is often predicted, but none match exactly at string level [2605.15467].

A further qualitative pattern is overprediction of clinically plausible but sparsely annotated fields. The same analysis reports that `Secondary diagnosis` appears **3** times in the gold data but is predicted **61** times, while `Patient identification` appears **3** times in gold and is predicted **26** times. This indicates that part of the residual error is not purely clinical misunderstanding but disagreement between model extraction behavior and the dataset’s annotation regime [2605.15467].

## 6. Limitations, reproducibility, and open directions

The most fundamental limitation is that SYNUR is **synthetic**. The dataset was explicitly designed to mitigate privacy barriers, but the original paper also states that synthetic data may not fully capture the complexity of real clinical language. Additional caveats in that study include incomplete disclosure of some nursing-method details, evaluation limited to closed-weight models for the nursing task, and the possibility that use of the same model family in synthesis may confer an advantage in synthetic evaluations. The paper also notes that frequent updates to closed models reduce reproducibility [2507.05517].

Reproducibility gaps persist in later shared-task systems. The training-free RAG paper does not specify how BlueBERT similarity is computed, how dense and lexical retrieval scores are fused, what ANN or vector indexing infrastructure is used, or how final transcript-level outputs are deduplicated or reconciled across segments. It also mentions a “refined prompt” in some test configurations without fully defining the modification [2603.26046]. The modular RAG paper is more explicit about retrieval and postprocessing, but it still identifies annotation inconsistencies—especially leading-zero ids and temperature-unit encodings—as significant obstacles to stable evaluation [2605.15467].

The remaining technical difficulties are characteristic of **schema-constrained normalization under exact matching**. The output space is large, heterogeneous, and long-tailed; conversational evidence is diffuse; units may be encoded as separate concepts; and string-valued slots are highly brittle. Error correlations in one shared-task paper reinforce that difficulty profile: errors correlate more with **number of gold items (0.45)** and **number of predicted items (0.53)** than with **transcript length (0.24)**, suggesting that content density is a stronger challenge than sequence length alone [2605.15467].

The research directions proposed in the literature are correspondingly pragmatic. One paper suggests a **preliminary classifier** to filter out segments without extractable medical content before invoking retrieval and generation, primarily to reduce unnecessary computation [2603.26046]. Across the shared-task studies, other implied priorities are better value and unit standardization, improved handling of rare concepts, concept-specific calibration for frequently confused schema items, and more robust span-grounded treatment of exact-string fields [2603.26046][2605.15467].

Taken together, the literature positions SYNUR as a benchmark for a narrowly defined but operationally consequential clinical NLP problem: converting nurse-centered narrative documentation into ontology-compatible structured observations. Its importance lies less in raw extraction alone than in forcing systems to combine retrieval, schema adherence, value normalization, and workflow compatibility under clinically realistic constraints [2507.05517][2605.15467].

Source: https://www.emergentmind.com/topics/synur