---
title: 'NDB: Noisy Diagnostic Benchmark'
url: https://www.emergentmind.com/topics/noisy-diagnostic-benchmark-ndb
type: topic
---

# NDB: Noisy Diagnostic Benchmark

Searching arXiv for the benchmark and closely related diagnostic/noise benchmark papers.
The **Noisy Diagnostic Benchmark (NDB)** is a synthetic benchmark of **patient self-description narratives** designed to evaluate how well language models infer diagnoses when the input resembles **noisy, vague, colloquial, and distractible patient language** rather than clean clinical prose [2509.11803]. It was introduced in *“From Fuzzy Speech to Medical Insight: Benchmarking LLMs on Noisy Patient Narratives”* as a response to a benchmark gap in medical NLP: most prior evaluations rely on **clean clinical notes**, **exam-style questions**, **curated case reports**, or other structured text, whereas real patient communication is often informal, incomplete, emotionally colored, typo-prone, ambiguous, and full of lay terminology [2509.11803]. In that sense, NDB is both a dataset and a robustness stress test for diagnostic classification under realistic linguistic degradation.

## 1. Definition and intended scope

NDB is presented as a **structured dataset of noisy, synthetic patient descriptions** intended to **stress-test and compare the diagnostic capabilities of large language models under realistic linguistic conditions** [2509.11803]. Its defining unit is a **single-patient self-description narrative** paired with a **ground-truth diagnosis**, with parallel versions of the same underlying clinical scenario rewritten at different noise levels. The benchmark is therefore not centered on clinician-authored notes, dialogue transcripts, or consumer health question-answer data; it is specifically organized around **patient-style noise** and around preserving diagnostic labels across multiple degraded renderings of the same symptom constellation [2509.11803].

The paper’s stated motivation is a mismatch between common evaluation conditions and deployment settings. Systems intended for **symptom checkers**, **virtual triage**, **AI-assisted consultation**, and **patient-facing diagnostic support** are often assessed on edited or structured clinical text, yet real patient descriptions are syntactically messy, colloquial, and semantically imprecise [2509.11803]. NDB operationalizes that mismatch as a controlled robustness problem: models are fine-tuned on a **clinically clean subset** and then evaluated on progressively noisier narratives, so the primary empirical question becomes how much diagnostic performance degrades when the same medical content is expressed with reduced communicative clarity [2509.11803].

A terminological distinction is important. The benchmark named **NDB** should not be conflated with **NDBench**, whose full name is **Netflix Data Benchmark**, a cloud benchmarking framework for persistence-layer systems and microservices rather than a medical or linguistic diagnostic benchmark [1807.10792].

## 2. Dataset construction and organization

The construction pipeline begins from a **symptom-based disease labelling dataset (SDPD)**. The paper states that synthetic patient self-description narratives are created by **sampling symptoms** from this source and then generating, for each symptom set, one **“no-noise”** description, one **“medium noise”** description, and one **“heavy noise”** description [2509.11803]. The clinical content and diagnosis labels are therefore inherited from a symptom–disease source rather than improvised independently during text generation.

The benchmark is reported to contain **3600 narrative triplet examples** [2509.11803]. The paper also notes an ambiguity in that phrase: it does not clarify whether this denotes **3600 triplets total** or **3600 texts arranged into triplets**. As a result, the exact number of individual narratives is not specified. What is clear is the triplet structure itself: each clinical scenario is represented by aligned clean, medium-noise, and heavy-noise variants with the same diagnostic label [2509.11803].

Clinical consistency is claimed through that anchoring procedure. The paper states that the dataset contains **clinically consistent scenarios annotated with ground-truth diagnoses**, and the mechanism explicitly described is that all three versions of a case correspond to the same underlying symptom set and diagnosis [2509.11803]. At the same time, the paper does **not** describe a formal constraint-checking procedure, physician review protocol, inter-annotator agreement study, contradiction detection method, or automatic validation rule for the generated narratives themselves. The diagnosis labels appear to be inherited from the source symptom-disease dataset and attached to generated narratives rather than re-annotated after generation [2509.11803].

The condition inventory is only partially exposed. The paper states that NDB spans a **range of diagnostic categories**, but it does **not** enumerate the disease set, the number of classes, specialty coverage, class frequencies, or imbalance statistics; the only diagnosis explicitly shown in the worked example is **psoriasis** [2509.11803]. This leaves the clinical coverage of the benchmark underreported relative to its narrative-generation design.

## 3. Noise model and communication regimes

NDB’s central design variable is communication noise. The paper defines three levels: **No Noise**, **Medium Noise**, and **Heavy Noise** [2509.11803]. Two of these are operationalized by explicit word-count ranges:

| Noise level | Reported characterization |
|---|---|
| No Noise | clean patient description |
| Medium Noise | **80–220 words** |
| Heavy Noise | **150–390 words** |

The benchmark also states that it systematically introduces **lexical distortion**, **syntactic distortion**, and **semantic distortion** [2509.11803]. More broadly, the intended perturbation family includes **linguistic noise**, **fuzzy language**, **layperson terminology**, **informal phrasing**, and **partially incorrect language**. The emphasis is not on arbitrary corruption, but on realistic degradation of patient self-expression.

The worked triplet example described in the paper clarifies how these regimes differ in practice. The **clean** version is compact, direct, and symptom-focused. The **medium-noise** version adds filler and interjections, colloquial morphology, metaphorical phrasing, autobiographical digressions, uncertain chronology, and emotional concern. The **heavy-noise** version adds longer digressions, self-interruptions, false starts, unrelated personal details, speculative explanations, repeated topic drift, and further symptom elaboration that may be distractive [2509.11803]. This indicates that the benchmark’s perturbations extend beyond spelling or local lexical corruption to include discourse-level noise and pragmatic distraction.

The paper does **not** define a formal scalar noise function, a symbolic corruption operator, or a detailed generation template [2509.11803]. It also does **not** identify which language model generated the narratives. This suggests that NDB’s control surface is qualitative and example-driven rather than mathematically formalized. A plausible implication is that the benchmark’s main strength lies in matched diagnosis-preserving rewrites rather than in a formal noise calculus.

## 4. Evaluation protocol and empirical findings

The reported task is a **supervised classification setup** in which the input is a patient self-description narrative and the output is a diagnosis class label [2509.11803]. The evaluation protocol is asymmetric by design: models are **fine-tuned on a clinically clean subset** and then tested on clean, medium-noise, and high-noise narratives to quantify robustness under distribution shift from clean to noisy patient language [2509.11803].

Three models are evaluated: **BERT**, **ClinicalBERT**, and **Flan-T5** [2509.11803]. The primary reported metric is **accuracy**; the paper does **not** report macro/micro \(F_1\), precision, recall, top-\(k\) accuracy, calibration, per-class performance, or significance tests. It also does **not** provide exact pretrained checkpoints, model sizes, optimizer settings, learning rate, epochs, batch size, seeds, confidence intervals, or detailed split methodology [2509.11803]. The benchmark setup is therefore conceptually clear but experimentally sparse.

The main quantitative results are reported as follows [2509.11803]:

| Model | Clean | Medium-noise | High-noise |
|---|---:|---:|---:|
| BERT | **98.3%** | **86.7%** | **79.2%** |
| ClinicalBERT | **97.9%** | **83.8%** | **86.2%** |
| Flan-T5 | **97.1%** | **92.5%** | **87.1%** |

Two results dominate interpretation. First, all three models achieve very high clean-text accuracy, roughly **97–98%**, which indicates that the diagnostic classification problem is learnable under idealized linguistic conditions [2509.11803]. Second, performance degrades once patient-style noise is introduced, showing that clean-text success does not guarantee robustness to realistic self-description narratives. Among the reported models, **Flan-T5** is described as the most robust architecture tested, retaining **92.5%** accuracy at medium noise and **87.1%** at high noise [2509.11803].

The behavior of **ClinicalBERT** is notable because the reported table is non-monotonic: its **high-noise accuracy** is higher than its **medium-noise accuracy**. The paper records the general degradation trend but does **not** explain this non-monotonicity [2509.11803]. The paper also contains **no central mathematical formalism** for NDB: there are no formulas for the dataset notation, the noise injection function, the training objective, the evaluation metric, or a standalone robustness score [2509.11803].

## 5. Relation to other diagnostic-benchmark paradigms

NDB belongs to a broader class of benchmarks that use controlled perturbations or structured task design to expose failure modes that ordinary accuracy on clean inputs can conceal. This relationship is clearest when it is contrasted with several adjacent benchmark programs.

**CLUTRR** is a diagnostic benchmark for inductive reasoning from text in which robustness is probed by adding **curated noise facts** of distinct structural types—**irrelevant facts**, **supporting facts**, and **disconnected facts**—to short kinship stories [1908.06177]. Unlike NDB, which focuses on patient-style linguistic degradation in self-descriptions, CLUTRR defines noise relative to a known latent proof path in a grounded kinship graph. The comparison suggests two complementary benchmark philosophies: one centered on **naturalistic communication degradation**, the other on **structurally typed distractor insertion**.

**OpenExempt** is a diagnostic benchmark for legal reasoning built from expert-crafted symbolic representations of bankruptcy statutes and generated natural-language tasks, including suites for **Distractor Robustness**, **Sycophancy Robustness**, and **Obfuscation Robustness** [2601.13183]. Its noise consists of **legally immaterial** irrelevant facts and opinions. Relative to NDB, OpenExempt makes explicit the idea that the same obfuscating statements may have little effect on simple tasks but cause substantial degradation on longer reasoning chains. This suggests a more general diagnostic principle: robustness to noisy text may depend on reasoning depth, not only on local susceptibility to distraction.

**SketchJudge** evaluates multimodal large language models as graders of **hand-drawn STEM diagrams** and explicitly frames its challenge as diagnosis under **noisy visual conditions**, including uneven strokes, construction traces, stylistic diversity, and structural ambiguity [2601.06944]. NDB and SketchJudge therefore instantiate two different modalities of noisy diagnostic evaluation: one centered on **patient-generated language**, the other on **student-generated symbolic sketches**.

**DDX-TRACE** is relevant in a different way. It is not a noisy diagnostic benchmark in the usual sense of explicit corruption or perturbation; instead, it evaluates **sequential diagnostic trajectories under hidden evidence and partial observability** in multimodal neuroradiology [2605.23629]. It exposes failure modes such as unsupported correct guesses, premature closure, and poor uncertainty updating. Relative to NDB, it broadens the notion of diagnostic robustness from noisy input language to incomplete evidence acquisition.

**LNMBench** addresses yet another axis of diagnostic noise: **noisy labels** in medical image classification, with evaluation across **7 datasets**, **6 imaging modalities**, and **3 noise patterns**, including **real-world label noise** [2512.09315]. Whereas NDB perturbs the textual expression of symptoms while holding the diagnosis fixed, LNMBench studies what happens when the label itself is unreliable. Together they indicate that “noisy diagnosis” can refer to noise in at least three distinct places: the **input narrative**, the **reasoning environment**, and the **supervisory signal**.

## 6. Limitations, interpretation, and practical status

NDB’s principal strength is its matched-triplet design: it makes the clean-to-noisy shift measurable while preserving the underlying diagnosis [2509.11803]. The benchmark also avoids direct privacy concerns associated with real patient narratives by relying on synthetic generation, and it is publicly released through the **PatientSignal** repository [2509.11803]. These properties make it usable as a reproducible stress test for patient-facing medical NLP.

Its limitations are equally clear. The benchmark is **synthetic**, and the paper provides no human realism ratings, clinician plausibility assessments, lexical comparison to real patient corpora, or external validation against authentic patient-authored text [2509.11803]. The disease inventory, class distribution, and specialty coverage are not enumerated. The split design is underspecified: the paper implies clean-train to noisy-test evaluation, but does **not** provide exact train/dev/test sizes, split ratios, whether parallel versions of the same case may cross splits, or the split-generation methodology [2509.11803]. Training details and reproducibility controls are likewise sparse.

The reported experiments are also narrow. Only **three fine-tuned models** are evaluated, only **accuracy** is reported, and the paper does **not** include error analysis, confusion matrices, diagnosis-wise breakdowns, ablations over lexical versus syntactic versus semantic distortion, or robustness scores beyond raw accuracy under each condition [2509.11803]. As a result, NDB is best interpreted as an initial benchmark for measuring degradation under noisy patient narratives rather than as a complete evaluation framework for clinical diagnostic robustness.

In the broader benchmark landscape, this suggests that NDB is most informative when read as an instance of a larger design pattern: a **diagnostic benchmark** whose purpose is to reveal what clean-input leaderboards conceal. Its specific contribution is to show that, in medical NLP, patient-style noise is not an incidental nuisance but a first-order evaluation variable.

Source: https://www.emergentmind.com/topics/noisy-diagnostic-benchmark-ndb