NDB: Noisy Diagnostic Benchmark
- Noisy Diagnostic Benchmark (NDB) is a synthetic dataset of patient self-description narratives paired with true diagnoses, designed to assess model robustness under varying noise conditions.
- It introduces controlled levels of noise—clean, medium, and heavy—by applying lexical, syntactic, and semantic distortions that mimic real, informal patient communication.
- Experimental results reveal high accuracy on clean texts with significant degradation under noisy conditions, highlighting the importance of robustness in clinical diagnostic applications.
Searching arXiv for the benchmark and closely related diagnostic/noise benchmark papers. The Noisy Diagnostic Benchmark (NDB) is a synthetic benchmark of patient self-description narratives designed to evaluate how well LLMs infer diagnoses when the input resembles noisy, vague, colloquial, and distractible patient language rather than clean clinical prose (Mama et al., 15 Sep 2025). It was introduced in “From Fuzzy Speech to Medical Insight: Benchmarking LLMs on Noisy Patient Narratives” as a response to a benchmark gap in medical NLP: most prior evaluations rely on clean clinical notes, exam-style questions, curated case reports, or other structured text, whereas real patient communication is often informal, incomplete, emotionally colored, typo-prone, ambiguous, and full of lay terminology (Mama et al., 15 Sep 2025). In that sense, NDB is both a dataset and a robustness stress test for diagnostic classification under realistic linguistic degradation.
1. Definition and intended scope
NDB is presented as a structured dataset of noisy, synthetic patient descriptions intended to stress-test and compare the diagnostic capabilities of LLMs under realistic linguistic conditions (Mama et al., 15 Sep 2025). Its defining unit is a single-patient self-description narrative paired with a ground-truth diagnosis, with parallel versions of the same underlying clinical scenario rewritten at different noise levels. The benchmark is therefore not centered on clinician-authored notes, dialogue transcripts, or consumer health question-answer data; it is specifically organized around patient-style noise and around preserving diagnostic labels across multiple degraded renderings of the same symptom constellation (Mama et al., 15 Sep 2025).
The paper’s stated motivation is a mismatch between common evaluation conditions and deployment settings. Systems intended for symptom checkers, virtual triage, AI-assisted consultation, and patient-facing diagnostic support are often assessed on edited or structured clinical text, yet real patient descriptions are syntactically messy, colloquial, and semantically imprecise (Mama et al., 15 Sep 2025). NDB operationalizes that mismatch as a controlled robustness problem: models are fine-tuned on a clinically clean subset and then evaluated on progressively noisier narratives, so the primary empirical question becomes how much diagnostic performance degrades when the same medical content is expressed with reduced communicative clarity (Mama et al., 15 Sep 2025).
A terminological distinction is important. The benchmark named NDB should not be conflated with NDBench, whose full name is Netflix Data Benchmark, a cloud benchmarking framework for persistence-layer systems and microservices rather than a medical or linguistic diagnostic benchmark (Papapanagiotou et al., 2018).
2. Dataset construction and organization
The construction pipeline begins from a symptom-based disease labelling dataset (SDPD). The paper states that synthetic patient self-description narratives are created by sampling symptoms from this source and then generating, for each symptom set, one “no-noise” description, one “medium noise” description, and one “heavy noise” description (Mama et al., 15 Sep 2025). The clinical content and diagnosis labels are therefore inherited from a symptom–disease source rather than improvised independently during text generation.
The benchmark is reported to contain 3600 narrative triplet examples (Mama et al., 15 Sep 2025). The paper also notes an ambiguity in that phrase: it does not clarify whether this denotes 3600 triplets total or 3600 texts arranged into triplets. As a result, the exact number of individual narratives is not specified. What is clear is the triplet structure itself: each clinical scenario is represented by aligned clean, medium-noise, and heavy-noise variants with the same diagnostic label (Mama et al., 15 Sep 2025).
Clinical consistency is claimed through that anchoring procedure. The paper states that the dataset contains clinically consistent scenarios annotated with ground-truth diagnoses, and the mechanism explicitly described is that all three versions of a case correspond to the same underlying symptom set and diagnosis (Mama et al., 15 Sep 2025). At the same time, the paper does not describe a formal constraint-checking procedure, physician review protocol, inter-annotator agreement study, contradiction detection method, or automatic validation rule for the generated narratives themselves. The diagnosis labels appear to be inherited from the source symptom-disease dataset and attached to generated narratives rather than re-annotated after generation (Mama et al., 15 Sep 2025).
The condition inventory is only partially exposed. The paper states that NDB spans a range of diagnostic categories, but it does not enumerate the disease set, the number of classes, specialty coverage, class frequencies, or imbalance statistics; the only diagnosis explicitly shown in the worked example is psoriasis (Mama et al., 15 Sep 2025). This leaves the clinical coverage of the benchmark underreported relative to its narrative-generation design.
3. Noise model and communication regimes
NDB’s central design variable is communication noise. The paper defines three levels: No Noise, Medium Noise, and Heavy Noise (Mama et al., 15 Sep 2025). Two of these are operationalized by explicit word-count ranges:
| Noise level | Reported characterization |
|---|---|
| No Noise | clean patient description |
| Medium Noise | 80–220 words |
| Heavy Noise | 150–390 words |
The benchmark also states that it systematically introduces lexical distortion, syntactic distortion, and semantic distortion (Mama et al., 15 Sep 2025). More broadly, the intended perturbation family includes linguistic noise, fuzzy language, layperson terminology, informal phrasing, and partially incorrect language. The emphasis is not on arbitrary corruption, but on realistic degradation of patient self-expression.
The worked triplet example described in the paper clarifies how these regimes differ in practice. The clean version is compact, direct, and symptom-focused. The medium-noise version adds filler and interjections, colloquial morphology, metaphorical phrasing, autobiographical digressions, uncertain chronology, and emotional concern. The heavy-noise version adds longer digressions, self-interruptions, false starts, unrelated personal details, speculative explanations, repeated topic drift, and further symptom elaboration that may be distractive (Mama et al., 15 Sep 2025). This indicates that the benchmark’s perturbations extend beyond spelling or local lexical corruption to include discourse-level noise and pragmatic distraction.
The paper does not define a formal scalar noise function, a symbolic corruption operator, or a detailed generation template (Mama et al., 15 Sep 2025). It also does not identify which LLM generated the narratives. This suggests that NDB’s control surface is qualitative and example-driven rather than mathematically formalized. A plausible implication is that the benchmark’s main strength lies in matched diagnosis-preserving rewrites rather than in a formal noise calculus.
4. Evaluation protocol and empirical findings
The reported task is a supervised classification setup in which the input is a patient self-description narrative and the output is a diagnosis class label (Mama et al., 15 Sep 2025). The evaluation protocol is asymmetric by design: models are fine-tuned on a clinically clean subset and then tested on clean, medium-noise, and high-noise narratives to quantify robustness under distribution shift from clean to noisy patient language (Mama et al., 15 Sep 2025).
Three models are evaluated: BERT, ClinicalBERT, and Flan-T5 (Mama et al., 15 Sep 2025). The primary reported metric is accuracy; the paper does not report macro/micro , precision, recall, top- accuracy, calibration, per-class performance, or significance tests. It also does not provide exact pretrained checkpoints, model sizes, optimizer settings, learning rate, epochs, batch size, seeds, confidence intervals, or detailed split methodology (Mama et al., 15 Sep 2025). The benchmark setup is therefore conceptually clear but experimentally sparse.
The main quantitative results are reported as follows (Mama et al., 15 Sep 2025):
| Model | Clean | Medium-noise | High-noise |
|---|---|---|---|
| BERT | 98.3% | 86.7% | 79.2% |
| ClinicalBERT | 97.9% | 83.8% | 86.2% |
| Flan-T5 | 97.1% | 92.5% | 87.1% |
Two results dominate interpretation. First, all three models achieve very high clean-text accuracy, roughly 97–98%, which indicates that the diagnostic classification problem is learnable under idealized linguistic conditions (Mama et al., 15 Sep 2025). Second, performance degrades once patient-style noise is introduced, showing that clean-text success does not guarantee robustness to realistic self-description narratives. Among the reported models, Flan-T5 is described as the most robust architecture tested, retaining 92.5% accuracy at medium noise and 87.1% at high noise (Mama et al., 15 Sep 2025).
The behavior of ClinicalBERT is notable because the reported table is non-monotonic: its high-noise accuracy is higher than its medium-noise accuracy. The paper records the general degradation trend but does not explain this non-monotonicity (Mama et al., 15 Sep 2025). The paper also contains no central mathematical formalism for NDB: there are no formulas for the dataset notation, the noise injection function, the training objective, the evaluation metric, or a standalone robustness score (Mama et al., 15 Sep 2025).
5. Relation to other diagnostic-benchmark paradigms
NDB belongs to a broader class of benchmarks that use controlled perturbations or structured task design to expose failure modes that ordinary accuracy on clean inputs can conceal. This relationship is clearest when it is contrasted with several adjacent benchmark programs.
CLUTRR is a diagnostic benchmark for inductive reasoning from text in which robustness is probed by adding curated noise facts of distinct structural types—irrelevant facts, supporting facts, and disconnected facts—to short kinship stories (Sinha et al., 2019). Unlike NDB, which focuses on patient-style linguistic degradation in self-descriptions, CLUTRR defines noise relative to a known latent proof path in a grounded kinship graph. The comparison suggests two complementary benchmark philosophies: one centered on naturalistic communication degradation, the other on structurally typed distractor insertion.
OpenExempt is a diagnostic benchmark for legal reasoning built from expert-crafted symbolic representations of bankruptcy statutes and generated natural-language tasks, including suites for Distractor Robustness, Sycophancy Robustness, and Obfuscation Robustness (Servantez et al., 19 Jan 2026). Its noise consists of legally immaterial irrelevant facts and opinions. Relative to NDB, OpenExempt makes explicit the idea that the same obfuscating statements may have little effect on simple tasks but cause substantial degradation on longer reasoning chains. This suggests a more general diagnostic principle: robustness to noisy text may depend on reasoning depth, not only on local susceptibility to distraction.
SketchJudge evaluates multimodal LLMs as graders of hand-drawn STEM diagrams and explicitly frames its challenge as diagnosis under noisy visual conditions, including uneven strokes, construction traces, stylistic diversity, and structural ambiguity (Su et al., 11 Jan 2026). NDB and SketchJudge therefore instantiate two different modalities of noisy diagnostic evaluation: one centered on patient-generated language, the other on student-generated symbolic sketches.
DDX-TRACE is relevant in a different way. It is not a noisy diagnostic benchmark in the usual sense of explicit corruption or perturbation; instead, it evaluates sequential diagnostic trajectories under hidden evidence and partial observability in multimodal neuroradiology (Pan et al., 22 May 2026). It exposes failure modes such as unsupported correct guesses, premature closure, and poor uncertainty updating. Relative to NDB, it broadens the notion of diagnostic robustness from noisy input language to incomplete evidence acquisition.
LNMBench addresses yet another axis of diagnostic noise: noisy labels in medical image classification, with evaluation across 7 datasets, 6 imaging modalities, and 3 noise patterns, including real-world label noise (Ma et al., 10 Dec 2025). Whereas NDB perturbs the textual expression of symptoms while holding the diagnosis fixed, LNMBench studies what happens when the label itself is unreliable. Together they indicate that “noisy diagnosis” can refer to noise in at least three distinct places: the input narrative, the reasoning environment, and the supervisory signal.
6. Limitations, interpretation, and practical status
NDB’s principal strength is its matched-triplet design: it makes the clean-to-noisy shift measurable while preserving the underlying diagnosis (Mama et al., 15 Sep 2025). The benchmark also avoids direct privacy concerns associated with real patient narratives by relying on synthetic generation, and it is publicly released through the PatientSignal repository (Mama et al., 15 Sep 2025). These properties make it usable as a reproducible stress test for patient-facing medical NLP.
Its limitations are equally clear. The benchmark is synthetic, and the paper provides no human realism ratings, clinician plausibility assessments, lexical comparison to real patient corpora, or external validation against authentic patient-authored text (Mama et al., 15 Sep 2025). The disease inventory, class distribution, and specialty coverage are not enumerated. The split design is underspecified: the paper implies clean-train to noisy-test evaluation, but does not provide exact train/dev/test sizes, split ratios, whether parallel versions of the same case may cross splits, or the split-generation methodology (Mama et al., 15 Sep 2025). Training details and reproducibility controls are likewise sparse.
The reported experiments are also narrow. Only three fine-tuned models are evaluated, only accuracy is reported, and the paper does not include error analysis, confusion matrices, diagnosis-wise breakdowns, ablations over lexical versus syntactic versus semantic distortion, or robustness scores beyond raw accuracy under each condition (Mama et al., 15 Sep 2025). As a result, NDB is best interpreted as an initial benchmark for measuring degradation under noisy patient narratives rather than as a complete evaluation framework for clinical diagnostic robustness.
In the broader benchmark landscape, this suggests that NDB is most informative when read as an instance of a larger design pattern: a diagnostic benchmark whose purpose is to reveal what clean-input leaderboards conceal. Its specific contribution is to show that, in medical NLP, patient-style noise is not an incidental nuisance but a first-order evaluation variable.