- The paper presents a manually annotated, prion-specific NER resource with 6,955 entities across 15 coarse and 31 fine-grained types.
- The dataset is constructed via rigorous PubMed abstract screening and expert annotation, achieving strong inter-annotator agreement (81% F1).
- Benchmarking shows supervised models outperform zero-shot LLMs, emphasizing challenges in extracting structurally complex rare biomedical entities.
PrionNER: A Domain-Specific NER Resource for Prion Disease Biomedical Literature
Motivation and Dataset Design
Named entity recognition in biomedical NLP has historically depended on broad-domain corpora and generalized annotation schemes. Existing benchmarks frequently prioritize high-frequency clinical entities, or focus on well-characterized diseases, limiting their utility for rare, diagnostically nuanced disorders such as prion diseases. Prion diseases are rapidly progressive, fatal neurodegenerative disorders with challenging clinical recognition due to their nonspecific presentation and the need for integrating heterogeneous diagnostic evidence. The absence of a publicly available, prion-focused NER dataset has impeded rare-disease extraction tasks and structured clinical benchmarking.
PrionNER fills this gap with a manually annotated NER dataset tailored to prion disease clinical information extracted from PubMed abstracts. The resource comprises 317 abstracts, 2,943 sentences, and 6,955 annotated entities, spanning 15 coarse-grained and 31 fine-grained types categorized into evidence groups that reflect clinical diagnostic workflows: Case Input, Case Diagnosis, and Clinical Course/Context. This schema extends beyond disease and symptom spans, incorporating diverse forms of biomedical evidence (diagnostics, findings, anatomy, treatment, temporal/statistical information), supporting realistic rare-disease knowledge extraction scenarios.
Corpus Construction, Annotation, and Quality Control
Corpus assembly used a domain-specific, Boolean PubMed query combining prion disease nomenclature and clinical context terms, while explicitly excluding basic science and non-human research. After initial retrieval, GPT-5.4 was employed for high-recall, over-inclusive automated abstract screening, reducing manual workload; post-screening, two annotators performed human review for final eligibility, yielding 772 relevant abstracts, of which 317 were annotated due to practical constraints.
Entity annotation leveraged iterative pilot phases with medical doctors experienced in neurological disease. Schema refinement was conducted through adjudicated joint review to maintain clinical coherence and NLP span-level consistency. The test split consists of 70 double-annotated, adjudicated abstracts, with 81.78 entity-level exact-match F1 for inter-annotator agreement, and supporting metrics (Jaccard similarity 69.17, token-level BIO kappa 81.05) confirming strong reliability despite the long-tailed label distribution and span-boundary ambiguity inherent to rare-disease contexts.
Dataset Characterization: Label Distribution and Structural Complexity
PrionNER exhibits strongly long-tailed entity frequency: symptoms, generic prion mentions, and anatomical locations dominate, but medically salient categories such as subtypes, findings, and epidemiological metrics remain rare. The span-level annotation preserves clinically adjacent labels (e.g., sCJD, vCJD, FFI, GSS, iCJD) and structurally complex entity forms: discontinuous, nested, and overlapping mentions are retained in training and evaluation, reflecting real-world diagnostic narratives and complicating extraction. The diversity of normalized surface forms further imposes generalization requirements—unique mention forms far exceed raw counts, and train–test vocabulary overlap is limited, especially on sparse categories.
PrionNER was benchmarked using leading supervised biomedical encoders (BioBERT, ClinicalBERT, PubMedBERT) and structured NER models (W2NER), alongside zero-shot LLMs (GPT-5.4, Gemma-4-31B, GLiNER2 variants, GLiNER-BioMed). Flat-ner settings evaluated span and label recovery for contiguous, non-overlapping gold labels; non-flat-ner evaluated nested/discontinuous/overlapping structures. Evaluation was performed for both coarse- and fine-grained schema.
Key numerical findings:
- Flat-ner supervised: W2NER achieved 81.86 coarse-grained and 80.46 fine-grained F1. PubMedBERT reached 81.62 and 79.49, respectively.
- Flat-ner zero-shot: Gemma-4-31B reached 71.41 coarse-grained and 68.41 fine-grained F1; GPT-5.4 trailed slightly.
- Non-flat-ner: All models struggled; W2NER best at 13.48/13.70 F1 (coarse/fine), Gemma-4-31B led zero-shot at 6.97/6.55.
- Substantial precision–recall trade-offs: zero-shot LLMs maintained moderate precision but suffered recall losses, especially on rare and structurally complex mentions.
Fine-grained and per-type analysis revealed that supervised models outperform zero-shot alternatives across most frequent clinical entities. Zero-shot LLMs exhibited isolated strengths for distinctive subtypes (FFI, fCJD, Genetic_test), but none matched schema-wide robustness of supervised encoders. Label confusions clustered around clinically adjacent, context-dependent categories (prion subtypes, imaging findings, treatment vs. disease, outcome complications).
Implications, Limitations, and Future Directions
PrionNER establishes a clinically motivated, rare-disease NER benchmark with strong annotation quality and realistic entity structure. The dataset highlights persistent challenges in biomedical extraction tasks for rare disorders: extreme label sparsity, fine-grained diagnostic distinctions, structural entity complexity, and semantic context requirements. Benchmarking results demonstrate that schema-specific supervised models generalize better and recover rare evidence types more consistently, but even advanced LLMs remain limited under zero-shot conditions, particularly for non-flat and nested cases.
From a practical perspective, PrionNER's schema and publicly released annotations support downstream tasks such as relation extraction, document-level inference, and ontology-driven retrieval for rare-disease cohorts. Theoretically, the resource provides a compact case study for information extraction under low-resource, high-specificity, and structurally complex settings, and surfaces generalizable requirements for future rare-disease NLP (label taxonomy design, annotation reliability, model architecture, evaluation metrics).
Limitations include corpus size and coverage (317 abstracts), domain focus on prion disorders, and reliance on biomedical abstracts rather than clinical notes. Expanding PrionNER to broader text sources, refining and increasing coverage of sparse evidence types, and leveraging synthetic data generation or active learning may further improve rare-disease extraction pipelines. Integrating PrionNER with broader biomedical knowledge graphs and diagnostic support systems would enable practical clinical applications and inform model development for rare-disease domains.
Conclusion
PrionNER provides a high-quality, clinically grounded NER benchmark for prion disease biomedical literature, supporting both rare-disease NLP research and practical information extraction applications. It demonstrates the necessity of domain-specific annotation, tailored schema design, and rigorous quality control for specialized clinical evidence extraction. Benchmarking results reveal the strengths and ongoing limitations of supervised and zero-shot extraction systems, motivating future advances in model architecture and domain adaptation for rare-disease information extraction.