PrionNER: Prion Disease NER Dataset
- PrionNER is a manually annotated NER dataset capturing detailed clinical entities from PubMed abstracts related to prion diseases.
- It employs a two-level annotation schema with 15 coarse-grained and 31 fine-grained entity types that mirror diagnostic evidence.
- The dataset’s design aids downstream applications such as knowledge graph construction and rare disease diagnostic support.
to=arxiv_search.search ฝ่ายขายละคร json {"query":"PrionNER named entity recognition prion disease biomedical literature", "max_results": 5} to=arxiv_search.search 大发快三如何 天天中彩票提款json {"query":"W2NER named entity recognition arXiv", "max_results": 5} to=arxiv_search.search ＿久久爱 天天中彩票这个json {"query":"prion disease molecular dynamics resistance variant arXiv", "max_results": 5} PrionNER is a manually annotated named entity recognition dataset for prion disease clinical information in PubMed abstracts, designed to support clinically grounded information extraction from prion disease biomedical literature (Dao et al., 27 May 2026). Its current release comprises 317 abstracts, 2,943 sentences, and 6,955 text-bound entity annotations spanning 15 coarse-grained and 31 fine-grained clinically oriented entity types covering diseases, symptoms, diagnostics, findings, anatomy, treatments, and temporal and statistical evidence. The dataset is motivated by the diagnostic profile of prion diseases as rare, rapidly progressive, and fatal neurodegenerative disorders whose early clinical presentation is often nonspecific, making structured extraction of diagnostically relevant evidence from the literature especially valuable (Dao et al., 27 May 2026).
1. Scope, motivation, and domain orientation
PrionNER was introduced to address the absence of a publicly available prion-disease-focused dataset capturing a broad range of clinically relevant entities from the biomedical literature (Dao et al., 27 May 2026). The design premise is that prion-disease diagnosis is not reducible to disease-name detection alone. Rather, it depends on heterogeneous evidence types, including symptoms, tests, imaging findings, anatomical sites, temporal cues, treatments, complications, and quantitative evidence such as sensitivity and prevalence.
This domain orientation distinguishes PrionNER from broader biomedical datasets such as NCBI Disease, BC5CDR, and MedMentions, which are mentioned as comparators but not as prion-specific resources. PrionNER instead focuses on a single rare disease domain and uses a schema designed around the diagnostic process itself. A plausible implication is that the dataset is intended not only for benchmark evaluation, but also for downstream clinical evidence aggregation, knowledge graph construction, and diagnostic-support systems, all of which are explicitly named use cases in the source paper (Dao et al., 27 May 2026).
The corpus reflects the clinically salient evidence space of prion disease. The paper notes that it captures hallmarks such as dementia, myoclonus, and ataxia; includes subtypes such as FFI, GSS, vCJD, and iCJD; and represents disease-course expressions such as within a year. This suggests that PrionNER is structured to align with how prion disorders are discussed in clinical abstracts rather than with a purely ontological or lexicon-driven tagging strategy (Dao et al., 27 May 2026).
2. Corpus construction and released dataset
The corpus was assembled from PubMed abstracts using a Boolean query over title and abstract fields. The query combined broad prion terms such as “Prion Diseases,” “Creutzfeldt-Jakob Disease,” “CJD,” “Kuru,” “Gerstmann-Straussler-Scheinker,” and “Fatal Familial Insomnia,” with clinical-context terms including diagnosis, clinical, symptoms, case, progression, and treatment, while excluding non-clinical or basic-science terms such as mice, mouse, rat, animal, cell, protein, and in vitro (Dao et al., 27 May 2026).
The initial query returned 3,414 abstracts. After removing empty abstracts and preprocessing, 3,138 abstracts remained. Because many keyword matches were still off-target, including basic science papers, animal studies, non-human studies, purely epidemiological or economic papers, and articles where prion disease was only a secondary topic, the authors used GPT-5.4 as a high-recall relevance screener. Abstracts were judged relevant only if they primarily concerned human prion disease in a clinical context, specifically diagnosis, symptoms, progression, or treatment (Dao et al., 27 May 2026).
On an audited set of 1,383 manually reviewed abstracts, GPT-5.4 achieved 90.60 accuracy, 89.65 balanced accuracy, 86.98 precision for the related class, 97.80 recall for the related class, and 92.07 F1 for the related class. The model missed only 17 truly relevant abstracts and over-called 113 irrelevant abstracts as relevant. The paper treats this trade-off as acceptable because recall was prioritized during corpus construction (Dao et al., 27 May 2026).
The released dataset currently contains 317 abstracts, split into 247 train abstracts and 70 test abstracts. The test split is the adjudicated version of a double-annotated subset and is therefore the highest-confidence evaluation portion.
| Split | Abstracts | Sentences | Entities |
|---|---|---|---|
| Train | 247 | 2,156 | 5,149 |
| Test | 70 | 787 | 1,806 |
| Total | 317 | 2,943 | 6,955 |
The appendix also reports a split-level summary in terms of entity types, mentions, and unique surface forms: the train split contains 28 entity types, 4,655 mentions, and 1,889 unique surface forms, while the test split contains 29 entity types, 1,650 mentions, and 729 unique surface forms (Dao et al., 27 May 2026).
3. Annotation schema and representational choices
PrionNER uses a two-level schema consisting of 15 coarse-grained entity types and 31 fine-grained entity types (Dao et al., 27 May 2026). The schema is organized into three clinically meaningful groups: Case Input, Case Diagnosis, and Clinical Course and Context.
Within Case Input, the coarse-grained types are Age, Symptom, Test_name, Sequences, Anatomic_location, and Findings. Test_name is refined into Imaging_test, Electrophysio_test, Blood_biomarker_test, Genetic_test, Molecular_assay, and Autopsy. Sequences includes Imaging_sequence. Findings includes Imaging_finding and Autopsy_finding. Within Case Diagnosis, the schema includes Generic_Prion, Sporadic_Prion, Familial_Prion, Acquired_Prion, and Differential_Diagnosis, with fine-grained subtype labels such as sCJD, sFI, VPSPr, fCJD, GSS, FFI, vCJD, iCJD, and Kuru. Within Clinical Course and Context, the schema includes Treatment, Complication, Time, and Stats, with Time divided into Duration and Time_point, and Stats divided into Sensitivity, Specificity, Prevalence, and Incidence (Dao et al., 27 May 2026).
The annotation guidelines enforce mention-level annotation with minimal span selection. Annotators are instructed to annotate only explicit mentions, choose the most specific label, keep spans minimal but complete, annotate both long form and abbreviation if both are explicit, avoid keyword matching alone, and label a span according to its clinical meaning in context. Several design decisions are stated explicitly: brain is annotated as Anatomic_location; death and died are annotated as Complication; imaging sequences such as DWI, FLAIR, and ADC are labeled Imaging_sequence rather than Imaging_test; and statistical expressions are annotated by the value span only, so in sensitivity of 100%, the span 100% is labeled Sensitivity (Dao et al., 27 May 2026).
A common misconception would be to treat PrionNER as a standard flat disease-mention corpus. The paper explicitly rejects that characterization. The schema is clinically oriented rather than solely disease-centric, and it preserves structurally complex entity forms. This suggests that PrionNER is intended to model diagnostic evidence composition, not merely terminology lookup.
4. Annotation process, quality control, and structural complexity
The annotation workflow had three stages. First, a pilot annotation phase covered 20 abstracts, with two annotators labeling independently and disagreements reviewed jointly; the guidelines were established from this stage. Second, the training annotation phase covered 151 abstracts by Annotator 1 and 96 abstracts by Annotator 2, with iterative guideline refinement during annotation. Third, a double annotation and adjudication phase covered 70 abstracts independently annotated by both annotators; agreement was measured before adjudication, disagreements were then resolved jointly, and the finalized subset became the test split (Dao et al., 27 May 2026).
Inter-annotator agreement was measured on the pre-adjudication 70-abstract subset using entity-level exact-match F1, Jaccard similarity, and Cohen’s kappa computed on token-level BIO labels. The reported agreement was 81.78 entity-level exact F1, 69.17 Jaccard similarity, and 81.05 Cohen’s kappa, which the paper interprets as strong consistency despite fine-grained categories and difficult span decisions (Dao et al., 27 May 2026). Agreement was highest for clear disease labels such as vCJD, GSS, FFI, and sCJD, and lower for sparse or boundary-sensitive categories such as Complication, Prevalence, Incidence, and Imaging_finding.
PrionNER explicitly preserves discontinuous entities, nested entities, and overlapping entities. In the test set there are 34 discontinuous entities, 80 nested pairs, and 2 overlapping pairs; in the train set there are 97 discontinuous entities, 235 nested pairs, and 5 overlapping pairs (Dao et al., 27 May 2026). Nested structures are therefore much more common than overlaps. This is significant because many conventional BIO-based tagging systems are optimized for flat contiguous spans and cannot directly capture such structures.
The evaluation framing reflects this representational distinction. For the flat setting, the paper uses BIO tags of the form , where is a schema type. It then evaluates both Flat NER, which includes only contiguous, non-overlapping entities, and Non-flat NER, which includes only nested, discontinuous, and overlapping entities (Dao et al., 27 May 2026).
5. Benchmark design and empirical results
The benchmark includes both supervised and zero-shot systems (Dao et al., 27 May 2026). The supervised models are BioBERT, ClinicalBERT, PubMedBERT, and W2NER with PubMedBERT embeddings. The BERT baselines use fixed hyperparameters: maximum length 512, 5 epochs, learning rate , weight decay 0.01, train batch size 8, evaluation batch size 16, and seed 42. The zero-shot models are GPT-5.4, Gemma-4-31B, Gemma-4-26B, GLiNER2-short, GLiNER2-def, and GLiNER-BioMed. Because zero-shot systems often return entity strings rather than exact offsets, their outputs require a normalization and alignment step (Dao et al., 27 May 2026).
For flat NER, W2NER is the strongest supervised model and Gemma-4-31B is the strongest zero-shot model. The main coarse- and fine-grained flat results are as follows.
| Setting | Best model | Precision | Recall | F1 |
|---|---|---|---|---|
| Coarse-grained flat NER | W2NER | 84.23 | 79.63 | 81.86 |
| Fine-grained flat NER | W2NER | 83.07 | 78.01 | 80.46 |
| Best zero-shot, coarse flat | Gemma-4-31B | 78.33 | 65.61 | 71.41 |
| Best zero-shot, fine flat | Gemma-4-31B | 80.14 | 59.67 | 68.41 |
Among the standard BERT baselines, PubMedBERT performs best, with 81.62 F1 on coarse-grained flat NER and 79.49 F1 on fine-grained flat NER. BioBERT and ClinicalBERT trail slightly behind, while the GLiNER variants are markedly weaker, especially GLiNER-BioMed and GLiNER2-short (Dao et al., 27 May 2026).
The non-flat setting is substantially harder. W2NER again performs best, but only reaches 13.48 coarse F1 and 13.70 fine F1. The zero-shot models remain far below supervised W2NER, although Gemma-4-26B reaches 7.75 coarse F1; its fine F1 is only 0.63. GPT-5.4 scores 5.84 coarse F1 and 5.84 fine F1, while GLiNER-BioMed scores 0.00 on both (Dao et al., 27 May 2026).
The per-type analysis reported in the paper shows that supervised models are strongest on common labels such as Symptom, Anatomic_location, Generic_Prion, sCJD, and Imaging_test. Zero-shot models can be relatively strong on distinctive labels such as FFI, Kuru, fCJD, Genetic_test, and Molecular_assay, but they are much less stable across the full schema. The paper’s main empirical pattern is that zero-shot systems can be fairly precise, yet they omit too many mentions, so recall lags well behind supervised systems (Dao et al., 27 May 2026).
6. Difficulty profile, significance, and limitations
The benchmark is described as challenging, especially for structurally complex mentions and fine-grained clinically adjacent label distinctions (Dao et al., 27 May 2026). The paper attributes this difficulty to five factors: limited training data; a long-tailed label distribution; nested and discontinuous mention structures; fine distinctions between clinically adjacent labels; and semantic ambiguity in expressions such as subtype names and findings. High lexical variability and train/test mismatch further complicate learning. The dataset is therefore not simply low-resource; it is simultaneously low-resource, fine-grained, and non-flat.
The long-tailed nature of the label space is a central property. The most frequent labels are Symptom, Generic_Prion, Anatomic_location, Imaging_test, and Duration, whereas rare but clinically important labels include Sensitivity, Specificity, Prevalence, Incidence, and sparse subtype labels such as sFI and vCJD. The paper notes that Sensitivity and Specificity are absent from training and appear only in the test split in small numbers, reflecting real-world rarity rather than artificial balancing (Dao et al., 27 May 2026).
PrionNER’s significance lies in its clinically grounded coverage of the prion-disease evidence space. The released resources include the dataset annotations, annotation guidelines, and evaluation scripts at https://github.com/daotuanan/PrionNER/ (Dao et al., 27 May 2026). The paper positions the dataset as useful for NER benchmarking, information extraction pipelines, knowledge base construction, rare-disease decision support, and downstream tasks such as relation extraction, document-level evidence aggregation, and normalization.
The stated limitations are equally clear. The released corpus is small, with only 317 abstracts. It is domain-specific, so transferability to other diseases may be limited. It is derived from PubMed abstracts rather than clinical notes, making it more condensed and structured than many real-world clinical text sources. Sparse labels remain sparse, including prevalence- and sensitivity-related categories (Dao et al., 27 May 2026). Future work is said to require corpus expansion, improved sparse-label coverage, and incorporation of additional clinical text sources.
In summary, PrionNER is a prion-specific biomedical NER benchmark whose principal contribution is a clinically aligned representation of prion disease literature rather than a simple disease-name annotation layer. Its combination of 15 coarse-grained and 31 fine-grained entity types, long-tail label distribution, and preservation of nested, discontinuous, and overlapping mentions makes it simultaneously a resource for rare-disease information extraction and a stress test for current NER methodology (Dao et al., 27 May 2026).