BIONNE: Biomedical Nested NER Benchmark
- BIONNE is a nested biomedical NER task that extracts overlapping and hierarchical entity spans in multilingual PubMed abstracts.
- It extends conventional flat NER by requiring accurate recovery of entities across multiple nesting levels and eight key biomedical categories.
- The task compares methodologies such as span/type matching, layered sequence labeling, and hybrid LLM methods across English and Russian datasets.
BIONNE, introduced in the twelfth BioASQ challenge at CLEF 2024, is Biomedical Nested Named Entity Recognition, a shared task designed to extract and classify biomedical named entities that can be nested inside one another in unstructured PubMed abstracts. Its defining objective is to move beyond the flat, non-nested entity recognition that is common in biomedical NLP and instead recover all entity spans at multiple nesting levels together with their biomedical types. The task was organized for English and Russian and offered three tracks: Bilingual, English-oriented, and Russian-oriented (Nentidis et al., 28 Aug 2025).
1. Task definition and nested-entity formulation
BIONNE targets a true nested NER setting. In this setting, entity spans may overlap by containment, the same textual region may participate in multiple entities at different levels, and systems must recognize both broad and fine-grained biomedical mentions. The paper motivates this formulation with structures such as [[[eye] movement] disorders], where a broader span contains a more specific span and nested annotation is required to recover both mentions correctly (Nentidis et al., 28 Aug 2025).
A common misconception in biomedical information extraction is that realistic entity recognition can be handled adequately by flat NER. BIONNE is explicitly framed against that assumption. The task description emphasizes that flat methods generally cannot represent structures in which one entity is embedded within another, whereas biomedical text frequently exhibits precisely such phenomena. In that sense, BIONNE is not merely a multilingual extension of conventional biomedical NER; it is a reformulation of the extraction problem around hierarchical span structure.
The multilingual design is also central to the task definition. Rather than treating English alone as the default setting, BIONNE places Russian and English biomedical abstracts from PubMed within the same evaluation framework. The three-track structure separates monolingual optimization from bilingual modeling and thereby exposes the extent to which nested biomedical extraction can benefit from cross-lingual transfer.
2. Annotation inventory, label space, and representational assumptions
The task is based on a cleaned and biomedical-adapted subset of NEREL-BIO, which is itself an extension of the general-domain NEREL dataset. The organizers report three explicit normalization steps: they corrected annotator errors, merged PRODUCT and DEVICE into a unified DEVICE class, and retained the eight most frequent medical entity types: These eight classes define the operative label inventory for BIONNE and are also the classes used in the evaluation formula (Nentidis et al., 28 Aug 2025).
The paper does not provide the full operational annotation format in detail. However, the system descriptions reveal that at least one participating team implemented a multi-level BIO tagging scheme for nested recognition. This suggests that layered tagging was a practical representation used by competitors, although the official task itself is best understood as span-based at evaluation time: systems predict entity spans and labels, and the final scoring aggregates class-wise performance rather than evaluating a flat token-label sequence.
This distinction matters methodologically. BIONNE is defined by nested span recovery, not by commitment to a single encoding formalism. Sequence-labeling with layered BIO tags, span/type matching, and hybrid extraction pipelines can therefore all be viewed as implementation choices over a common underlying problem: recovering contained biomedical entity structures in multilingual PubMed text.
3. Dataset composition and preparation
The training and validation material was derived from NEREL-BIO and consists of 662 annotated PubMed abstracts in Russian together with 104 parallel abstracts in both Russian and English. The paper reports 40,782 annotated entities in Russian and 8,099 annotated entities in English for these data (Nentidis et al., 28 Aug 2025).
For evaluation, the organizers created a new test set specifically for the shared task. This test set contains 154 abstracts in both English and Russian, with approximately 10,000 annotated entities total. The paper does not provide a more detailed train/dev/test numerical split beyond these official counts.
The preprocessing decisions are significant because they narrow the problem to a more consistent biomedical core. Correcting annotation errors improves label reliability; merging PRODUCT and DEVICE simplifies the inventory; selecting the eight most frequent biomedical entity types concentrates the task on high-frequency medically salient categories. A plausible implication is that BIONNE was designed not only as a nested NER benchmark, but also as a controlled test bed in which multilingual nested extraction could be studied without the additional instability of a very large or noisy type inventory.
4. Evaluation protocol and metric definition
The primary metric for BIONNE is F1-score. The paper states that the reported task score is an average of per-class F1 values across the eight biomedical categories listed in the label inventory (Nentidis et al., 28 Aug 2025).
In prose, the evaluation computes class-wise performance over
and then averages these values. The paper characterizes this as class-averaged F1 and identifies it as the official task score.
The main text does not specify a separate partial-match rule, an exact-match tolerance, or a nesting-specific overlap formula. The safe interpretation given in the paper is that systems are evaluated on the correctness of predicted entity spans and labels, with nesting handled through the underlying gold and predicted span annotations. It therefore presents BIONNE as a nested NER benchmark whose official outcome variable is span-level F1 aggregated over the eight biomedical classes, while leaving lower-level matching details unspecified in the overview.
5. Participating systems and comparative methodology
Three teams are described in the paper. Their submissions illustrate three distinct methodological families for nested biomedical NER: span/type matching with multilingual pretraining, explicit layered sequence labeling, and hybrid LLM-plus-rule pipelines (Nentidis et al., 28 Aug 2025).
fulstock submitted to all three tracks. Its system used BINDER, described as a bi-encoder-style approach, with XLM-RoBERTa as the backbone. The team experimented with multiple forms of entity-type descriptions/prompts: keyword only; 2, 5, or 10 most frequent component words; a contextual prompt with an example sentence; and a lexical prompt where the target is masked with the entity label. Training ran for 64 epochs. In the task summary, this was the strongest and most task-aligned system because BINDER is a span/type matching framework suited to nested entity recognition.
hasin.rehana submitted to the bilingual, English, and Russian tracks. This system implemented a BIO-tagging scheme with six levels of BIO tagging, added six classification layers, one per NER tag level, and expanded the original 8 classes to 17 to fit the BIO tagging setup. It also used the UMLS Metathesaurus for vocabulary expansion. The base encoders were PubMedBERT for English, SBERT-Large-NLU-RU for Russian, and BERT-Base-Multilingual-uncased for bilingual modeling. Among the participants, this is the clearest explicit sequence-labeling baseline.
wenxinzh submitted only to the English-oriented track. Its pipeline combined Mixtral, a spaCy NER model trained on BC5CDR, and UMLS-based rules. Candidate entities were first extracted with Mixtral and the flat BC5CDR-based model, then mapped to UMLS semantic types to assign final labels. The paper characterizes this as a hybrid, largely rule-augmented LLM/NER system.
These systems collectively show that BIONNE could be approached either as a nested span-retrieval problem or as a layered token-labeling problem. The performance distribution reported by the organizers suggests that architectures directly optimized for span-type matching were better aligned with the task than pipelines built around flat NER outputs and post hoc semantic typing.
6. Results, error patterns, and research implications
The official test-set results show a clear separation among approaches. fulstock achieved 0.7044 on the Bilingual track, 0.6181 on the English-oriented track, and 0.6981 on the Russian-oriented track. hasin.rehana scored 0.5053, 0.5636, and 0.6007 on those same tracks, respectively. wenxinzh, which entered only the English-oriented track, scored 0.3480 (Nentidis et al., 28 Aug 2025).
The paper identifies fulstock as the winner on all tracks it entered. Its strongest performance was on the bilingual and Russian-oriented tracks, both near 0.70 F1. hasin.rehana placed second on all three tracks where it participated. The English-only wenxinzh system lagged substantially behind, and the paper interprets this as indicative of weak zero-shot capability from a general LLM plus flat NER model.
Two conclusions are explicit in the task overview. First, pre-trained LLMs without fine-tuning performed poorly for BIONNE. The reason given is the lack of specialized biomedical nested-NER training and inadequate biomedical-specific coverage in models such as Mixtral when used zero-shot. Second, the top system’s advantage indicates that nested biomedical extraction benefits from architectures that directly learn span-type matching, and that multilingual transfer can be effective when training and model design are appropriate.
The broader implications drawn in the paper are that specialized training data is essential, nested structure matters, and multilingual nested NER is feasible but still challenging. BIONNE is positioned as a new benchmark intended to encourage progress in nested entity representation, cross-lingual biomedical adaptation, and methods that move beyond flat tagging and shallow LLM prompting. Within BioASQ 2024 more generally, it marks an expansion beyond English-only and beyond conventional flat biomedical NLP tasks.
