---
title: 'BIONNE: Biomedical Nested NER Benchmark'
url: https://www.emergentmind.com/topics/bionne
type: topic
---

# BIONNE: Biomedical Nested NER Benchmark

BIONNE, introduced in the twelfth BioASQ challenge at CLEF 2024, is **Biomedical Nested Named Entity Recognition**, a shared task designed to extract and classify biomedical named entities that can be nested inside one another in unstructured **PubMed abstracts**. Its defining objective is to move beyond the flat, non-nested entity recognition that is common in biomedical NLP and instead recover **all entity spans at multiple nesting levels** together with their biomedical types. The task was organized for **English and Russian** and offered three tracks: **Bilingual**, **English-oriented**, and **Russian-oriented** [2508.20532].

## 1. Task definition and nested-entity formulation

BIONNE targets a true **nested NER** setting. In this setting, entity spans may **overlap by containment**, the same textual region may participate in multiple entities at different levels, and systems must recognize both broad and fine-grained biomedical mentions. The paper motivates this formulation with structures such as `[[[eye] movement] disorders]`, where a broader span contains a more specific span and nested annotation is required to recover both mentions correctly [2508.20532].

A common misconception in biomedical information extraction is that realistic entity recognition can be handled adequately by flat NER. BIONNE is explicitly framed against that assumption. The task description emphasizes that flat methods generally cannot represent structures in which one entity is embedded within another, whereas biomedical text frequently exhibits precisely such phenomena. In that sense, BIONNE is not merely a multilingual extension of conventional biomedical NER; it is a reformulation of the extraction problem around hierarchical span structure.

The multilingual design is also central to the task definition. Rather than treating English alone as the default setting, BIONNE places **Russian and English biomedical abstracts from PubMed** within the same evaluation framework. The three-track structure separates monolingual optimization from bilingual modeling and thereby exposes the extent to which nested biomedical extraction can benefit from cross-lingual transfer.

## 2. Annotation inventory, label space, and representational assumptions

The task is based on a cleaned and biomedical-adapted subset of **NEREL-BIO**, which is itself an extension of the general-domain **NEREL** dataset. The organizers report three explicit normalization steps: they **corrected annotator errors**, **merged PRODUCT and DEVICE into a unified DEVICE class**, and retained the **eight most frequent medical entity types**:
\[
\{\text{FINDING, DISO, INJURY\_POISONING, PHYS, DEVICE, LABPROC, ANATOMY, CHEM}\}.
\]
These eight classes define the operative label inventory for BIONNE and are also the classes used in the evaluation formula [2508.20532].

The paper does not provide the full operational annotation format in detail. However, the system descriptions reveal that at least one participating team implemented a **multi-level BIO tagging scheme** for nested recognition. This suggests that layered tagging was a practical representation used by competitors, although the official task itself is best understood as **span-based at evaluation time**: systems predict entity spans and labels, and the final scoring aggregates class-wise performance rather than evaluating a flat token-label sequence.

This distinction matters methodologically. BIONNE is defined by nested span recovery, not by commitment to a single encoding formalism. Sequence-labeling with layered BIO tags, span/type matching, and hybrid extraction pipelines can therefore all be viewed as implementation choices over a common underlying problem: recovering contained biomedical entity structures in multilingual PubMed text.

## 3. Dataset composition and preparation

The training and validation material was derived from **NEREL-BIO** and consists of **662 annotated PubMed abstracts in Russian** together with **104 parallel abstracts in both Russian and English**. The paper reports **40,782 annotated entities in Russian** and **8,099 annotated entities in English** for these data [2508.20532].

For evaluation, the organizers created a **new test set** specifically for the shared task. This test set contains **154 abstracts in both English and Russian**, with approximately **10,000 annotated entities** total. The paper does not provide a more detailed train/dev/test numerical split beyond these official counts.

The preprocessing decisions are significant because they narrow the problem to a more consistent biomedical core. Correcting annotation errors improves label reliability; merging **PRODUCT** and **DEVICE** simplifies the inventory; selecting the eight most frequent biomedical entity types concentrates the task on high-frequency medically salient categories. A plausible implication is that BIONNE was designed not only as a nested NER benchmark, but also as a controlled test bed in which multilingual nested extraction could be studied without the additional instability of a very large or noisy type inventory.

## 4. Evaluation protocol and metric definition

The **primary metric** for BIONNE is **F1-score**. The paper states that the reported task score is an **average of per-class F1 values** across the eight biomedical categories listed in the label inventory [2508.20532].

In prose, the evaluation computes class-wise performance over
\[
\{\text{FINDING, DISO, INJURY\_POISONING, PHYS, DEVICE, LABPROC, ANATOMY, CHEM}\}
\]
and then averages these values. The paper characterizes this as class-averaged F1 and identifies it as the official task score.

The main text does **not** specify a separate partial-match rule, an exact-match tolerance, or a nesting-specific overlap formula. The safe interpretation given in the paper is that systems are evaluated on the correctness of predicted **entity spans and labels**, with nesting handled through the underlying gold and predicted span annotations. It therefore presents BIONNE as a nested NER benchmark whose official outcome variable is span-level F1 aggregated over the eight biomedical classes, while leaving lower-level matching details unspecified in the overview.

## 5. Participating systems and comparative methodology

Three teams are described in the paper. Their submissions illustrate three distinct methodological families for nested biomedical NER: span/type matching with multilingual pretraining, explicit layered sequence labeling, and hybrid LLM-plus-rule pipelines [2508.20532].

**fulstock** submitted to **all three tracks**. Its system used **BINDER**, described as a bi-encoder-style approach, with **XLM-RoBERTa** as the backbone. The team experimented with multiple forms of **entity-type descriptions/prompts**: keyword only; 2, 5, or 10 most frequent component words; a contextual prompt with an example sentence; and a lexical prompt where the target is masked with the entity label. Training ran for **64 epochs**. In the task summary, this was the strongest and most task-aligned system because BINDER is a span/type matching framework suited to nested entity recognition.

**hasin.rehana** submitted to the **bilingual**, **English**, and **Russian** tracks. This system implemented a **BIO-tagging scheme** with **six levels of BIO tagging**, added **six classification layers**, one per NER tag level, and expanded the original **8 classes to 17** to fit the BIO tagging setup. It also used the **UMLS Metathesaurus** for vocabulary expansion. The base encoders were **PubMedBERT** for English, **SBERT-Large-NLU-RU** for Russian, and **BERT-Base-Multilingual-uncased** for bilingual modeling. Among the participants, this is the clearest explicit sequence-labeling baseline.

**wenxinzh** submitted only to the **English-oriented** track. Its pipeline combined **Mixtral**, a **spaCy NER model** trained on **BC5CDR**, and **UMLS-based rules**. Candidate entities were first extracted with Mixtral and the flat BC5CDR-based model, then mapped to UMLS semantic types to assign final labels. The paper characterizes this as a hybrid, largely rule-augmented LLM/NER system.

These systems collectively show that BIONNE could be approached either as a nested span-retrieval problem or as a layered token-labeling problem. The performance distribution reported by the organizers suggests that architectures directly optimized for span-type matching were better aligned with the task than pipelines built around flat NER outputs and post hoc semantic typing.

## 6. Results, error patterns, and research implications

The official test-set results show a clear separation among approaches. **fulstock** achieved **0.7044** on the **Bilingual** track, **0.6181** on the **English-oriented** track, and **0.6981** on the **Russian-oriented** track. **hasin.rehana** scored **0.5053**, **0.5636**, and **0.6007** on those same tracks, respectively. **wenxinzh**, which entered only the English-oriented track, scored **0.3480** [2508.20532].

The paper identifies **fulstock** as the winner on all tracks it entered. Its strongest performance was on the **bilingual** and **Russian-oriented** tracks, both near **0.70 F1**. **hasin.rehana** placed second on all three tracks where it participated. The English-only **wenxinzh** system lagged substantially behind, and the paper interprets this as indicative of weak zero-shot capability from a general LLM plus flat NER model.

Two conclusions are explicit in the task overview. First, **pre-trained LLMs without fine-tuning performed poorly** for BIONNE. The reason given is the lack of specialized biomedical nested-NER training and inadequate biomedical-specific coverage in models such as Mixtral when used zero-shot. Second, the top system’s advantage indicates that nested biomedical extraction benefits from architectures that directly learn **span-type matching**, and that **multilingual transfer can be effective** when training and model design are appropriate.

The broader implications drawn in the paper are that **specialized training data is essential**, **nested structure matters**, and **multilingual nested NER is feasible but still challenging**. BIONNE is positioned as a new benchmark intended to encourage progress in **nested entity representation**, **cross-lingual biomedical adaptation**, and methods that move beyond **flat tagging** and **shallow LLM prompting**. Within BioASQ 2024 more generally, it marks an expansion beyond English-only and beyond conventional flat biomedical NLP tasks.

Source: https://www.emergentmind.com/topics/bionne