MultiCardioNER: Multilingual Clinical NER

• MultiCardioNER offers a cardiology-specific benchmark that evaluates exact-span NER for diseases and drugs across multiple languages.
• The task combines mixed-specialty resources with a specialized CardioCCC dataset to boost recall through domain adaptation and strict span matching.
• Transformer-based models and ensemble methods dominate system submissions, leveraging multilingual checkpoints for optimal performance in Spanish, English, and Italian.

MultiCardioNER is a shared task introduced in the twelfth BioASQ challenge to advance clinical named entity recognition in the cardiology domain and to test multilingual adaptation beyond Spanish (Nentidis et al., 28 Aug 2025). It focuses on flat NER over clinical case reports, with strict span-level evaluation and two entity categories central to cardiology: Diseases and Drugs. The task narrows the scope of earlier Spanish clinical case report benchmarks such as DisTEMIST, MedProcNER/ProcTEMIST, and SympTEMIST to a specialty domain, while expanding multilingual coverage to Spanish, English, and Italian. Within BioASQ 2024, 37 competing teams submitted more than 700 distinct runs across four shared tasks; MultiCardioNER contributed a cardiology-specific, multilingual benchmark for exact-span entity detection rather than normalization or nested extraction (Nentidis et al., 28 Aug 2025).

1. Position within BioASQ and task definition

MultiCardioNER was formulated around two core objectives: adaptation from mixed-specialty clinical reports to cardiology-specific case reports, and multilingual transfer from Spanish to English and Italian. Its rationale was to add a multilingual dimension to clinical NER while focusing on a specialty “of high clinical impact.” The task therefore combines specialty adaptation with cross-lingual modeling, rather than treating multilingualism as an isolated benchmark condition (Nentidis et al., 28 Aug 2025).

The shared task is partitioned into two subtracks. CardioDis targets Diseases in Spanish cardiology texts only, with annotations in CardioCCC following DisTEMIST guidelines. MultiDrug targets Drugs in Spanish, English, and Italian, with languages evaluated separately. In both subtracks, the task is flat NER: there is no nesting, no partial or overlap credit, and systems are evaluated only when they return the exact span boundaries together with the correct entity type. Multi-word mentions are common and are judged under the same strict regime, as illustrated by disease mentions such as “atrial flutter with rapid ventricular response” and “Takotsubo syndrome,” and drug mentions such as “nytroglicerine” and “clopidogrel” (Nentidis et al., 28 Aug 2025).

A common misconception is to treat MultiCardioNER as a multilingual normalization task because DisTEMIST and related resources include richer downstream semantics. In the shared task itself, however, the target is detection of mentions, not entity linking or normalization. A second misconception is to conflate it with nested biomedical extraction. That is explicitly excluded: MultiCardioNER is flat NER, whereas nested structure is addressed by the separate BIONNE task within the same BioASQ edition (Nentidis et al., 28 Aug 2025).

2. Corpora, multilingual construction, and data regime

The data design combines mixed-specialty training resources with a cardiology-specific evaluation corpus. DisTEMIST provides Spanish disease annotations over 1,000 clinical case reports, with 406,137 tokens, 2,335,968 characters, 10,664 annotations, and 6,739 unique annotations. DrugTEMIST uses the same collection of case reports for Drugs and provides 1,000 documents per language: Spanish has 406,137 tokens, 2,335,968 characters, 2,778 annotations, and 925 unique annotations; English has 404,194 tokens, 2,230,631 characters, 2,814 annotations, and 875 unique annotations; Italian has 421,251 tokens, 2,393,002 characters, 2,808 annotations, and 893 unique annotations (Nentidis et al., 28 Aug 2025).

CardioCCC is the cardiology clinical case collection used for specialty adaptation and formal testing. It contains 508 documents, with 250 reserved as test and 258 released for training and development at participants’ discretion. For Spanish Diseases, CardioCCC contains 568,297 tokens, 3,215,774 characters, 18,232 annotations, and 7,692 unique annotations. For Drugs, the same 508 documents are available multilingualy: Spanish has 568,297 tokens, 3,215,774 characters, 4,227 annotations, and 755 unique annotations; English has 576,772 tokens, 3,114,833 characters, 4,231 annotations, and 734 unique annotations; Italian has 595,332 tokens, 3,345,466 characters, 4,385 annotations, and 752 unique annotations (Nentidis et al., 28 Aug 2025).

Resource	Scope	Size and annotation summary
DisTEMIST	ES Diseases; mixed-specialty training resource	1,000 docs; 406,137 tokens; 10,664 annotations; 6,739 unique
DrugTEMIST	Drugs; mixed-specialty training resource in ES/EN/IT	1,000 docs per language; ES 2,778, EN 2,814, IT 2,808 annotations
CardioCCC	Cardiology-specific corpus for adaptation and test	508 docs; 258 train/dev, 250 test; Diseases in ES, Drugs in ES/EN/IT
Background set	Additional clinical case reports in ES/EN/IT	7,625 docs; Silver Standard aggregating participant predictions will be released

The multilingual component is especially explicit in MultiDrug. Spanish texts were machine-translated into English and Italian and then revised and annotated by native clinical experts, creating aligned multilingual resources that support cross-lingual transfer. Some documents in the 7,625-document background set are in their original language, while the remainder are translated. This suggests that MultiCardioNER was designed not only to benchmark end-task performance, but also to support controlled experiments on transfer between aligned corpora and specialty adaptation under a shared annotation scheme (Nentidis et al., 28 Aug 2025).

3. Annotation scheme, representation, and official evaluation

Disease annotations in CardioCCC follow the DisTEMIST guidelines, and Drug annotations follow the DrugTEMIST guidelines. In the implementation study centered on the shared task, all annotations are described as BRAT standoff spans with start and end offsets, created with the BRAT tool through multiple quality-control cycles and consistency analyses (Danu et al., 20 Oct 2025). The overview paper, however, does not report inter-annotator agreement or detailed quality-control metrics numerically, and licensing details and evaluation scripts are likewise not specified (Nentidis et al., 28 Aug 2025).

The official evaluation is strict, span-level, and type-correct. No partial or overlap credit is assigned, and there is no special evaluation treatment reported for abbreviations. Languages are evaluated separately in MultiDrug, and leaderboard ranking is by micro-averaged F1. The reported formulas are:

$P = \frac{TP}{TP + FP}$

$R = \frac{TP}{TP + FN}$

$F1 = \frac{2PR}{P + R}$

Micro-averaging is computed over all predicted and gold spans of the single target entity type per subtrack: Diseases in CardioDis, and Drugs per language in MultiDrug. Macro-averaging is not used (Nentidis et al., 28 Aug 2025).

A model-facing representation described in the BERT-based study casts the task as BIO sequence labeling. For Diseases in Spanish, the label list is [B-ENFERMEDAD, I-ENFERMEDAD, O, CLS, SEP]. For medications in Spanish, English, and Italian, the label list is [B-FARMACO, I-FARMACO, O, CLS, SEP]. CLS marks sentence start and also serves as PAD; SEP marks sentence end. BIO outputs are then post-processed back into BRAT spans with exact offsets (Danu et al., 20 Oct 2025).

4. Participating systems and benchmark outcomes

Seven teams submitted 70 runs in total, with up to five runs per subtrack and per language. The dominant methodological pattern was transformer-based NER, often with ensemble construction. Systems used both language-specific and multilingual checkpoints, including RoBERTa, roberta-es-clinical-trials-ner, CLIN-X-ES, XLM-R, BioLinkBERT, SciBERT, some general-domain BERTs for Italian, and occasional use of mDeBERTa. Additional strategies included ensemble learning, multi-head CRF layers, domain adaptation via CardioCCC, data augmentation, window sliding, custom drug dictionaries, and post-processing through string matching and filtering. ICUE explored multilingual and language-specific BERTs with additional re-training and rules, and GPT-3.5 was used in some pipelines for translation support. No separate organizer baseline is reported (Nentidis et al., 28 Aug 2025).

The best CardioDis result on the 250-document Spanish test set was an F1 of 0.8199 by BIT.UA (run1-all-full), with $P=0.8155$ and $R=0.8243$. The second-best F1 was 0.8145 by BIT.UA (run0-top5-full), with $R=0.8181$. Enigma obtained F1 $=0.8049$ with CLIN-X-ES fine-tuned on DisTEMIST+CardioCCC. The overview also states that the best precision reported was 0.8886 by PICUSLab using an ensemble plus string matching, although a table row shown for one PICUSLab run lists $P=0.7794$, $R=0.8030$, and $F1=0.791$ (Nentidis et al., 28 Aug 2025).

For MultiDrug, the best Spanish score was F1 $R = \frac{TP}{TP + FN}$0 by ICUE, with $R = \frac{TP}{TP + FN}$1 and $R = \frac{TP}{TP + FN}$2; the second-best Spanish F1 was 0.9238 by Enigma, with $R = \frac{TP}{TP + FN}$3. In English, the best F1 was 0.9223 by Enigma, with $R = \frac{TP}{TP + FN}$4 and $R = \frac{TP}{TP + FN}$5, followed by ICUE at approximately 0.9107. In Italian, the best F1 was 0.8842 by Enigma, with $R = \frac{TP}{TP + FN}$6 and $R = \frac{TP}{TP + FN}$7; the second-best F1 was 0.8838, also by Enigma, when trained on Italian/Spanish RoBERTa (Nentidis et al., 28 Aug 2025).

These outcomes support two recurrent observations from the official discussion. First, systems that incorporated CardioCCC tended to improve recall and overall F1 relative to systems trained only on mixed-specialty corpora. Second, Drug recognition was generally easier than Disease recognition, with higher scores across languages. The overview attributes the latter to lower heterogeneity and lower boundary complexity for Drugs than for Diseases (Nentidis et al., 28 Aug 2025).

5. BERT-based modeling study and implementation profile

A dedicated study on MultiCardioNER develops multiple deep contextual embedding models based on monolingual and multilingual BERT checkpoints trained on general-domain text, then adapted to the clinical and cardiology setting (Danu et al., 20 Oct 2025). The subtasks are named SDR for Spanish Diseases Recognition, SMR for Spanish Medications Recognition, EMR for English Medications Recognition, and IMR for Italian Medications Recognition. The study evaluates Spanish mrm8488/bert-spanish-cased-finetuned-ner, English dslim/bert-base-NER, Italian nickprock/bert-italian-finetuned-ner, and multilingual Davlan/bert-base-multilingual-cased-ner-hrl (Danu et al., 20 Oct 2025).

The adaptation regime distinguishes “Clinical” models from “Cardio” models. Clinical-SDR and Clinical-SMR/EMR/IMR are obtained by adapting the general-domain checkpoints on DisTEMIST or DrugTEMIST and validating on CardioCCC development sets. Cardio-SDR and Cardio-SMR/EMR/IMR then further fine-tune on CardioCCC development data to specialize for cardiology. A multilingual medication model, MultiCardio-MMR, is trained on the aggregated multilingual dataset and evaluated separately by language. Reports are split into sentences of maximum length 256, tokenized with character offset preservation, encoded with BIO labels, and decoded back into BRAT spans. The reported hyperparameters are 10 epochs, max sequence length 256, batch size 8, and learning rate $R = \frac{TP}{TP + FN}$8, trained on an NVIDIA GeForce RTX 3090 (24GB). No CRF layer is reported, and decoding is standard per-token classification over BIO tags (Danu et al., 20 Oct 2025).

The best test-set F1 values reported in that study are 77.88% for SDR, 92.09% for SMR, 91.74% for EMR, and 88.90% for IMR. The corresponding best systems are Cardio-SDR for SDR, MultiCardio-MMR for SMR and EMR, and Cardio-IMR for IMR. These exceed the cited BioASQ test leaderboard mean and median F1 values for all subtasks: 69.61% / 75.66% for SDR, 81.22% / 90.18% for SMR, 89.20% / 88.96% for EMR, and 82.80% / 87.76% for IMR (Danu et al., 20 Oct 2025).

The study also sharpens the task-level interpretation. Fine-tuning on CardioCCC development data improved Spanish disease recognition from 65.93% F1 for Clinical-SDR to 77.88% F1 for Cardio-SDR. Multilingual transfer improved Spanish medications from 91.65% to 92.09% F1 and English medications from 91.46% to 91.74% F1, while Italian remained slightly better under monolingual specialization, with Cardio-IMR at 88.90% versus 88.67% for the multilingual model. A development-to-test gap of 18.35% for Cardio-SDR and 16.3% for MultiCardio-SDR suggests overfitting in the disease subtask after fine-tuning on CardioCCC development data; the gaps for the medication subtasks were smaller, approximately 2–7% (Danu et al., 20 Oct 2025).

6. Interpretation, limitations, and relation to adjacent benchmarks

MultiCardioNER is often read primarily as a multilingual benchmark, but the official discussion emphasizes that specialty adaptation is equally central. Systems trained only on mixed-specialty corpora tended to achieve high precision but lower recall on cardiology-specific entities, whereas incorporating CardioCCC consistently improved recall and overall F1. At the same time, the overview explicitly leaves open whether the gains arise from domain adaptation, from increased dataset volume, or from both; disentangling those effects is identified as a topic for further study (Nentidis et al., 28 Aug 2025).

The task also exposes language asymmetries. Italian performance was modestly below Spanish and English, and the official discussion attributes this likely to fewer high-quality, domain-specific Italian models and resources. Teams mitigated this using Spanish-to-Italian adaptation, multilingual transformers such as XLM-R, or mixed Italian/Spanish training. The BERT-based study reaches a similar conclusion in a different form: multilingual training helped Spanish and English medication extraction, while Italian preserved a small advantage under monolingual cardiology specialization (Nentidis et al., 28 Aug 2025, Danu et al., 20 Oct 2025).

Several limitations are explicit. The task covers only Diseases in Spanish and Drugs in Spanish, English, and Italian; other entity types, such as signs and symptoms, procedures, and anatomy, are out of scope. It is confined to flat NER and does not tackle normalization or linking. The corpora consist of clinical case reports rather than broader EHR notes, so broader genre transfer may require additional adaptation. The overview does not provide fine-grained error taxonomies for phenomena such as boundary errors, abbreviations, or negation, although the implementation study notes that clinical texts contain specialized terminology, abbreviations, misspellings, and synonyms, with these factors especially pronounced in cardiology (Nentidis et al., 28 Aug 2025, Danu et al., 20 Oct 2025).

Within BioASQ 2024, the clearest neighboring task is BIONNE. MultiCardioNER and BIONNE are both biomedical NER benchmarks, but they target materially different problem settings. MultiCardioNER is flat NER on clinical case reports with two entity types and a cardiology focus in Spanish, English, and Italian. BIONNE targets nested NER on PubMed abstracts in English and Russian, with eight biomedical classes and explicit nested structures. MultiCardioNER therefore emphasizes specialty domain adaptation and multilingual transfer under strict exact-span evaluation, whereas BIONNE emphasizes nested structure and broader entity variety (Nentidis et al., 28 Aug 2025).

The task’s resources are publicly available on Zenodo, including the Gold Standard datasets, the background set, and the relevant annotation guidelines. A Silver Standard aggregating predictions on the 7,625-document background set is to be released in the same repository. Code availability is not provided in the BERT-based study, and the overview does not specify licensing details or evaluation scripts. Even so, the combination of public datasets, guideline documents, official evaluation, and detailed leaderboard results has made MultiCardioNER a well-defined reference point for multilingual clinical NER in cardiology (Nentidis et al., 28 Aug 2025, Danu et al., 20 Oct 2025).