Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Benchmark Evaluation of Clinical Named Entity Recognition in French (2403.19726v1)

Published 28 Mar 2024 in cs.CL, cs.AI, and q-bio.QM

Abstract: Background: Transformer-based LLMs have shown strong performance on many Natural LanguageProcessing (NLP) tasks. Masked LLMs (MLMs) attract sustained interest because they can be adaptedto different languages and sub-domains through training or fine-tuning on specific corpora while remaining lighterthan modern LLMs. Recently, several MLMs have been released for the biomedicaldomain in French, and experiments suggest that they outperform standard French counterparts. However, nosystematic evaluation comparing all models on the same corpora is available. Objective: This paper presentsan evaluation of masked LLMs for biomedical French on the task of clinical named entity recognition.Material and methods: We evaluate biomedical models CamemBERT-bio and DrBERT and compare them tostandard French models CamemBERT, FlauBERT and FrALBERT as well as multilingual mBERT using three publicallyavailable corpora for clinical named entity recognition in French. The evaluation set-up relies on gold-standardcorpora as released by the corpus developers. Results: Results suggest that CamemBERT-bio outperformsDrBERT consistently while FlauBERT offers competitive performance and FrAlBERT achieves the lowest carbonfootprint. Conclusion: This is the first benchmark evaluation of biomedical masked LLMs for Frenchclinical entity recognition that compares model performance consistently on nested entity recognition using metricscovering performance and environmental impact.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. In ICML Workshop on "Challenges in Deploying and monitoring Machine Learning Systems".
  2. AliBERT: A pre-trained language model for French biomedical text. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 223–236, Toronto, Canada. Association for Computational Linguistics.
  3. Présentation de la campagne d’évaluation DEFT 2020 : similarité textuelle en domaine ouvert et extraction d’information précise dans des cas cliniques (presentation of the DEFT 2020 challenge : open domain textual similarity and precise information extraction from clinical cases ). In Actes de la 6e conférence conjointe Journées d’Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Atelier DÉfi Fouille de Textes, pages 1–13, Nancy, France. ATALA et AFCP.
  4. On the usability of transformers-based models for a French question-answering task. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 244–255, Held Online. INCOMA Ltd.
  5. Contextualized French language models for biomedical named entity recognition. In Actes de la 6e conférence conjointe Journées d’Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Atelier DÉfi Fouille de Textes, pages 36–48, Nancy, France. ATALA et AFCP.
  6. A Modern Introduction to Probability and Statistics: Understanding Why and How. SPRINGER NATURE.
  7. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  8. Cyril Grouin. 2016. Controlled propagation of concept annotations in textual corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4075–4079, Portorož, Slovenia. European Language Resources Association (ELRA).
  9. Finding mentions of abbreviations and their definitions in spanish clinical cases: The barr2 shared task evaluation results. IberEval@ SEPLN, 2150:280–289.
  10. The georges pompidou university hospital clinical data warehouse: a 8-years follow-up experience. International journal of medical informatics, 102:21–28.
  11. DrBERT: A robust pre-trained model in French for biomedical and clinical domains. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16207–16221, Toronto, Canada. Association for Computational Linguistics.
  12. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  13. FlauBERT: Unsupervised language model pre-training for French. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2479–2490, Marseille, France. European Language Resources Association.
  14. Stratégies d’adaptation pour la reconnaissance d’entités médicales en français (adaptation strategies for biomedical named entity recognition in French). In Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principale, pages 215–225, Avignon, France. ATALA.
  15. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  16. ehop clinical data warehouse: From a prototype to the creation of an inter-regional clinical data centers network. Studies in health technology and informatics, 264:1536–1537.
  17. Automatic de-identification of medical texts in spanish: the meddocan track, corpus, guidelines, methods and evaluation of results. In IberLEF@ SEPLN, pages 618–638.
  18. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online. Association for Computational Linguistics.
  19. Aggregating umls semantic types for reducing conceptual complexity. Studies in health technology and informatics, 84(0 1):216.
  20. Clef ehealth evaluation lab 2015 task 1b: Clinical named entity recognition. In CLEF (Working Notes).
  21. Evaluation of doc’eds: a french semantic search tool to query health documents from a clinical data warehouse. BMC medical informatics and decision making, 22(1):34.
  22. Brat: A web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL ’12, page 102–107, USA. Association for Computational Linguistics.
  23. Camembert-bio: Un modèle de langue français savoureux et meilleur pour la santé.
  24. Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association, 14(5):550–563.
  25. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556.
  26. Erasmus mc at clef ehealth 2016: Concept recognition and coding in french texts. In CEUR Workshop Proceedings, pages 171–178.
  27. Annotating the biomedical literature for the human variome. Database, 2013:bat019.
  28. Perceval Wajsbürt. 2021. Extraction and normalization of simple and structured entities in medical documents. Theses, Sorbonne Université.
  29. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  30. Natalia Grabar and Rémi Cardon. 2018. CLEAR – simple corpus for medical French. In Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA), pages 3–9, Tilburg, the Netherlands. Association for Computational Linguistics.
  31. CAS: French corpus with clinical cases. In Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis, pages 122–128, Brussels, Belgium. Association for Computational Linguistics.
  32. The e3c project: European clinical case corpus. Language, 1(L2):L3.
  33. The quaero french medical corpus: A ressource for medical entity recognition and normalization. Proc of BioTextMining Work, pages 24–30.
  34. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Nesrine Bannour (1 paper)
  2. Christophe Servan (16 papers)
  3. Aurélie Névéol (10 papers)
  4. Xavier Tannier (16 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets