Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes (2403.05795v1)

Published 9 Mar 2024 in cs.CL

Abstract: The advancement of NLP systems in healthcare hinges on LLM ability to interpret the intricate information contained within clinical notes. This process often requires integrating information from various time points in a patient's medical history. However, most earlier clinical LLMs were pretrained with a context length limited to roughly one clinical document. In this study, We introduce ClinicalMamba, a specialized version of the Mamba LLM, pretrained on a vast corpus of longitudinal clinical notes to address the unique linguistic characteristics and information processing needs of the medical domain. ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a superior performance in modeling clinical language across extended text lengths compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba achieves notable benchmarks in speed and accuracy, outperforming existing clinical LLMs and general domain large models like GPT-4 in longitudinal clinical notes information extraction tasks.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  2. David Blumenthal. 2010. Launching hitech. The New England journal of medicine, 362 5:382–5.
  3. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  4. Language models are few-shot learners for prognostic prediction. ArXiv, abs/2302.12692.
  5. Localtweets to localfealth: A mental health surveillance framework based on twitter data.
  6. K. Eva. 2005. What every teacher needs to know about clinical reasoning. Medical Education, 39.
  7. Clinical concept extraction: A methodology review. Journal of biomedical informatics, page 103526.
  8. Units: Building a unified time series model.
  9. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816–3830, Online. Association for Computational Linguistics.
  10. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv, abs/2312.00752.
  11. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685.
  12. PLM-ICD: Automatic ICD coding with pretrained language models. In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 10–20, Seattle, WA. Association for Computational Linguistics.
  13. Clinicalbert: Modeling clinical notes and predicting hospital readmission. ArXiv, abs/1904.05342.
  14. Domain-specific continued pretraining of language models for capturing long context in mental health. ArXiv, abs/2304.10447.
  15. Document-level n-ary relation extraction with multiscale representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3693–3704, Minneapolis, Minnesota. Association for Computational Linguistics.
  16. Health system-scale language models are all-purpose prediction engines. Nature, 619:357 – 362.
  17. Mimic-iii, a freely accessible critical care database. Scientific Data, 3.
  18. Pretraining to recognize pico elements from randomized controlled trial literature. Studies in health technology and informatics, 264:188 – 192.
  19. Publicly shareable clinical large language model built on synthetic clinical notes. ArXiv, abs/2309.00237.
  20. MedJEx: A medical jargon extraction model with Wiki’s hyperlink span and contextualized masked language model score. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11733–11751, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  21. Do we still need clinical language models? ArXiv, abs/2302.08091.
  22. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  23. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 146–157, Online. Association for Computational Linguistics.
  24. Fine-tuning bidirectional encoder representations from transformers (bert)–based models on large-scale electronic health record notes: An empirical study. JMIR Med Inform, 7(3):e14830.
  25. Fei Li and Hong Yu. 2020. Icd coding from clinical text using multi-filter residual convolutional neural network. Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 34 5:8180–8187.
  26. Clinical-longformer and clinical-bigbird: Transformers for long clinical sequences. ArXiv, abs/2201.11838.
  27. ClinicalT5: A generative language model for clinical text. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5436–5443, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  28. Scaling data-constrained language models. ArXiv, abs/2305.16264.
  29. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1101–1111, New Orleans, Louisiana. Association for Computational Linguistics.
  30. Longbox: Evaluating transformers on long-sequence clinical tasks. ArXiv, abs/2311.09564.
  31. A study of generative large language model for medical research and healthcare. NPJ Digital Medicine, 6.
  32. Long range arena: A benchmark for efficient transformers. ArXiv, abs/2011.04006.
  33. Clinical prompt learning with frozen language models. IEEE Transactions on Neural Networks and Learning Systems, pages 1–11.
  34. Recent advances in predictive modeling with electronic health records.
  35. Are large language models ready for healthcare? a comparative study on clinical language understanding.
  36. Zifeng Wang and Jimeng Sun. 2022. PromptEHR: Conditional electronic healthcare records generation with prompt learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2873–2885, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  37. Clinical concept extraction for document-level coding. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 261–272, Florence, Italy. Association for Computational Linguistics.
  38. Zero-shot clinical trial patient matching with llms. ArXiv, abs/2402.05125.
  39. Deep learning in clinical natural language processing: a methodical review. Journal of the American Medical Informatics Association : JAMIA.
  40. A large language model for electronic health records. NPJ Digital Medicine, 5.
  41. Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. Nature Communications, 14.
  42. Knowledge injected prompt based fine-tuning for multi-label few-shot ICD coding. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1767–1781, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  43. Ning Zhang and Maciej Jankowski. 2022. Hierarchical bert for medical document understanding. ArXiv, abs/2204.09600.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zhichao Yang (37 papers)
  2. Avijit Mitra (7 papers)
  3. Sunjae Kwon (16 papers)
  4. Hong Yu (114 papers)
Citations (12)
View on Semantic Scholar