JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models (2409.13317v1)

Abstract: Recent developments in Japanese LLMs primarily focus on general domains, with fewer advancements in Japanese biomedical LLMs. One obstacle is the absence of a comprehensive, large-scale benchmark for comparison. Furthermore, the resources for evaluating Japanese biomedical LLMs are insufficient. To advance this field, we propose a new benchmark including eight LLMs across four categories and 20 Japanese biomedical datasets across five tasks. Experimental results indicate that: (1) LLMs with a better understanding of Japanese and richer biomedical knowledge achieve better performance in Japanese biomedical tasks, (2) LLMs that are not mainly designed for Japanese biomedical domains can still perform unexpectedly well, and (3) there is still much room for improving the existing LLMs in certain Japanese biomedical tasks. Moreover, we offer insights that could further enhance development in this field. Our evaluation tools tailored to our benchmark as well as the datasets are publicly available in https://huggingface.co/datasets/Coldog2333/JMedBench to facilitate future research.

• The paper presents JMedBench, a robust benchmark that evaluates Japanese biomedical LLMs across five core NLP tasks using 20 diverse datasets.
• It demonstrates that models with specialized training in Japanese biomedical data outperform general-purpose LLMs, particularly in zero-shot and few-shot settings.
• The study highlights that effective prompt design and dataset translation are critical to improving model accuracy in specialized biomedical applications.

An In-depth Review of "JMedBench: A Benchmark for Evaluating Japanese Biomedical LLMs"

The paper "JMedBench: A Benchmark for Evaluating Japanese Biomedical LLMs" presents a significant contribution to the field of NLP by addressing a notable gap in the development and benchmarking of Japanese-language biomedical LLMs. This paper offers a comprehensive evaluation framework, inclusive of various domains and tasks, and sets a precedent for subsequent research and model development in this specialized area.

Overview

The primary aim of this paper is to provide a robust, large-scale benchmark tailored for the evaluation of Japanese biomedical LLMs. While previous advancements in LLMs have predominantly emphasized general domains and English biomedical tasks, there has been a relative neglect in the Japanese biomedical field. The lack of a domain-specific benchmark has made it challenging to systematically assess and improve Japanese biomedical NLP models.

Datasets and Benchmark Construction

The authors construct the JMedBench benchmark, which includes a diverse set of 20 datasets across five core NLP tasks:

1. Multi-choice Question Answering (MCQA): Encompasses datasets like IgakuQA and JMMLU-medical, as well as translations of prominent English datasets such as MedQA, USMLE-QA, and PubMedQA.
2. Machine Translation (MT): The EJMMT dataset is used to evaluate the translation capabilities from English to Japanese and vice versa, focusing on biomedical texts.
3. Named Entity Recognition (NER): Includes datasets such as MRNER-Disease, MRNER-Medicine, and translations of BC2GM, BC5Chem, BC5Disease, JNLPBA, and NCBI-Disease.
4. Document Classification (DC): Involves datasets from the JMED-LLM project, such as CRADE, RRTNM, and SMDIS.
5. Semantic Text Similarity (STS): Uses the JCSTS dataset, primarily designed for evaluating semantic similarity in the biomedical context.

The authors argue that many existing Japanese biomedical datasets are relatively small and focused; hence, they augment their benchmark by translating larger English datasets to ensure robust evaluation. This extensive dataset collection helps in mitigating the variability often encountered in small-scale evaluations and provides a comprehensive framework for evaluating the effectiveness of Japanese biomedical LLMs.

Evaluation and Results

The benchmark evaluates eight representative models across four categories:

• General LLMs (e.g., Llama2, Llama3, Qwen2, Mistral)
• Biomedical LLMs in other languages (e.g., Meditron)
• General Japanese LLMs (e.g., LLM-jp, SwallowLM)
• Japanese Biomedical LLMs (e.g., MMed-Llama3)

Key findings from the paper include:

1. Domain-Specific Knowledge: LLMs with a better understanding of Japanese and richer biomedical knowledge tend to perform better in Japanese biomedical tasks.
2. Unexpected Proficiency: LLMs not specifically designed for Japanese biomedical domains can still perform relatively well.
3. Need for Improvement: There exists considerable room for enhancing the current LLMs on specific Japanese biomedical tasks, indicating the necessity for ongoing research and development.

The paper also highlights the importance of the prompt design, showing that the type and structure of prompts can significantly influence model performance. The evaluation protocol includes zero-shot and few-shot settings, the latter being facilitated by demonstrations to the model, thus showcasing the models' in-context learning (ICL) abilities.

Implications and Future Directions

The release of JMedBench lays a strong foundation for future Japanese biomedical NLP research. Practically, this benchmark enables more standardized and rigorous comparisons across models, facilitating the identification of weaknesses and guiding the development of more robust models. Theoretically, the insights gained from this benchmark can spur innovations in multi-lingual and domain-specific model training, transfer learning, and continual learning.

Conclusion

The JMedBench paper offers an essential contribution to the field of Japanese biomedical NLP by providing a comprehensive, scalable benchmark and a detailed evaluation of current LLMs. By addressing previous inadequacies and proposing a robust benchmarking methodology, this work promotes the advancement of Japanese biomedical LLMs, potentially leading to significant improvements in healthcare and medical research.

The authors have made their datasets and evaluation tools publicly available, thereby encouraging collaboration and furthering progress in this specialized arena. Future work could extend this benchmark to include more natural language generation (NLG) tasks, improve multilingual evaluation, and ensure that advances in Japanese biomedical LLMs can be seamlessly integrated into practical, real-world applications.