Improving Clinical NLP Performance through Language Model-Generated Synthetic Clinical Data (2403.19511v1)

Abstract: Generative models have been showing potential for producing data in mass. This study explores the enhancement of clinical natural language processing performance by utilizing synthetic data generated from advanced LLMs. Promising results show feasible applications in such a high-stakes domain.

• The paper introduces a novel method to generate synthetic clinical text using LLMs combined with a label correction step.
• By augmenting real datasets with refined synthetic data, the approach achieved competitive and sometimes superior performance on clinical NLP tasks.
• The study demonstrates that integrating LLM-generated data with expert-annotated data can reduce the reliance on costly manual annotations.

Enhancing Clinical NLP with LLM-Generated Synthetic Clinical Data

Introduction

The generation of large, annotated datasets is a critical bottleneck in clinical NLP development, hindered by time-consuming annotation processes requiring expert knowledge, patient privacy concerns, and data governance issues. This paper presents a novel approach using LLMs to generate synthetic annotated clinical text datasets to overcome these challenges. By incorporating a unique label correction step to improve the quality of these synthetic datasets, the authors demonstrate improved performance on clinical NLP tasks when these datasets are used for training models.

Methods

The paper focuses on both curated clinical benchmark tasks from the DR.BENCH dataset, developed using the MIMIC III dataset, and a real-world, long-document clinical task involving the detection of esophagitis severity from cancer patient notes. Two versions of the Llama-2 LLM, with varying parameter sizes, were employed for generating synthetic data across several datasets. An active learning step termed "label correction" was introduced to refine the quality of the generated synthetic data labels. The research evaluated the utility of synthetic data in various configurations: standalone, in conjunction with real datasets (augmentation), and in scenarios without label corrections. Performance evaluations were based on the comparison of models trained under these different setups.

Results

The paper reports a pronounced decline in model performance when relying solely on synthetic data without label correction. However, with the application of label correction, models trained on synthetic data, both solely and in augmentation with real datasets, displayed competitive, and in some instances, superior performance compared to models trained on gold-standard datasets alone. Notably, in the Assessment and Plan (A/P) Reasoning task, the use of augmented synthetic data surpassed the performance achieved by the gold-standard dataset. This trend held across different tasks, demonstrating the effectiveness of synthetic data in enhancing model performance. The application to the esophagitis grading task in a clinical setting further validated the potential of synthetic data to maintain high performance levels, with the best outcomes observed when synthetic and real data were used in combination.

Discussion

The findings underscore the potential of LLMs in generating synthetic data that can match or exceed the performance on clinical NLP benchmarks and real-world tasks, especially when used to augment real data. This could potentially alleviate the demand for large, annotated clinical datasets, facilitating broader applications in biomedical research and clinical care. The introduction of the label corrector step represents a significant advancement in synthetic text generation, yielding substantial performance improvements over methods relying solely on in-context learning.

Conclusion

This research highlights the substantial promise of utilizing synthetic data generated by LLMs in advancing clinical NLP tasks. The combination of synthetic with real annotated datasets offers a practical solution to the challenges of data scarcity and the intensive requirement for expert annotation in the field. Looking forward, the paper paves the way for future investigations into improving synthetic data generation, label correction techniques, and the exploration of synthetic data sharing across institutions to foster advancements in clinical NLP.

Implications and Future Directions

The practical applications of this research are vast, extending to reducing dependence on extensive clinical data collections, easing privacy and data governance concerns, and lowering the barrier for entry into clinical NLP research. Future directions may include establishing benchmarks using synthetic data, multi-institutional collaborations to validate findings, and further exploration into the biases that synthetic data may introduce. This paper sets a precedent for the potential role of synthetic data in clinical NLP, offering a glimpse into a future where data limitations are less of a constraint on the advancement of medical informatics.