JMIR Publications

ABSTRACT

Background:

Extracting adverse drug event (ADE) information from unstructured clinical notes is challenging due to the semantic complexity. While large language models (LLMs) show promise, they are prone to “hallucinations”.

Objective:

This study aims to evaluate the effectiveness of retrieval-augmented generation (RAG) in improving the identification of ADEs using LLMs from Chinese clinical narratives and to establish a paradigm for this task.

Methods:

We collected and preprocessed 19,983 Chinese clinical notes, from which we created a gold-standard reference set (n=2,510) and an ADE knowledge base (n=5,144). Three LLMs (DeepSeek-V3, Ernie 3.5-8K, and GPT-4o) were evaluated using three distinct prompt strategies: none-augmented generation (NAG), static-augmented generation (SAG), and RAG. Performance was rigorously assessed using precision, recall, and F1 score via 10-fold cross-validation.

Results:

We successfully constructed and publicly released the first Chinese ADE corpus from clinical notes. The RAG framework significantly improved the performance across all tested LLMs compared to NAG and SAG. The optimal configuration, DeepSeek-V3 with RAG, achieved an impressive overall F1 score of 0.9614. Notably, RAG dramatically increased the recall rate for models such as GPT-4o (from 64.76% to 92.39%) and effectively resolved common identification errors that were intractable for non-augmented models.

Conclusions:

Integrating a curated knowledge base with LLMs via a RAG framework is a highly effective strategy for accurately identifying ADEs in unstructured Chinese clinical notes. This approach provides a robust technical foundation for pharmacovigilance and holds significant potential to enhance drug safety research and clinical practice.