JMIR Publications

ABSTRACT

Background:

The Coronavirus Disease 2019 (COVID-19) pandemic highlighted the importance of timely infectious disease surveillance.

Objective:

We aimed to develop a natural language processing algorithm to extract structured information on symptoms and vaccination history from free-text clinical notes in Japanese primary care. This approach enables low-latency monitoring using real-world electronic health record data.

Methods:

A total of 773 clinical notes, originating from 526 unique patients, were provided by M3 Inc. through the Japan Medical Data Survey and used for analysis. Three clinicians annotated information related to infectious disease symptoms and vaccination history. The data were divided into 622 training cases and 151 evaluation cases, ensuring no patient overlap between the two sets. We developed a rule-based extraction algorithm using clinician-designed regular expressions and conducted few-shot learning and fine-tuning with both open-source and commercial large language models (LLMs). We compared the extraction accuracy (F1 score) of these approaches.

Results:

Generally, the format of clinical notes tends to be similar across clinicians, allowing rule-based algorithms to achieve reasonable extraction accuracy. LLMs using few-shot prompts could extract unstructured information, such as vaccination history, with high accuracy, achieving high performance across all evaluation metrics (Anthropic Claude 3.5 Sonnet, macro-average F1 score 0.902). We also demonstrated that a fine-tuned small open-source LLM can achieve remarkably higher accuracy (Google Gemma 2 2B, macro-average F1 score 0.932) than commercial models.

Conclusions:

We showed that a fine-tuned LLM can accurately extract and structure infectious disease-related information from free-text clinical notes, which implies the feasibility of digital surveillance using electronic medical record information.