JMIR Publications

ABSTRACT

Background:

To improve pharmacotherapy, patients’ oral expressions serve as valuable sources of clinical information. Natural language processing (NLP) is a useful approach for analyzing unstructured text data, such as patient narratives. However, few studies have focused on using NLP for narratives in the Japanese language.

Objective:

To develop a high-performance NLP system for extracting clinical information from patient narratives, we examined the performance progression as the amount of training data was gradually increased.

Methods:

Subjective texts from the pharmaceutical care records of Keio University Hospital from April 1, 2018 to March 31, 2019, comprising 12,004 records from 6,559 cases, were used. After preprocessing, we annotated diseases and symptoms within the texts. We then trained and evaluated deep learning models—bidirectional encoder representations from transformers combined with a conditional random field (BERT-CRF)—by 10-fold cross-validation. The annotated data were divided into 10 subsets, and the amount of training data was progressively increased over 10 steps. We also analyzed the causes of errors. Finally, we applied the developed system to the analysis of case report texts to evaluate its usability for texts from other sources.

Results:

The F1-score of the system improved from 0.67 to 0.82 as the amount of training data increased from 1,200 to 12,004 records. The F1-score reached 0.78 with 3,600 records and largely saturated thereafter. As performance improved, errors from incorrect extractions decreased significantly, increasing precision. For case reports, the F1-score also increased from 0.34 to 0.41 as the training dataset expanded from 1,200 to 12,004 records. Performance was lower for extracting symptoms from case report texts compared with pharmaceutical care records, suggesting that this system is more specialized for analyzing subjective data from pharmaceutical care records.

Conclusions:

We successfully developed a high-performance system specialized in analyzing subjective data from pharmaceutical care records by training a large dataset, with near-complete saturation of system performance with about 3,600 training records. This system will be useful for monitoring symptoms, offering benefits for both clinical practice and research.