JMIR Publications

ABSTRACT

Background:

Electronic health records (EHR) are a rich source of longitudinal patient data. However, missing information due to clinical care that predates the implementation of EHR system(s) or care that occurred at different medical institutions impedes complete ascertainment of a patient’s medical history.

Objective:

This study aimed to quantify information gaps by comparing the gynecological surgical history extracted from an EHR of a single institution using NLP techniques with the manually curated surgical history information through chart review of records from multiple independent regional healthcare institutions.

Methods:

We developed a rule-based NLP algorithm to detect gynecological surgery history from the unstructured narrative of the Mayo Clinic EHR. These results were compared to a gold standard cohort of 3870 women with gynecological surgery status adjudicated using the Rochester Epidemiology Project medical records-linkage system. We quantified and characterized the information gaps observed that led to misclassification of surgical status.

Results:

The NLP algorithm achieved 0.76 recall, 0.79 precision, and 0.76 F1-score in the test set (n = 265). We applied the algorithm to the validation set (n = 3340) and identified two types of information gaps through error analysis. First, 6% of women in this study had no recorded surgery information or partial information in the EHR. Second, 4% of women had inconsistent or inaccurate information within the clinical narrative due to misinterpreted information, erroneous “copy and paste,” or incorrect information provided by patients. Additionally, the NLP algorithm misclassified the surgery status of 4% of women.

Conclusions:

NLP techniques were able to adequately recreate the gynecologic surgical status from the clinical narrative. However, missing or inaccurately reported and/or recorded information resulted in much of the misclassification observed. Therefore, alternative approaches to collect or curate surgical history are needed.