JMIR Publications

ABSTRACT

Background:

Patient health and treatment information captured in structured and unstructured formats in computerized electronic health record repositories could potentially be used to augment the detection of safety signals for FDA-approved drug products. Natural language processing and other artificial intelligence techniques provide novel methodologies that could be leveraged to extract clinically useful information from such resources.

Objective:

Our aim is to develop a novel artificial intelligence (AI)-enabled software application to identify adverse drug event (ADE) safety signals from free text discharge summaries in electronic heath records (EHR) to enhance drug safety and research activities at the US Food and Drug Administration (FDA).

Methods:

A prototype web-based software application was developed that leverages keyword and trigger phrase searching with rule-base algorithms and deep learning to extract candidate ADEs for opioid drugs from discharge summaries in the Medical Information Mart for Intensive Care III (MIMIC III) database. It employs MedSpacy components to identify relevant sections of discharge summaries and a pretrained model, Spark NLP for Healthcare®, for named entity recognition. Fifteen FDA staff members participated in usability testing (UAT) to evaluate the application’s features, identify bugs, suggest design modifications, and provide insights on incorporating the software application among existing tools that support drug safety and research at the FDA.

Results:

Using this application, known labeled opioid-related adverse reactions were identified from EHR text. The AI-enabled model achieved accuracy/recall/precision/F1 scores of 0.66/0.69/0.64/0.67, respectively. The UAT participants assessed the software as highly desirable in user satisfaction, visualizations, and potential to support drug safety signal detection from EHR data while saving time and manual effort. Actionable design recommendations included: (a) Enlarge the tabs and visualizations; (b) Enable more flexibility and customizations to fit end users’ individual needs; (c) Provide additional instructional resources; (d) Add multiple graph export functionality; and (e) Add project summaries.

Conclusions:

This novel software application employs innovative AI-based techniques to automate searching, extracting, and analyzing clinically useful information captured in unstructured text in EHRs. It increases efficiency in harnessing real-world data for drug safety and increases the usability of the data to support regulatory review while decreasing the manual research burden