JMIR Publications

ABSTRACT

Background:

Patient Priorities Care (PPC) is a model of care that align health care recommendations with priorities of older adults with multiple chronic conditions. Following identification of patient priorities, this information is documented in the patient’s electronic health record (EHR).

Objective:

Our goal is to develop and validate a natural language processing (NLP) model that reliably documents when clinicians identify patient priorities (i.e., values, outcome goals and care preferences) within the EHR as a measure of PPC adoption.

Methods:

Design: Retrospective analysis of unstructured EHR free-text notes using an NLP model. Setting: National Veteran Health Administration (VHA) EHR. Participants (including the sample size): 689 patient notes of 470 patients from encounters with 144 social workers Measurements: Each patient’s free-text clinical note was reviewed by two independent reviewers for the presence of PPC language such as priorities, values, and goals. We developed a hybrid NLP model that utilized the rule‐based and statistical machine learning approaches. The performance of the NLP model in training with 10-fold cross-validation and test is reported via precision, recall, F1, and accuracy in comparison to the chart review.

Results:

Out of 689 notes, 491 (71%) were identified as containing PPC language by reviewers (Kappa = 0.97, p-value < 0.001). The NLP model in the validation stage had a precision of 0.85, a recall of 0.90, an F1 of 0.87, and an accuracy of 0.91. The NLP model in the test stage has a precision of 0.94, a recall of 0.89, an F1 of 0.91, and an accuracy of 0.85.

Conclusions:

An automated NLP model can be used to reliably measure with high precision, recall, and accuracy when clinicians document patient priorities as a key step in the adoption of PPC.