JMIR Publications

ABSTRACT

Background:

The use of patient reported outcome measures (PROMs) is an expected component of high-quality, measurement-based chiropractic care. The largest healthcare system offering integrated chiropractic care is the Veterans Health Administration (VHA). Challenges limit monitoring PROM use as a care quality metric at a national scale in the VHA. Structured data are unavailable with PROMs often embedded within clinic text notes as unstructured data requiring time-intensive, peer-conducted chart review for evaluation. Natural language processing (NLP) of clinic text notes is one promising solution to extracting care quality data from unstructured text.

Objective:

Our objective was to test NLP approaches to identify PROMs documented in VHA chiropractic text notes.

Methods:

VHA chiropractic notes from 10/1/2017 to 9/30/2020 were obtained from the VHA Musculoskeletal Diagnosis/Complementary and Integrative Health Cohort. A rule-based NLP model built using medspaCy and SpaCy was evaluated on text matching and note categorization tasks. SpaCy was used to build bag-of-words, convoluted neural network, and ensemble models for note categorization. Performance metrics for each model and task included precision (P), recall (R), F-measure (F). Cross-validation was used to validate performance metric estimates for the statistical and machine learning models.

Results:

Our sample included 377,213 visit notes from 56,628 patients. The rule-based model performance was good for soft-boundary text-matching (P=81.1%, R=96.7%, F=88.2%) and excellent for note categorization (P=90.3%, R=99.5%, F=94.7%). Cross-validation performance of the statistical and machine learning models for the note categorization task was very good overall, but lower than rule-based model performance. Overall prevalence of PROM documentation was low (17%).

Conclusions:

We evaluated multiple NLP methods across a series of tasks, with optimal performance achieved using a rule-based method. By leveraging NLP approaches, we can overcome the challenges posed by unstructured clinical text notes to track documented PROM use. Overall documented use of PROMs in chiropractic notes was low and highlights a potential for quality improvement. This work represents a methodological advancement in the identification and monitoring of documented use of PROMs to ensure consistent, high-quality chiropractic care for Veterans.