JMIR Publications

ABSTRACT

Background:

Chronic pain is a leading cause of disability and requires multidimensional assessment of pain intensity and functioning, yet electronic health records (EHRs) rarely capture these measures systematically. By contrast, surveys collecting patient-reported outcomes can assess pain over multiple dimensions but remain resource-intensive and difficult to scale for continuous population-level monitoring.

Objective:

The objective of this study is to develop and validate a domain-informed natural language processing (NLP) framework to derive pain severity and functional interference outcomes from unstructured clinical narratives. Using a benchmark dataset of 3,726 Veterans with longitudinal survey measures, we aim to demonstrate that NLP-derived outcomes can serve as a reliable, scalable surrogate for resource-intensive patient-reported surveys.

Methods:

This study utilizes a retrospective cohort of 3,726 Veterans with chronic musculoskeletal pain initiating Complementary and Integrative Health (CIH) therapies across 18 Veterans Health Administration (VA) Whole Health Flagship sites (2021–2023). The dataset encompasses longitudinal patient-reported outcome surveys serving as the benchmark, linked with both structured and unstructured data from the VA EHR. To guide extraction and enable scalable processing, we developed a seed lexicon and annotation guidelines based on established psychometric instruments and input from subject matter experts (SMEs). Currently, SMEs are annotating clinical notes in iterative batches. Upon completion, a large language model (LLM) will annotate additional notes. These annotations will be used to fine-tune a lightweight language model capable of processing the entire cohort. The study employs a three-stage validation process, assessing: (1) documentation completeness; (2) inference accuracy, evaluating agreement between model outputs and SME annotations at both the LLM and fine-tuned model levels; and (3) concordance with patient-reported outcomes captured independently from the EHR.

Results:

To date, a cohort of 3,726 Veterans has been identified. The longitudinal survey data have been linked to clinical notes corresponding to the survey time period. The seed lexicon and annotation guidelines have been developed. Annotation is underway in iterative batches to drive the subsequent LLM adaptation and three-stage validation.

Conclusions:

This protocol outlines a framework for identifying severe pain interference from clinical narratives, addressing a critical gap in health care system surveillance. To our knowledge, this is the first study to validate clinical text-based pain outcome extraction against patient-reported outcomes in a nationwide longitudinal cohort. If successful, this approach will enable health care systems to continuously monitor pain-related functional interference and support more holistic, patient-centered pain management at scale.