JMIR Publications

ABSTRACT

Background:

Clinical progress notes are generated for patients during each healthcare encounter and include information on current disease symptoms and rehabilitation progress. These notes provide a rich data source for the application of natural language processing (NLP) methods to identify patients experiencing successful treatment outcomes. However, to use NLP methods to develop an outcome measure that is reflective of both physician and patient perspectives, there needs to be an agreed upon set of patient experiences that reflect treatment success.

Objective:

The objective of this paper was to assess if physicians and patients agree on whether patient experiences captured in clinical progress notes reflect a successful patient outcome following orthopaedic treatment.

Methods:

We performed a cross-sectional analysis of a subset of clinical notes for patients presenting to a Level-1 Trauma Center Regional Health System for follow-up for an acute Proximal Humerus Fracture (PHF). Physicians and patients reviewed a sample of 100 clinical notes and labeled each note as reflecting one of four treatment outcome labels (treatment success, improvement, deterioration, or failure). The four outcome levels were then aggregated into a binary classifier indicating treatment success or failure, and agreement between physicians and patient labels were assessed. Descriptive analyses were performed to report characteristics of the sample. Cohen’s Kappa statistics were used to assess the degree of agreement between physician and patient evaluators.

Results:

The average age of the patients in the sample was 67 years of age (SD= 13.6) and 82% of the notes came from female patients. Patients were primarily white (91%) and had Medicare insurance coverage (65%). The note sample came from fracture-related encounters ranging from the second to the 10th encounter after the PHF visit. There were no significant differences in patient or visit characteristics across concordant and discordant notes. Physician and patient evaluators exhibited a fair level of agreement in what they deemed as treatment success based on a Cohen’s Kappa of 0.32 ((95% CI, 0.10 to .55), p = .01).

Conclusions:

Findings suggest that physicians and patients only demonstrated fair agreement when interpreting whether a patient had achieved treatment success following treatment for an acute shoulder fracture. Further research is needed to examine how different perceptions of treatment success may influence the development of NLP-based outcome measures.