JMIR Publications

ABSTRACT

Background:

Nurse-patient telephone outreach (NPTO) is integral to chronic respiratory disease management, directly promoting patient-centered communication and continuity of care. However, studies assessing NPTO have reported inconsistent effects on patient satisfaction and outcomes, likely reflecting the absence of standardized quality evaluation methods.

Objective:

By leveraging advances in natural language processing, this study aims to develop and validate a large language model (LLM) capable of systematically evaluating NPTO quality with accuracy comparable to or exceeding human evaluators.

Methods:

We analyzed 351 NPTO calls from five outpatient pulmonary clinics over a 12-month period. A reference standard was established by three evaluators independently rating 51 calls with discrepancies resolved through consensus adjudication. Of these consensus-adjudicated calls, 40 were used to train an LLM and calibrate an additional human evaluator. The remaining 11 calls were independently assessed by the LLM and the human evaluator. Model performance was benchmarked against the reference standard using binary and ordinal classification metrics and regression error metrics. Agreement between the LLM, external evaluator, and reference standard was compared.

Results:

The LLM achieved an overall classification accuracy of 87.9% and outperformed the external evaluator, achieving higher agreement with gold-standard in both binary quality ratings (accuracy 96.2% vs. 86.4%) and ordinal quality ratings (accuracy 84.3% vs. 62.8%). Precision/recall balance was superior for the LLM across rating types. Regression analyses showed closer approximation to the gold standard by the LLM (MAE = 0.13) compared with the external evaluator (MAE = 0.28).

Conclusions:

By achieving high accuracy across quality domains and approximating gold-standard ratings with minimal error, this model demonstrated a scalable and reproducible framework for NPTO quality assessment, surpassing human evaluation in both binary and ordinal tasks. While performance was strongest in domains such as communication behaviors and safety-netting, areas like professionalism and medical history review remain opportunities for refinement. Future work should focus on expanding training datasets and prospectively validating model performance across diverse clinical settings. These findings highlight the potential for LLMs to standardize call quality evaluation and support continuous improvement in patient-centered care delivery.