JMIR Publications

ABSTRACT

Background:

Patient-reported survey data represent an underutilized source of information for identifying diagnostic breakdowns. Traditional approaches rely on manual review, which is resource-intensive and difficult to scale. Recent advances in natural language processing (NLP), including transformer-based models and large language models (LLMs), offer new opportunities to automate this task; however, their comparative performance and practical applicability in real-world clinical datasets remain unclear.

Objective:

To evaluate the performance of progressively advanced NLP approaches for identifying diagnostic breakdowns in patient survey comments and to assess the value of prompt-based LLMs with expert-informed iterative refinement.

Methods:

We analyzed 523 free-text comments from ambulatory patient surveys. Diagnostic breakdowns were defined using a structured framework encompassing failures in diagnostic accuracy, timeliness, or communication. We implemented a three-phase modeling pipeline: (1) a baseline TF-IDF feature representation with XGBoost classification, (2) a fine-tuned transformer model based on a pretrained Twitter RoBERTa architecture, and (3) a prompted LLM approach using in-context learning with iterative prompt refinement. Prompt development followed a human-in-the-loop process, incorporating domain expert feedback to refine definitions, examples, and decision boundaries. Model performance was evaluated using cross-validation with standard classification metrics, including precision, recall, F1 score, area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve (AUPRC).

Results:

The baseline TF-IDF model demonstrated limited performance, with challenges in capturing contextual nuance and high false-positive rates. The fine-tuned transformer improved overall discrimination but remained sensitive to ambiguous expressions of dissatisfaction. The prompted LLM approach achieved the best balance between sensitivity and specificity following iterative refinement. Initial prompt versions exhibited high recall but low precision due to over-classification of general complaints as diagnostic breakdowns. Subsequent refinements, informed by expert input, substantially improved precision while maintaining strong recall, resulting in the highest overall F1-score among the three approaches. Qualitative analysis indicated improved alignment with clinically meaningful definitions, particularly in borderline cases.

Conclusions:

Prompted LLMs, when combined with expert-guided iterative refinement, outperform traditional machine learning and fine-tuned transformer models for identifying diagnostic breakdowns in patient-reported text. This approach enables scalable, interpretable, and clinically aligned analysis of patient feedback. Future work should focus on external validation across institutions and integration into real-time quality improvement workflows.