JMIR Publications

ABSTRACT

Background:

Patient satisfaction feedback is crucial for hospital service quality, but manual reviews are time-consuming, and traditional natural language processing methods remain inadequate. Large Language Models (LLMs) show promise but are prone to extrinsic faithfulness hallucinations—fabricated or illogical outputs that limit their reliability in healthcare.

Objective:

This study aimed to evaluate the Self-Logical Consistency Assessment (SLCA), an original method designed to enhance LLM feedback classification reliability by enforcing a logically-structured chain of thought.

Methods:

SLCA uses two validation steps: self-consistency (identifying the most coherent response) and logical consistency (ensuring alignment with the original statement and expert classifications). We evaluated SLCA using GPT-4 and Llama-3.1 405B on 12,600 classifications from 100 patient feedback samples to assess hallucinations, and tested its performance on a 49,140-classification benchmark derived from 1,170 feedbacks.

Results:

SLCA reduced hallucinations among detected categories from 15.80% (168/1063) to 0.51% (4/786) with GPT-4 and from 7.17% (51/711) to 1.67% (10/599) with Llama-3.1, with residual errors confined to the emergency feedback category. On the benchmark, SLCA achieved precision-recall scores of 0.86-0.78 for GPT-4 and 0.84-0.58 for Llama-3.1. These results demonstrate SLCA’s ability to achieve human-level performance across LLMs.

Conclusions:

SLCA offers a scalable, explainable solution for improving LLM classification reliability in healthcare. Its capacity to enhance performance without fine-tuning or additional training data positions it as a valuable tool for analyzing patient feedback and supporting hospital service quality improvement.