JMIR Publications

ABSTRACT

Background:

Although Large Language Models (LLMs) show great promises in processing medical text, they are prone to generating incorrect information, commonly referred to as hallucinations. These inaccuracies present a significant risk for clinical applications where precision is critical. Additionally, relying on human experts to review LLM-generated content to ensure accuracy is costly and time-consuming, which sets a barrier against large-scale deployment of LLMs in healthcare settings.

Objective:

The primary objective of this study is to develop an automatic Artificial Intelligence (AI) system capable of extracting structured information from unstructured medical data and employing advanced reasoning techniques to support reliable clinical decision making. A key aspect of this objective is ensuring that the system incorporates self-verification mechanisms, enabling it to assess the accuracy and reliability of its own outputs. By integrating such mechanisms, we aim to enhance the system’s robustness, reduce reliance on human intervention, and improve the overall trustworthiness of AI-driven medical summarization and evaluation.

Methods:

The proposed framework comprises two layers: a summarization layer and an evaluation layer. The summarization layer employs Llama2-70B and Mistral-7B models to generate concise summaries from unstructured medical data, focusing on tasks such as consumer health question summarization, biomedical answer summarization, and dialog summarization. The evaluation layer uses GPT-4-turbo as a judge, leveraging pairwise comparison strategies and different prompt strategies to evaluate summaries across four dimensions: coherence, consistency, fluency, and relevance. To validate the framework, we compare the judgments generated by the LLMs in the evaluation layer with those provided by medical experts, offering valuable insights into the alignment and reliability of AI-driven evaluations within the medical domain. We also explore a way to handle disagreement among human experts and discuss our methodology in addressing diversity in human perspectives.

Results:

The study found variability in expert consensus, with average Agreement Rates (ARs) of 19.2% among all experts and 51.6% among groups of three experts. GPT-4 demonstrated alignment with expert judgments, achieving an average AR of 78.44% with at least one expert and comparable performance in cross-validation tests. The enhanced guidance in prompt design improved GPT-4’s alignment with expert evaluations, highlighting the importance of effective prompt engineering in auto-evaluation of summarization tasks.

Conclusions:

This study highlights the potential of LLMs as reliable tools for medical summarization and evaluation, reducing the dependency on human experts. The proposed framework demonstrates scalability and adaptability for clinical applications while addressing key challenges like hallucination and position bias.