JMIR Publications

ABSTRACT

Background:

In the critical domain of healthcare, the efficacy of medical decision-making and diagnostic accuracy is essential for managing medical conditions effectively. Large Language Models (LLMs) have emerged as sophisticated AI systems that can autonomously generate responses to inquiries and engage in interactive dialogues. Their potential to enhance various facets of healthcare, especially in diagnostics, has garnered significant interest. However, there is a noticeable absence of comprehensive analysis examining the evolution and analytical appraisal of LLMs in medical diagnosis.

Objective:

This scoping review aims to provide an overview of the current state of research regarding the use of LLMs in medical diagnostics. The study seeks to answer three primary subquestions: (1) Which LLMs are commonly used and how are they assessed in diagnosis? (2) What is the current performance of LLMs in diagnosing diseases? (3) Which medical domains are currently investigating the application of LLMs?

Methods:

This scoping review was conducted according to the JBI Manual for evidence synthesis and adheres to the PRISMA extension for scoping reviews (PRISMA-ScR). Relevant literature was searched from Web of Science, PubMed, Embase, IEEE Xplore, and ACM Digital Library databases from 2022 to 2025. Articles were screened and selected based on predefined inclusion and exclusion criteria. Bibliometric analysis was performed using VoSviewer to identify major research clusters and trends. Data extraction included details on LLM types, application domains, and performance metrics.

Results:

A total of 95 articles were included in the final review. The bibliometric analysis identified three major clusters: (1) The assessment of LLMs in medical diagnosis, (2) The application of LLMs in medical diagnosis, and (3) The impacts of utilizing LLMs in medical diagnosis. GPT-4 and its variants were the most commonly used LLMs, with 73.7% of studies employing them. LLMs showed significant potential in disease classification, medical question answering, and generating high-quality diagnostic content. However, concerns regarding bias and ethical issues were highlighted. The application of LLMs was most prevalent in radiology (17.1%) and psychiatry, with promising results in improving diagnostic accuracy and facilitating clinical decision-making.

Conclusions:

LLMs, particularly GPT-4 and its variants, have shown substantial promise in enhancing diagnostic accuracy and facilitating clinical decision-making across various medical specialties. However, critical challenges remain, including the potential for bias and the complexity of real-world clinical scenarios. Future research should focus on refining LLMs to address these challenges, with a particular emphasis on eliminating bias and protecting patient privacy. Practical implementation studies evaluating the impact of LLMs on patient outcomes and healthcare workflow are also needed.