Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Currently submitted to: JMIR Dermatology

Date Submitted: Nov 2, 2026
Open Peer Review Period: Nov 17, 2026 - Jan 12, 2027
(currently open for review)

Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.

Evaluating Large Language Model Diagnostic Accuracy in Dermatology: A Comparative Analysis of ChatGPT, Gemini, and Claude Sonnet

  • Ideen Tayebi; 
  • Sapna Thaker; 
  • Derek Nguyen; 
  • Karan Thaker; 
  • Madhu Varma

ABSTRACT

Background:

Patients increasingly consult large language models (LLMs) for health information, and some may use these tools to evaluate skin lesions in place of specialist care. However, the ability of these models to distinguish benign from malignant skin lesions remains uncertain, raising concerns about their potential to delay appropriate diagnosis and treatment.

Objective:

This study compared the diagnostic accuracy of GPT-4.0, Gemini 2.5 Pro, and Claude Sonnet 4.5 in classifying benign and malignant skin lesions from dermoscopic images.

Methods:

Histologically confirmed dermoscopic images of basal cell carcinoma (BCC), melanoma, and benign keratosis-like lesions (BKL) were selected from the HAM10000 dataset. Each model received the same 100 images per lesion type (n = 300) in the same order, with a standardized prompt requesting a single diagnosis. Responses were scored against the histologic diagnosis and categorized as benign or malignant to determine missed cancers, sensitivity, and specificity. Accuracy was compared among models using Cochran's Q test with pairwise McNemar tests and Bonferroni correction. Agreement among the three models was also examined for each image.

Results:

Overall accuracy was below 50% for all three models (GPT-4.0, 42.3%; Gemini 2.5 Pro, 40.3%; Claude Sonnet 4.5, 46.0%) and did not differ significantly among them (p = 0.226). Within each lesion type, however, the models differed significantly, and no single model performed best across all three. Gemini 2.5 Pro had the highest accuracy for BCC (50%), Claude Sonnet 4.5 for melanoma (80%), and GPT-4.0 for BKL (37%). GPT-4.0 and Gemini 2.5 Pro labeled more than one-third of melanomas as benign (35% and 37%, respectively), compared with 12% for Claude Sonnet 4.5 (p < 0.001). Claude Sonnet 4.5 identified the most malignant lesions as malignant (88.5%) but also labeled the most benign lesions as malignant (64%). Many errors were shared across models. All three misclassified the same 33% of BCC images and 52% of BKL images, and melanoma was the most common incorrect diagnosis for both.

Conclusions:

All three models correctly classified fewer than half of the lesions evaluated, and no model was consistently superior across lesion types. Errors were frequently shared rather than specific to one model, with melanoma the most common incorrect diagnosis for both BCC and BKL. These findings do not support the use of consumer LLMs for independent evaluation of skin lesions, and clinicians should continue to recommend assessment by a board-certified dermatologist for concerning lesions.


 Citation

Please cite as:

Tayebi I, Thaker S, Nguyen D, Thaker K, Varma M

Evaluating Large Language Model Diagnostic Accuracy in Dermatology: A Comparative Analysis of ChatGPT, Gemini, and Claude Sonnet

JMIR Preprints. 02/11/2026:115999

DOI: 10.2196/preprints.115999

URL: https://preprints.jmir.org/preprint/115999

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.