Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating Artificial Intelligence Models in Dermatology: A Comparative Analysis
ABSTRACT
Background:
Dermatology is an ever-evolving field; the integration of artificial intelligence (AI) offers promising avenues for enhancing clinical decision-making and improving patient outcomes. This study focuses on evaluating the effectiveness of DermGPT, an AI model specifically designed to address dermatological inquiries with precision, compared to ChatGPT-4o (OpenAI), an AI chatbot and virtual assistant developed by OpenAI. DermGPT was developed by a practicing dermatologist, and it is marketed as a superior alternative to general-purpose AI models like ChatGPT; it contains a curated database of authoritative sources tailored to the community of dermatological professionals.
Objective:
To assess the validity of these claims, we conducted a comparative analysis between ChatGPT and DermGPT.
Methods:
We conducted a survey study. A series of 15 dermatology-specific questions were presented to both AI tools, with the responses subsequently evaluated by a cohort of 19+ practicing dermatologists (attendings, residents and fellows) across two institutions. The dermatologists, who were blinded to the origin of the answers, rated the quality and accuracy of the responses in the form of a survey. Preferences for answers and comparison of sources references were evaluated, and the efficacy of the sources of data were compared. Statistical analysis was performed using SAS OnDemand. Univariable analysis was performed for categorical variables via chi-square tests, with statistical significance set at p<0.05.
Results:
A total of 19 attendings, dermatology residents and fellows completed the survey. For answer preference, DermGPT responses were favored overall (48%) compared to ChatGPT (28%), with a statistically significant chi-square test result (p=0.039). In subgroup analysis, attendings preferred DermGPT (47%) over ChatGPT (28%), while residents showed a narrower preference margin (32% vs. 31%). For source preference, ChatGPT citations were favored (46%) over DermGPT (24%), though the chi-square test did not reach statistical significance (p=1.385). Attendings preferred ChatGPT sources (48%) over DermGPT (23%), while residents also favored ChatGPT (41%) over DermGPT (24%).
Conclusions:
DermGPT demonstrated strong potential for improving answer clarity and conciseness in dermatology-related queries, while ChatGPT provided more robust source citations, enhancing trust in evidence-based responses. Notably, ChatGPT’s citation quality was preferred, whereas DermGPT’s concise style made it more accessible for quick clinical reference. These findings suggest that integrating the strengths of both models could optimize AI-assisted medical consultations, balancing clarity with academic rigor. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.