Using a large language model chatbot for allergen avoidance counseling in contact dermatitis: An evaluation of validity and limitations
ABSTRACT
Background:
Artificial intelligence (AI), particularly large language models (LLMs) such as ChatGPT, is increasingly being explored for its potential role in supporting clinical decision-making. In dermatology, LLMs may assist with patient counseling, including allergen avoidance recommendations for individuals with contact dermatitis. However, their usefulness in recommending allergen-free products remains underexplored.
Objective:
This study aimed to evaluate the potential use of ChatGPT in recommending allergen-free products for the management of contact allergens.
Methods:
Thirty allergen groups from the international standard patch test series were assessed. For each allergen, ChatGPT was prompted to recommend five allergen-free products using three question formats of varying complexity. Responses were included if they specified a product. Dermatologists manually classified each response into one of five categories: valid, invalid, cross-reaction, unknown, or unresponsive.
Results:
ChatGPT generated valid product recommendations in 79.07% of responses. Invalid responses accounted for 1.78%, cross-reactive recommendations for 2.13%, unknown responses for 8.67%, and unresponsive outputs for 8.36%. The simplest question format produced the highest accuracy, with 85.33% valid responses. ChatGPT performed best for cosmetics and medications, with valid response rates of 90.48% and 82.5%, respectively. Accuracy was lower for sensitive allergens, including fragrance mixes and para-phenylenediamine (PPD). Frequent cross-reactive recommendations were observed, particularly involving Peru balsam.
Conclusions:
LLM-based chatbots such as ChatGPT may have potential as supportive tools for allergen-avoidance counseling in contact dermatitis. However, their performance varies by allergen type and prompt complexity, and further refinement is needed before they can be reliably used in clinical counseling.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.