JMIR Publications

ABSTRACT

Background:

Americans are increasingly using artificial intelligence (AI) chatbots for health information and advice, yet most have not undergone US Food and Drug Administration (FDA) review. Under the FDA’s software as a medical device framework, a key criterion for requiring review is whether a tool is intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease.

Objective:

This study aimed to quantify how often consumer-facing AI chatbot responses provide diagnostic or treatment content meeting FDA criteria for software as a medical device, and how often they include disclaimers describing their intended use or limitations.

Methods:

We conducted a cross-sectional content analysis in July 2026 of 24 publicly accessible AI chatbots using 3 standardized clinical vignettes (migraine, stye, and potential depression). Responses were captured verbatim and coded using an a priori codebook for diagnostic content, treatment content, and disclaimer language. Reporting followed the Chatbot Assessment Reporting Tool (CHART) guideline.

Results:

Across 72 tool-vignette instances, 66 (91.7%) provided diagnostic information, including a primary diagnosis (15/72, 20.8%), a possible diagnosis (20/72, 27.8%), or a differential diagnosis (31/72, 43.1%). Treatment content appeared in 51 (70.8%) instances, including prescription medication recommendations in 36 (50.0%). Disclaimers were present in 55 (76.4%) instances; the most common stated that the tool does not provide a diagnosis or medical advice (39/72, 54.2%).

Conclusions:

Despite frequent disclaimers, consumer-facing AI chatbots routinely provided diagnostic and treatment guidance, meeting a key FDA criterion for software as a medical device. These findings point to a regulatory gap and raise the question of whether widespread medical use could establish objective intent for a medical purpose.