JMIR Publications

ABSTRACT

Background:

Consumer-facing large language model (LLM) platforms are increasingly used for medication questions. Aggregate medical accuracy benchmarks, however, may miss product-specific storage and device rules that determine patient actions.

Objective:

To evaluate product-specific action concordance, safety-critical concordance, robustness to misleading patient premises, near-neighbor product-rule discrimination, and short-term repeatability across four consumer LLM platforms in psoriasis biologic self-management.

Methods:

We created 32 neutral and 16 paired misleading prompts for COSENTYX, TALTZ, BIMZELX, and TREMFYA using frozen US Food and Drug Administration–approved prescribing information and instructions for use. ChatGPT, Gemini, Claude, and DeepSeek were queried in three planned rounds using standardized consumer-interface settings. Two platform-label–masked dermatology raters independently scored eligible outputs, and a senior dermatologist adjudicated disagreements. The primary endpoint was scenario-weighted product-specific action concordance (PSAC).

Results:

Of 576 planned outputs, 572 were eligible. Neutral PSAC was 98.96% for ChatGPT, 100% for Gemini, 90.63% for Claude, and 88.54% for DeepSeek; none of the six prespecified pairwise comparisons remained significant after Holm correction. Safety-critical concordance was 98.44%. Product-rule discrimination and repeatability were more heterogeneous. Complete discrimination of all five contrastive blocks occurred for Gemini and four of five for ChatGPT, but for none of the five for Claude or DeepSeek; overall all-three agreement among complete repeated-query cells was 89.89%. Misleading premises did not reduce PSAC overall (+2.08 percentage points, 95% bootstrap CI -0.52 to +4.69; exact P=.31), with premise-induced safety failure in 4/177 eligible matched instances (2.26%).

Conclusions:

Consumer LLM platforms showed high aggregate and safety-critical concordance in this fixed benchmark, while contrastive product-rule testing and repeated queries identified weaknesses that were less apparent in aggregate concordance. Patient-facing LLM evaluation should test exact product/device actions, near-neighbor rule contrasts, misleading premises, and repeated behavior rather than relying on aggregate concordance alone.