JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) may enhance AI-assisted shared decision support by translating behavioral and contextual information into complementary patient- and clinician-facing messages. However, it remains unclear whether the perceived quality of such communication differs across prompting strategies in cannabis-related care.

Objective:

This study compared direct generation, standard Chain-of-Thought (CoT), and Multi-Perspective CoT (Multi-CoT) across multiple LLM backbones with respect to expert selections for patient helpfulness, clinical expertise, empathy, and overall preference in simulated cannabis-related shared decision support scenarios.

Methods:

We developed an AI-assisted shared decision support prototype that integrates behavioral context, community-experience retrieval, emotion-aware prompting, explainable artificial intelligence (XAI) summaries, and role-specific response generation. Four blinded evaluators with diverse health care–related backgrounds each assessed 50 simulated cases, including 20 shared and 30 evaluator-specific cases, yielding 200 case-level assessments. Each evaluator was assigned a distinct pair of LLMs. For each case and evaluation criterion, evaluators selected 1 of 6 response pairs, each comprising a patient-facing message and a clinician-facing summary generated by the 2 assigned LLMs under the 3 prompting strategies. Results were summarized descriptively using counts and percentages.

Results:

Across 200 case-level assessments, Multi-CoT received the most selections for overall preference (122/200, 61.0%) and patient helpfulness (100/200, 50.0%). Multi-CoT and standard CoT received the same number of selections for clinical expertise (83/200, 41.5% each), whereas standard CoT received the most selections for empathy (95/200, 47.5%). Selection patterns varied substantially across evaluator–model pairings; Multi-CoT’s overall-preference selection rate ranged from 12% to 92% across the 4 evaluations.

Conclusions:

In this simulation-based expert evaluation, Multi-CoT was favored in aggregate for patient helpfulness and overall preference but not for empathy, and its selection patterns varied markedly across evaluator–model pairings. These findings support Multi-CoT as a promising, context-dependent prompting approach for the response-generation layer of AI-assisted shared decision support. They do not establish clinical effectiveness, improved shared decision-making, or readiness for clinical use. Further work should include multi-stakeholder evaluation, real patient–clinician interactions, and formal assessments of factual accuracy and safety.