JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly deployed as conversational agents for medical communication, but their ability to adapt responses to a patient’s emotional state and literacy level remains insufficiently characterized. In patient-facing scenarios, effective communication requires not only accurate medical content, but also appropriate tone, clarity, and responsiveness to user vulnerability.

Objective:

This study aims to evaluate the ability of contemporary LLMs to adapt medical responses to different emotional sensitivity and literacy levels, and to assess their consistency in both generating and interpreting such adaptations.

Methods:

A controlled benchmark is designed to evaluate how four LLMs adapt medical responses to predefined emotional and literacy levels for a realistic patient profile asking an open-ended question. To establish a robust performance baseline, one model is locally deployed (Gemma 3 27B), and three are cloud-based (Claude Opus 4.5, Gemini 3 Pro, and GPT 5.2). Each model is tested across all combinations of five emotional levels, three literacy levels, and three prompt variants of verbosity, yielding 45 responses per model. Generated responses are assessed against human-reviewed ground truth for emotional and literacy alignment. A separate self-evaluation procedure measures each model’s ability to interpret its own outputs using the same adaptation framework. The study also examines generation-interpretation consistency and quantitative text properties, with particular attention to response length and verbosity.

Results:

Literacy adaptation was consistently accurate across models, often achieving near-zero error, indicating strong capability in adjusting explanatory complexity. In contrast, emotional adaptation showed greater variability and was more sensitive to model architecture and prompt design. Larger cloud-based models demonstrated more stable performance overall, while the local model exhibited higher variability, particularly under shorter prompts. A systematic asymmetry was observed between generation and interpretation: models frequently misclassified low emotional and literacy levels as “normal”, suggesting limitations in recognizing user needs. Quantitative analysis revealed that response length increased with both emotional sensitivity and literacy level, with prompt verbosity and model-specific tendencies acting as secondary factors.

Conclusions:

Current LLMs can support adaptive medical communication, but their performance is uneven across dimensions, especially in terms of emotional generation and interpretation. For healthcare applications, these systems should be treated as assistive tools under human oversight rather than autonomous communicators, with careful consideration of ethical, privacy, and safety implications in sensitive health contexts.