JMIR Publications

ABSTRACT

Inadequate post-care patient education contributes to preventable readmissions and adverse outcomes that disproportionately affect patients from medically complex, high-need communities. Large language models (LLMs) show significant promise for generating personalized, plain-language patient education at scale. However, existing LLM evaluation frameworks prioritize technical accuracy over patient accessibility and health literacy alignment, and few, if any, explicitly account for the attitudinal influences that clinician evaluators may introduce into the rating process. This paper introduces the Technology Acceptance Model and Medical Condition Regard Scale (TAM-MCRS) LLM Evaluation Framework, a novel clinician-centered approach designed to compare which large language models produce the highest-quality post-care patient education across accuracy, appropriateness, clarity, and completeness. LLM-generated patient education outputs will be evaluated by an expert clinician panel using AIM-AHEAD-informed clinical vignettes, while accounting for measured evaluator attitudinal variance. The framework was developed through the National Institutes of Health AIM-AHEAD CLINAQ fellowship program in partnership with Ochsner Health and Xavier University of Louisiana. The TAM-MCRS framework integrates two theoretical lenses. The Technology Acceptance Model (TAM) maps Perceived Usefulness onto accuracy and completeness criteria, and Perceived Ease of Use onto clarity and appropriateness criteria. This framework produces a four-criterion evaluation matrix scored using FActScore, G-Eval, Flesch-Kincaid grade level, and the Patient Education Materials Assessment Tool (PEMAT). The Medical Condition Regard Scale (MCRS) is administered as a post-scoring attitudinal covariate survey to examine whether clinician attitudinal regard for stigmatized patient conditions systematically influences rubric ratings. Three LLMs representing distinct design architectures serve as comparison targets: a frontier general-purpose benchmark model, a safety-aligned frontier model, and an open-weight medical domain-trained model. A standardized Master Prompt Template is applied uniformly across all models at temperature zero to reduce sampling variability and support reproducibility. The TAM-MCRS framework addresses three specific gaps in LLM evaluation for patient education: to our knowledge, no published study has examined whether medical domain-trained models outperform frontier general-purpose models for patient-facing post-care education; to our knowledge, no existing evaluation framework integrates TAM constructs with MCRS as an attitudinal covariate; and to our knowledge, no published study accounts for measured evaluator attitudinal variance toward stigmatized conditions in clinician panel LLM ratings. By integrating TAM and MCRS, the framework produces evaluation evidence that is objective, theoretically grounded, clinically realistic, and population-responsive. Implications for clinician informaticists, health system governance, and responsible AI deployment are discussed.