JMIR Publications

ABSTRACT

Background:

Generative artificial intelligence (GenAI) is increasingly used to produce patient-friendly clinical documentation, yet evaluation of these outputs remains inconsistent and difficult to scale. Patient-friendliness is commonly reduced to narrow readability metrics, such as Flesch-Kincaid grade level, without accounting for clinical accuracy, completeness, or the patient perspective. No standardized framework exists to evaluate the quality and safety of AI-generated patient-friendly documentation across document types or the full documentation lifecycle.

Objective:

To develop and preliminarily validate CLEAR (Clinical Language Evaluation and AI Documentation Review), a theoretically grounded evaluation framework for AI-generated patient-friendly clinical documentation across the generation, review, and monitoring stages of the AI documentation lifecycle.

Methods:

CLEAR was developed using Messick's validity framework across four stages: content validation, response process, internal structure, and consequences. Domains were identified through a targeted literature review and reviewed by a panel of six clinical and operational experts. An iterative, consensus-based process involving four board-certified internists across 10 rounds refined domain definitions and scoring instructions. Inter-rater reliability was assessed on 50 AI-generated patient-friendly discharge summaries using Cohen's kappa and Gwet's AC1 for binary domains and intraclass correlation coefficients (ICC) and Gwet's AC2 for continuous domains. Additionally, 19 semi-structured stakeholder interviews with clinicians, informaticists, institutional leaders, and patient education experts explored operational needs and implementation contexts.

Results:

CLEAR comprises five domains for evaluating patient-friendly AI documentation: readability, understandability, patient-centeredness, accuracy, and completeness. Inter-rater reliability was good to almost perfect across all subjectively scored domains per Gwet's agreement coefficients. Stakeholder interviews independently identified three operational gaps aligned with the CLEAR lifecycle: lack of structured guidance for prompt engineering, subjectivity in human review, and absence of scalable monitoring infrastructure, directly validating the framework's real-world relevance. CLEAR was applied across three illustrative implementation contexts: prompt engineering for patient-friendly echocardiogram reports, structured human review of discharge summaries, and development of LLM-as-judge automated monitoring tools.

Conclusions:

CLEAR provides a preliminarily validated evaluation framework designed to span the full AI documentation lifecycle, from prompt engineering through human review to automated monitoring. By conceptualizing patient-friendliness as a multidimensional construct that integrates communication quality with patient safety, CLEAR offers practical infrastructure for consistent and scalable governance of patient-facing AI documentation in healthcare systems.