JMIR Publications

ABSTRACT

Background:

Patients frequently access laboratory results through patient portals, but many struggle to interpret these values and formulate relevant questions for their clinicians. Question Prompt Lists (QPLs) can enhance communication but are rarely tailored to individual clinical contexts.

Objective:

This study evaluated the feasibility of using large language models (LLMs) to generate patient-friendly, clinically relevant questions grounded in electronic health record (EHR) laboratory data.

Methods:

We extracted de-identified clinical profiles, including laboratory results, diagnoses, and medications, from patients with chronic conditions (e.g., diabetes, chronic kidney disease). Using nine de-identified clinical profiles from OneFlorida Data Trust, we used GPT-4o and LLaMA 3.2 to generate 300 questions (20 per profile) across three iterative rounds, with refinements informed by clinician ratings consisting of two binary questions (i.e., clear phrasing, clinical validity), and three Likert-scale questions (i.e., clinical significance, appropriateness, and willingness to answer). Refinements were incorporated after each round. Patient participants then evaluated selected questions for understandability, perceived usefulness, and intention to use. Readability was assessed with standard indices.

Results:

Iterative clinician feedback improved question clarity and reduced clinically irrelevant suggestions. Across rounds, GPT-4o consistently produced more coherent and patient-friendly questions, while LLaMA 3.2 demonstrated competitive performance on Likert-scale metrics, it exhibited greater variability in clinical appropriateness as noted by clinicians. In Round 3, binary clinician ratings were near ceiling for both models (means ≥0.97), while Likert-scale evaluations consistently favored LLaMA 3.2 across all three clinicians for likelihood of being asked in primary care (3.37–4.82 vs 3.02–4.67), perceived significance to patient health (3.38–4.28 vs 2.97–3.83), and willingness to answer (3.17–4.70 vs 2.82–4.47), with multiple comparisons reaching statistical significance after Bonferroni correction. Patient evaluation (N = 134) showed 25 out of 30 questions with moderate to high understandability (average Likert value ≥ 3.5) and 19 out of 30 questions with moderate to high usefulness (average Likert value ≥ 3.5).

Conclusions:

This study supports the feasibility of using LLMs with structured EHR-derived laboratory data to generate contextualized QPLs, but model outputs varied in clinical appropriateness and readability. Clinician-in-the-loop review remains necessary before patient-facing use.