Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Nov 6, 2025
Date Accepted: Jun 12, 2026
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Generating Question Prompt Lists from EHR Data Using Large Language Models: An Iterative Evaluation Study
ABSTRACT
Background:
Patients frequently access laboratory results through patient portals, but many struggle to interpret these values and formulate relevant questions for their clinicians. Question Prompt Lists (QPLs) can enhance communication but are rarely tailored to individual clinical contexts.
Objective:
This study evaluated the feasibility of using large language models (LLMs) to generate patient-friendly, clinically relevant questions grounded in electronic health record (EHR) laboratory data.
Methods:
We extracted de-identified clinical profiles, including laboratory results, diagnoses, and medications, from patients with chronic conditions (e.g., diabetes, chronic kidney disease). Using nine de-identified clinical profiles from OneFlorida Data Trust, we used GPT-4o and LLaMA 3.2 to generate 300 questions (20 per profile) across three iterative rounds, with refinements informed by clinician ratings consisting of two binary questions (i.e., clear phrasing, clinical validity), and three Likert-scale questions (i.e., clinical significance, appropriateness, and willingness to answer). Refinements were incorporated after each round. Patient participants then evaluated selected questions for understandability, perceived usefulness, and intention to use. Readability was assessed with standard indices.
Results:
Iterative clinician feedback improved question clarity and reduced clinically irrelevant suggestions. Across rounds, GPT-4o consistently produced more coherent and patient-friendly questions, while LLaMA 3.2 demonstrated competitive performance with careful prompt design. In Round 3, mean binary ratings were near 1.0 for both models (p = 0.5914), and Likert ratings were similar (3.75 vs. 4.25, p = 0.2510). Questions generated by GPT-4o (Flesch–Kincaid 6.8 ± 1.8; 139 words) were more readable than those generated by LLaMA 3.2 (9.3 ± 3.5; 173 words). Patient evaluation (N = 30) showed 25 out of 30 questions with moderate to high understandability (average Likert value ≥ 3.5) and 19 out of 30 questions with moderate to high usefulness (average Likert value ≥ 3.5).
Conclusions:
This study demonstrates the feasibility of integrating LLMs with structured EHR data to produce contextualized QPLs. A clinician-in-the-loop process was critical for ensuring accuracy and actionability.
Citation
