JMIR Publications

ABSTRACT

Background:

Messages to patients are more accessible when written at a 4th to 6th grade level. There has been relatively little work on the potential of large language models (LLMs) to summarize continuous glucose monitor (CGM) data to generate simpler patient messages.

Objective:

Generate simpler patient messages using LLMs.

Methods:

We used data from 8 publicly available datasets, including adults and youth, to evaluate LLM-generated dot phrases (pre-defined message templates). We engineered prompts that specified the desired reading level, word count, and dot phrase requirements. We used ChatGPT-4o, Perplexity, and Claude 3.5 Sonnet to generate messages based on time in range and time below range from 8 T1D datasets. Messages were considered incomplete if they did not include all required elements, such as clinical targets and patient time in range.

Results:

We generated 648 messages using LLM. The LLM-generated dot phrases had lower mean Flesch–Kincaid grade levels than the original clinician-generated dot phrases (6.0 ±3.2 vs 7.7). Compared to the original clinician-generated dot phrase, five of our nine prompts had significantly lower grade levels (all p<0.001). LLM-generated dot phrases were longer than the original clinician-generated dot phrase by an average of 18 words. There was a significant difference in dot phrase length between clinician-generated and LLM-generated dot phrases across all LLMs and prompts (all p<0.004). Nearly half (48.6%) of all LLM-generated messages did not include all required elements, although explicitly stating message requirements in the prompt meaningfully increased completeness.

Conclusions:

LLMs demonstrated the potential to simplify messages summarizing CGM data and make management recommendations. Explicitly stating desired message length, reading levels, and required elements in the prompt increases completeness and readability.