JMIR Publications

ABSTRACT

Background:

Novel glucagon-like peptide 1 receptor agonists (GLP1RAs) for obesity treatment have generated much dialogue on digital media platforms. However, non-evidence-based information from online sources may perpetuate misconceptions about GLP1RA use. A promising new digital avenue for patient education is large language models (LLMs), which could potentially be used as an alternative to clarify questions about GLP1RA therapy.

Objective:

This study compared LLM (ChatGPT 4o) and internet (Google) search responses to simulated questions about GLP1RA therapy.

Methods:

Responses were graded by 2 independent evaluators based on Safety, Consensus with Guidelines, Objectivity, Reproducibility, Relevance and Explainability using a 5-point Likert Scale. Mean scores were compared using independent T-test. Qualitative observations were recorded.

Results:

LLM responses had significantly higher mean scores than Internet responses in the "objectivity" (3.91 ± 0.63 vs 3.36 ± 0.80, p=0.038) and "reproducibility" (3.85 ± 0.49 vs 3.00 ± 0.97, p=0.007) categories. There was no significant difference in the mean scores in "safety", "consensus", “relevance” and 'explainability". However, LLM responses lacked updated information pertaining to more contemporary concerns surrounding GLP1RA use such as the impact on fertility and mental health.

Conclusions:

The study highlights the importance of healthcare provider communication, as both LLM and internet searches have limitations and may perpetuate misconceptions about GLP1RAs.