JMIR Publications

ABSTRACT

Background:

Inadequate sleep is prevalent among athletes, affecting adaption to training and performance. While education on factors influencing sleep can improve sleep behaviors, Large Language Models (LLMs) may offer a scalable approach to provide sleep education to athletes.

Objective:

This study aims to i) investigate the quality of sleep recommendations generated by publicly available LLMs, as evaluated by experienced raters, and ii) determine whether recommendations quality varies with information input granularity.

Methods:

Two prompts with differing information input granularity (low and high) were created for two use cases and inserted into ChatGPT-4o (GPT-4o) and Google Gemini, resulting in n=8 different recommendations. Experienced raters (n=13) evaluated the recommendations on a 1-5 Likert-scale, based on n=10 sleep criteria derived from recent literature.

Results:

The highest summary rating was achieved by GPT-4o using high input information granularity, with n=8 ratings >3 (tendency towards good), n=3 ratings equal 3 (neutral), n=2 ratings <3 (tendency towards bad). GPT-4o significantly outperformed Google Gemini in 9 out of 10 criteria (P<.001 to P=.045). Recommendations generated with high input granularity received significantly higher ratings than those with low granularity across both LLMs and use cases (P<0.001 to P=.049).

Conclusions:

Both LLMs exhibit limitations, neglecting vital criteria of sleep education. Sleep recommendations by GPT-4o and Google Gemini were evaluated as suboptimal, with GPT-4o achieving higher overall quality. However, both LLMs demonstrated improved recommendation quality with higher information input granularity, emphasizing the need for specificity and a thorough review of outputs to securely implement AI technologies into sleep education.