JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly used by the public for health information, yet their reliability for individualized nutrition counseling remains uncertain.

Objective:

This study aims to benchmark five freely available LLM families across four clinically relevant nutrition scenarios, evaluating their performance on accuracy, personalization, comprehensiveness, user-friendliness, safety, and overall quality.

Methods:

In a repeated-measures expert evaluation design, standardized personas representing obesity, osteoporosis, inflammatory bowel disease, and endometriosis were used to generate nutrition consultations from ChatGPT, Gemini, DeepSeek, LeChat, and LLama in both 2025 and 2026 versions. Ten independent experts rated each output on a 1–5 Likert-type rubric, yielding 400 evaluations in total. Agreement was assessed with ICCs and Kendall’s W. The model comparisons used Friedman and paired Wilcoxon tests, with supplementary linear mixed-effects models adjusting for expert and persona clustering.

Results:

Model performance showed a stable hierarchy across analyses: ChatGPT and Gemini formed the top tier, DeepSeek occupied an intermediate position, consistently followed by LeChat, whereas LLama performed worst. Aggregated inter-rater reliability was excellent across all criteria, although individual-rater agreement was only fair to moderate. Between 2025 and 2026, most models improved, with the largest gains observed for ChatGPT and DeepSeek. Personalization remained the weakest domain, and all models showed systematic Western-centric dietary defaults, limiting cultural and contextual fit.

Conclusions:

Contemporary LLMs can generate broadly useful nutrition advice under benchmark conditions, but they remain insufficiently reliable for autonomous individualized counseling. Their main limitations are incomplete personalization, variable safety, and weak cultural adaptation, supporting a cautious role as adjunct tools rather than replacements for qualified nutrition professionals.