Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Performance of Nine Large Language Models on the Nutrition Instructor Qualification Examination in China: Comparative Analysis
ABSTRACT
Background:
Large language models (LLMs) have shown considerable promise in medical education and professional examinations, yet their competence in nutrition science—a discipline integrating biochemical, physiological, and behavioral knowledge—remains largely unexplored. China's Nutrition Instructor Qualification Examination serves as the primary certification mechanism for the national nutrition workforce, and no study has evaluated whether current LLMs can pass this high-stakes examination.
Objective:
We aimed to systematically evaluate the performance of nine mainstream LLMs on the Nutrition Instructor Qualification Examination in China.
Methods:
A total of 377 questions (327 single-choice [SCQs] and 50 multiple-choice [MCQs]) were assembled from authentic examinations (2023–2024) and the designated preparation textbook. Nine LLMs (ERNIE 5.1, DeepSeek V4, Doubao-Seed-2.0, Hy3, Qwen3.7, SparkX 2.0, MiMo-V2.5-Pro, K 2.6, and ChatGPT 5.5) were accessed via their official web interfaces between July 6 and 9, 2026, under a standardized system prompt with reasoning modes disabled. Accuracy was scored dichotomously. Cochran's Q test and McNemar tests with Bonferroni correction (α'=0.0014) were used for pairwise comparisons, with Phi coefficients quantifying effect sizes. Item difficulty was classified into five levels, test-retest reliability was assessed in a 30-item subsample, and all incorrect responses were classified into Knowledge Gap, Reasoning Error, or Option Confusion.
Results:
On SCQs, DeepSeek V4 achieved the highest accuracy (94.5%), followed by three models at 94.2%, with eight of nine models forming a statistically homogeneous group; SparkX 2.0 performed lowest (86.2%). On MCQs, ChatGPT 5.5 ranked highest (92.0%), while SparkX 2.0 again performed lowest (56.0%). All models declined on MCQs relative to SCQs, ranging from 1.6 to 30.2 percentage points, and effect sizes in MCQ comparisons (φ=0.454–0.513) were approximately double those for SCQs (φ=0.193–0.237). Test-retest agreement was high (mean Cohen's κ=0.549). Error classification of all 298 incorrect answers showed Knowledge Gap was predominant (76.8%), followed by Reasoning Error (21.8%) and Option Confusion (1.3%).
Conclusions:
Most LLMs demonstrated high baseline nutrition knowledge competence, with eight of nine exceeding 92% on SCQs. However, the MCQ format revealed substantial latent differences, with ChatGPT 5.5 demonstrating superior multi-option reasoning. The predominance of knowledge gap errors identifies training data coverage—not reasoning architecture—as the primary target for improvement, supporting cautious integration of LLMs into nutrition education with continued human oversight.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.