Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Oct 22, 2024
Date Accepted: Mar 31, 2025
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Comparing AI-Generated and Clinician-Created Personalized Self-Management Guidance for Knee Osteoarthritis Patients: A Blinded Observational Study
ABSTRACT
Background:
Background:
Personalized education is crucial for effective knee osteoarthritis (OA) management, but providing it remains challenging due to imbalanced patient-provider ratio and limited resources. This study explores the potential of GPT-4, a large language model, in generating tailored self-management guidance and compares its performance with physician-provided advice.
Objective:
Objective：This study aims to evaluate the effectiveness of GPT-4 in generating personalized education materials for patients with knee osteoarthritis (OA) and compare it with experienced clinicians. Specifically, the comparison is made in terms of efficiency, readability, accuracy, personalization, comprehensiveness, and safety. By leveraging patient data from previous trials, it is evaluated whether AI can improve the quality and accuracy of patient education and evaluate its potential in improving patient care and outcomes.
Methods:
Methods:
A two-phase, blinded, observational study was conducted using patient data from a previous trial. In phase one, two experienced orthopedic specialists created personalized education materials. In phase two, the same data were input into GPT-4 by a physician to generate content. Materials were evaluated for efficiency (words per minute), readability (Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, and SMOG Index), accuracy, personality, comprehensiveness, and safety.
Results:
Results:
GPT-4 demonstrated higher efficiency than clinicians (median 530.03 vs. 37.29 WPM, P < 0.001). GPT-4 content exhibited superior readability on the Flesch-Kincaid grade level, Gunning Fog Index, and SMOG Index (P < 0.001). Expert evaluations revealed that GPT-4 outperformed clinicians in accuracy (5.307 ± 1.731 vs. 4.76 ± 1.098, P = 0.047), personality (54.32 ± 6.212 vs. 33.2 ± 5.395, P < 0.001), comprehensiveness (51.74 ± 6.471 vs. 35.26 ± 6.657, P < 0.001), and safety (median 61 vs. 50, P < 0.001).
Conclusions:
Conclusion: GPT-4 shows promise in generating high-quality, personalized patient education for knee OA, surpassing human experts. This study provides novel evidence for the potential of AI in enabling precise and intelligent patient education. Further research is needed to validate the findings in larger populations and assess the impact on patient outcomes.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.