Accepted for/Published in: JMIR AI
Date Submitted: Aug 16, 2023
Date Accepted: Feb 3, 2024
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
CUCFATE Frameworks for Safe and Effective Large Language Models in Medical Education
ABSTRACT
Background:
World has witnessed increased adoption of Large Language Models (LLMs) in the last year. Although the products developed using LLMs have the potential to solve accessibility and efficiency problems in healthcare, there is a lack of guidelines available for developing LLMs for healthcare and especially medical education.
Objective:
The study aims to identify and prioritize the enablers for developing successful LLMs for medical education. The study also discusses the relationship among these identified enablers.
Methods:
The study first identifies key enablers for LLM development using the narrative review of extant literature. The next opinion of users of LLMs was taken to determine the relative importance of these enablers using the multi-criteria decision-making method called Analytical Hierarchy Process. Further, Total Interpretive Structural Modelling (TISM) was used to analyze product developers' perspectives and ascertain the relationship and hierarchy among these enablers. Finally, Cross-impact matrix multiplication applied to classification (MICMAC) was done to find these enablers' relative driving and dependence power. The non-probabilistic purposive sampling was used for the study.
Results:
The result of AHP concluded that Credibility with a priority weight of 0.37 is the most important enabler, while usability with a priority weight of 0.04 have negligible importance. The results of TISM concur with the findings of the AHP. The only striking difference was product developers gave the least importance to cost. The development of the MICMAC analysis suggests that cost has a strong influence on other enablers. The inputs of the focus group were found reliable with a consistency ratio (CR<0.1).
Conclusions:
Conclusion The study is the first to identify, prioritize, and analyze the relationship of enablers for effective LLMs for medical education. The study provides an easy to comprehendible prescriptive framework CUCFATE for the same. The study findings are useful for healthcare professionals, health technology experts, medical technology regulators, and policymakers. Clinical Trial: Not Applicable
Citation
