Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.
Who will be affected?
Readers: No access to all 28 journals. We recommend accessing our articles via PubMed Central
Authors: No access to the submission form or your user account.
Reviewers: No access to your user account. Please download manuscripts you are reviewing for offline reading before Wednesday, July 01, 2020 at 7:00 PM.
Editors: No access to your user account to assign reviewers or make decisions.
Copyeditors: No access to user account. Please download manuscripts you are copyediting before Wednesday, July 01, 2020 at 7:00 PM.
Cost, Usability, Credibility, Fairness, Accountability, Transparency, and Explainability Framework for Safe and Effective Large Language Models in Medical Education: Narrative Review and Qualitative Study
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
CUCFATE Frameworks for Safe and Effective Large Language Models in Medical Education
Vinaytosh Mishra;
Majdi Quttainah;
Somayya Madakam;
Yotam Lurie;
Shlomo Mark
ABSTRACT
Background:
World has witnessed increased adoption of Large Language Models (LLMs) in the last year. Although the products developed using LLMs have the potential to solve accessibility and efficiency problems in healthcare, there is a lack of guidelines available for developing LLMs for healthcare and especially medical education.
Objective:
The study aims to identify and prioritize the enablers for developing successful LLMs for medical education. The study also discusses the relationship among these identified enablers.
Methods:
The study first identifies key enablers for LLM development using the narrative review of extant literature. The next opinion of users of LLMs was taken to determine the relative importance of these enablers using the multi-criteria decision-making method called Analytical Hierarchy Process. Further, Total Interpretive Structural Modelling (TISM) was used to analyze product developers' perspectives and ascertain the relationship and hierarchy among these enablers. Finally, Cross-impact matrix multiplication applied to classification (MICMAC) was done to find these enablers' relative driving and dependence power. The non-probabilistic purposive sampling was used for the study.
Results:
The result of AHP concluded that Credibility with a priority weight of 0.37 is the most important enabler, while usability with a priority weight of 0.04 have negligible importance. The results of TISM concur with the findings of the AHP. The only striking difference was product developers gave the least importance to cost. The development of the MICMAC analysis suggests that cost has a strong influence on other enablers. The inputs of the focus group were found reliable with a consistency ratio (CR<0.1).
Conclusions:
Conclusion
The study is the first to identify, prioritize, and analyze the relationship of enablers for effective LLMs for medical education. The study provides an easy to comprehendible prescriptive framework CUCFATE for the same. The study findings are useful for healthcare professionals, health technology experts, medical technology regulators, and policymakers. Clinical Trial: Not Applicable
Citation
Please cite as:
Mishra V, Quttainah M, Madakam S, Lurie Y, Mark S
Cost, Usability, Credibility, Fairness, Accountability, Transparency, and Explainability Framework for Safe and Effective Large Language Models in Medical Education: Narrative Review and Qualitative Study
