Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.
Who will be affected?
Readers: No access to all 28 journals. We recommend accessing our articles via PubMed Central
Authors: No access to the submission form or your user account.
Reviewers: No access to your user account. Please download manuscripts you are reviewing for offline reading before Wednesday, July 01, 2020 at 7:00 PM.
Editors: No access to your user account to assign reviewers or make decisions.
Copyeditors: No access to user account. Please download manuscripts you are copyediting before Wednesday, July 01, 2020 at 7:00 PM.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Standardizing and Scaffolding Healthcare AI-Chatbot Evaluation
Yining Hua;
Winna Xia;
David Bates;
George Luke Hartstein;
Hyungjin Tom Kim;
Michael Li;
Benjamin W. Nelson;
Charles Stromeyer IV;
Darlene King;
Jina Suh;
Li Zhou;
John Torous
ABSTRACT
Background:
The rapid rise of healthcare chatbots, valued at $787.1 million in 2022 and projected to grow at 23.9% annually through 2030, underscores the need for robust evaluation frameworks. Despite their potential, the absence of standardized evaluation criteria and rapid AI advancements complicate assessments.
Objective:
This study addresses these challenges by developing the first comprehensive evaluation framework inspired by health app regulations and integrating insights from diverse stakeholders.
Methods:
Following PRISMA guidelines, we reviewed 11 existing frameworks, refining 271 questions into a structured framework encompassing three priority constructs, 18 second-level constructs, and 60 third-level constructs.
Results:
Our framework emphasizes safety, privacy, trustworthiness, and usefulness, aligning with recent concerns about AI in healthcare.
Conclusions:
This adaptable framework aims to serve as the initial step in facilitating the responsible integration of chatbots into healthcare settings. Clinical Trial: NA
Citation
Please cite as:
Hua Y, Xia W, Bates D, Hartstein GL, Kim HT, Li M, Nelson BW, Stromeyer C IV, King D, Suh J, Zhou L, Torous J
Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review