Previously submitted to: JMIR Medical Informatics (no longer under consideration since May 01, 2026)
Date Submitted: Feb 17, 2026
Open Peer Review Period: Mar 11, 2026 - May 6, 2026
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Real-World Validation of Arkangel AI, a Conversational Agent for Real-Time Evidence-Based Medical Question Answering: Randomized Controlled Trial
ABSTRACT
Background:
The volume of biomedical evidence makes it difficult for physicians to access up-to-date information quickly during routine practice. Large language models (LLMs) have shown promise for clinical support, but most evaluations use multiple-choice or simulated settings and do not assess real-world use by practicing clinicians. Evidence-traceable tools that combine LLMs with real-time retrieval of curated sources could support clinical question-answering; external validation of such tools in routine practice is lacking.
Objective:
To evaluate the effect of Arkangel AI use on response time and the validity of physicians’ answers to open-ended clinical questions, compared with traditional search methods without artificial intelligence support.
Methods:
Physicians were randomly assigned to two study groups—Group A (Arkangel AI–assisted search) and Group B (traditional search methods). A total of 202 physicians initiated the study, and 71 completed all responses and were included in the final analysis. Each participant solved four clinical cases, each comprising four open-ended questions. Responses were evaluated by clinical specialists blinded to group assignment using six predefined validity criteria. The association between Arkangel AI use and response validity was assessed using multivariable logistic regression, adjusting for academic and sociodemographic characteristics.
Results:
Physicians who used Arkangel AI had higher validity scores than those using traditional search. For total validity, the median was 2.83 (IQR 2.52–3.00) in Group A and 2.46 (IQR 2.21–2.67) in Group B (median difference 0.38; 95% CI 0.17–0.54; Mann-Whitney U test, P<.001). The effect size was large (Cliff delta 0.59; 95% CI 0.34–0.80), with a 79% superiority probability for Group A. In the multivariable model, the association between Arkangel AI use and higher response validity showed a positive trend (adjusted OR 2.42; 95% CI 0.82–7.16) but did not reach statistical significance (P=.11). Response times were comparable between groups, with no significant difference in time per question or number of searches.
Conclusions:
LLM-assisted clinical search with Arkangel AI was associated with higher response validity and comparable response times in this sample of practicing physicians. The findings support the potential role of evidence-based conversational agents as decision-support tools in medical education and clinical practice and justify further studies with larger samples. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.