JMIR Publications

ABSTRACT

Abstract The ethics of autonomous clinical AI are controversial. Traditional ethics assessments have focused on AI model performance rather than health system outcomes. Nichol and colleagues translate established research ethics principles into nine practical recommendations for trials of autonomous clinical artificial intelligence. Their findings support an evolution in clinical trial design. The accuracy and performance of the model is important, but the ethical unit of evaluation should be the complete care pathway, with prespecified obligations for subgroup performance, follow-up care, system failures, and model change. Keywords autonomous artificial intelligence; clinical trials; research ethics; health equity; implementation A timely shift beyond accuracy Clinical trials of artificial intelligence (AI) have often treated the algorithm as the intervention and diagnostic performance as the principal outcome. Nichol and colleagues challenge that narrow frame. Their modified Delphi study applies the seven ethical principles described by Emanuel et al to autonomous AI trials and produces nine recommendations spanning data transparency, bias, generalizability, stakeholder engagement, consent, standards of care, downstream access, and cost [1,2]. The central message is important. An accurate AI autonomous model has little social or clinical value if it does not improve health system outcomes. Panelists did not agree that autonomous AI creates wholly new ethical problems. Bias, inadequate access, weak consent, and poorly chosen comparators predate AI. Yet autonomy can amplify these familiar risks. A clinician may no longer review each result, errors could be repeated at scale, performance can change across sites and a positive screen can transfer clinical and financial burdens to patients or services that lack capacity. The care pathway is the ethical unit of evaluation The paper's most important contribution is its emphasis on health system outcomes rather than model accuracy. In the diabetic retinopathy example, a higher screening rate is an incomplete measure of benefit. A trial should also determine whether people with positive results receive confirmatory assessment, enter treatment when indicated, and avoid preventable loss to follow-up. It should measure time to care, patient burden, false-positive investigations, and the distribution of each outcome across relevant groups. Otherwise, an apparently successful intervention may improve detection while widening the gap between diagnosis and treatment. This pathway-based approach also sharpens comparator selection. The relevant comparison is not simply AI accuracy versus clinician accuracy. It is the AI-enabled pathway versus the incumbent care pathway. These care pathways include patient reach, waiting time, referral completion, safety-netting, and opportunity costs. An autonomous system that performs only one task well e.g. identify retinopathy may not identify additional findings that traditional screening is able to report e.g. a doctor detecting an incidental finding of papilledema at routine retinal screening. Trial protocols should therefore specify which functions are replaced, retained, or added and which patient-relevant outcome supports the claim of clinical value. Consensus must become prespecified obligations The nine recommendations are strong, but terms such as bias assessment, performance and equity assessments still leave decisive choices to investigators. Ethical guidance becomes enforceable only when protocols define thresholds and actions in advance. A subgroup performance difference should trigger a stated response, such as model modification, restricted use, enhanced monitoring or nondeployment. Sites should document referral capacity before enrollment, and trial budgets should identify who pays for confirmatory care and treatment. Independent reviewers should be able to determine whether these conditions were met rather than infer them after favorable results are known. Existing AI reporting guidance offers a foundation. SPIRIT-AI and CONSORT-AI extend protocol and trial-reporting standards for interventions involving AI, while DECIDE-AI addresses early live clinical evaluation [3-5]. The Delphi recommendations add a needed ethical layer, but the next step is integration. Protocol registries, ethics applications, and statistical analysis plans should contain a common set of pathway outcomes, equity analyses, escalation rules, and accountability assignments. This can be challenging for those AI developers who lack health system connections and expertise. Autonomy belongs to a changing clinical system Autonomy should also be treated as a property of the deployed system, not a permanent label attached to a model. Human involvement may remain in image acquisition, communication of results, referral, exception handling, and surveillance. Trials need to describe these roles, identify who can override or suspend the system, and test failure modes such as unusable inputs, uncertain outputs, workflow interruptions, and delayed follow-up. The ethical question is not only whether a model performs well, but whether responsibility remains visible when no specialist reviews the output at the point of care. Versioning adds another difficulty. A trial evaluates a specific model in a specific workflow, whereas deployment may include software updates, changes in devices, and shifts in patient populations. Good machine learning practice and regulatory approaches to planned model changes recognize this lifecycle problem [6,7]. Ethical review should therefore specify which changes require revalidation, renewed consent information, additional subgroup analysis, or independent review. From recommendations to a trial standard The Delphi panel was small, based entirely in the United States, and anchored in diabetic retinopathy screening. The recommendations may not transfer unchanged to therapeutic, triage or management tasks or to health systems with different financing and referral structures. Broader work should include patients who have experienced AI-mediated care, communities likely to bear implementation risks, frontline staff, payers, and regulators from varied settings. The next task is to convert these nine recommendations into a prospective, auditable standard for trial design. Autonomous AI should be judged by the pathway it creates, the care patients can actually reach, and the safeguards that remain when the model or setting changes. Accuracy is necessary evidence. It is not the endpoint of ethical evaluation. Acknowledgments None. Conflicts of Interest Nil References 1. Nichol AA, Youssef A, Larson DB, Abramoff M, Wolf RM, Char D, Martinez-Martin N. Ethics of autonomous AI clinical trials: Delphi study. J Med Internet Res. 2026;28:e91472. doi:10.2196/91472 2. Emanuel EJ, Wendler D, Grady C. What makes clinical research ethical? JAMA. 2000;283(20):2701-2711. doi:10.1001/jama.283.20.2701 3. Cruz Rivera S, Liu X, Chan AW, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med. 2020;26:1351-1363. doi:10.1038/s41591-020-1037-7 4. Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, CONSORT-AI and SPIRIT-AI Steering Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med. 2020;26:1364-1374. doi:10.1038/s41591-020-1034-x 5. Vasey B, Nagendran M, Campbell B, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med. 2022;28:924-933. doi:10.1038/s41591-022-01772-9 6. International Medical Device Regulators Forum. Good machine learning practice for medical device development: guiding principles. Published 2025. Accessed September 15, 2026. https://www.imdrf.org/sites/default/files/2025-02/IMDRF_AIML%20WG_GMLP_N88%20Final.pdf 7. US Food and Drug Administration. Predetermined change control plans for machine learning-enabled medical devices: guiding principles. Published 2025. Accessed September 15, 2026. https://www.fda.gov/medical-devices/software-medical-device-samd/predetermined-change-control-plans-machine-learning-enabled-medical-devices-guiding-principles