JMIR Publications

ABSTRACT

Generative artificial intelligence (AI) is a groundbreaking health care technology in the United States but could create ethical challenges without proper safeguards. Perhaps, nowhere is this more concerning than with AI clinical algorithms, which critics claims could be magnifying harmful biases and exacerbating health disparities. Since the initiation of three class action lawsuits in the latter-half of 2023, the potential misuse of these algorithms by health care insurers is now a public concern. The common thread in these suits are allegations these algorithms denied medically necessary care via an almost fully automated claims decision process, largely devoid of human oversight. Our case illustration detailed major flaws in the application of such an algorithm which could have left an elderly Medicare Advantage Plan beneficiary without access to medically necessary care. Although advocates at the state and national level are seeking remediation, it seems likely such practices will persist as insurers know only a small percentage will appeal these denied claims. Insurers will likely fight these suits, which may drag on for years, costing relatively little in comparison to the huge cost saving from denied claims. It is apparent that these algorithms are not a “crystal ball for care,” and there is a need for human input at critical junctures in the process. The AI and bioethics communities must work in concert to assure that the design, development and deployment of these generative AI tools are done in the utmost ethically responsible way, similar to what some health entities have already accomplished. Such perhaps can be best accomplished by adhering to the ethical principles established by the World Health Organization. By doing so, both the AI and bioethics communities can help assure that these generative AI clinical support tools are utilized as support tools, and do not replace the clinical team.