JMIR Publications

ABSTRACT

Artificial intelligence (AI) in healthcare has the promise of providing accurate and efficient results. However, AI can also be black boxes, where the logic behind their results is nonrational. There are concerns if these questionable results are used in patient care. As physicians have the duty to provide care based on their clinical judgment in addition to their patients’ values/preferences, it is crucial that physicians are validating the results from AI. Yet, there are some physicians who suffer from a phenomenon known as automation bias, where there is an assumption from the user that AI is always right. This is a dangerous mindset as those users will not validate the results given their trust in AI systems. Several factors impact a user’s susceptibility to automation bias, such as inexperience or being born in the digital age. In this viewpoint, I argue that these factors and a lack of AI education in the medical school curriculum causes automation bias. I also explore the harms of automation bias and why prospective physicians need to be vigilant when using AI. Furthermore, it is important to consider what attitudes are being taught to students when introducing ChatGPT, which will possibly be some of the students’ initiation to using AI prior to using AI in the clinical setting. Therefore, in attempts to avoid the problem of automation bias in the long-term, in addition to the necessity of AI education in the curriculum, the use of ChatGPT in medical education should be limited to certain tasks. Otherwise, no constraints on what ChatGPT should be used for could lead to automation bias.