JMIR Publications

ABSTRACT

Background:

Critical appraisal of research remains important for evidence-based practice. As generative artificial intelligence (genAI) becomes increasingly integrated into scientific writing and research discourse, clinicians must be able to identify AI-generated inaccuracies, misinformation, and hallucinations. To support the development of these skills, the UCLA neurology residency program implemented an AI-integrated journal club curriculum in 2025.

Objective:

The aim of this study was to explore how genAI acted as scaffolding to facilitate neurology residents’ learning of critical appraisal within a journal club curriculum. To explore this question, we employed a constructivist theoretical framework.

Methods:

We conducted a sequential qualitative study using focused ethnography of four monthly AI-integrated no-prep journal clubs followed by semi-structured interviews with participating neurology residents. Ethnographic data included observational session summaries and genAI transcripts, which were analyzed for prompt quality. Interviews were conducted one-on-one via Zoom. Observational summaries and interview transcripts were analyzed using inductive thematic analysis. Credibility and trustworthiness were enhanced through triangulation of ethnographic and interview data, multiple coders, and member checking.

Results:

Across four journal club sessions, ethnography revealed inconsistent prompt quality, general distrust of outputs, expression of uncertainty, collaboration in problem solving, and reliance on faculty scaffolding to contextualize AI outputs. Based on a grading rubric, prompt quality varied across sessions (Mean = 70%, SD = 22.5%) with no clear improvement over time. Nine neurology PGY 2-4 residents participated in interviews. Most reported limited prior exposure to AI but increased AI use following the curriculum. While some residents viewed AI as a useful scaffold for learning, others voiced concerns regarding cognitive offloading and reduced independent thinking. Residents described AI as untrustworthy without contextualization from faculty facilitators and desired more orientation on how to use AI tools effectively.

Conclusions:

Applications of genAI to medical education should carefully consider approaches that optimize the strengths and minimize the weaknesses of the technology. In this qualitative study of an AI-integrated no-prep journal club format, residents were mixed in their perspectives regarding the benefits of AI as a scaffold versus the risks of deskilling and cognitive offloading. Faculty facilitation was ultimately key in contextualizing outputs and verifying accuracy of information. Future iterations may consider prioritizing strategies for promoting independent learning of the critical thinking skills necessary for learning study design, with subsequent use of genAI tools to complement this learning, as opposed to using genAI at the outset.