JMIR Publications

ABSTRACT

Background:

Large Language Models (LLMs) are Artificial Intelligences that predict desired user outputs. There are gaps in Dermatology education that could benefit from the incorporation of LLMs. However, efforts to do so have been hindered by concerns over the accuracy, transparency, and reproducibility of responses. Furthermore, LLMs have historically performed inconsistently on standardized medical questions, possibly due to a lack of representative data within an LLM’s armamentarium. NotebookLM (NLM) by Google, an LLM that advertises to develop answers from user-uploaded sources and provide reliable citations, is a source-based LLM that may offer a possible solution to these shortcomings. It also has the potential to integrate student-developed notes into teaching and thereby utilize principles from Vygotsky’s Zone of Proximal Development, along with Cognitive Load Theory, to enhance learning quality.

Objective:

To evaluate how the provision of extensive student-created study guides affects NotebookLM performance with implications for its usability within the classroom and to compare its performance to other industry LLMs in answering Step1 Dermatology questions.

Methods:

Four LLMs were used in experimentation - NLM with uploaded pre-clerkship study guides, NLM with an inputted blank sheet of paper, ChatGPT-4o mini, and Google Gemini 1.5 Flash. Each model completed three trials of 121 text-based USMLE Step 1 Dermatology questions from the AMBOSS question bank. They were evaluated for overall accuracy, accuracy by question difficulty, reproducibility of responses across trials, and agreement in answer selection between different models. Data on each of these categories was gathered, charted, and analyzed using Chi-Squared tests of Independence and Fleiss’s Kappa statistics.

Results:

NLM w/ Notes exhibited significantly more omissions (unanswered questions) than other LLMs (10.5% vs ≤1.65%). When omissions were excluded from statistical analysis, ChatGPT-4o Mini had the greatest accuracy (86%). NLM had unchanged accuracy when compared with inputted study guides and without (76% vs 76%); among all other LLMs, NLM with inputted material had the highest rates of reproducibility (Fleiss Kappa of 0.939).

Conclusions:

All the LLMs tested here performed higher than previously reported in literature, demonstrating a rapid progression in LLM capabilities. NLM has improved response completeness and reproducibility with user-inputted data, but not factual accuracy. One interpretation is that user-inputted data is being utilized more as an end-state 'filter' rather than being integrated within core reasoning processes, but the unclear nature of LLM cognition precludes definitive answers. More research is needed to harness source-based LLM potential within the classroom under structured, theory-informed educational roles.