JMIR Publications

ABSTRACT

Objective:

We employed the free artificial intelligence (AI) tool Google NotebookLM®, powered by the large language model (LLM) Gemini 2.0, to construct a medical decision-making aid for diagnosing and managing airway diseases, and subsequently evaluated its functionality and performance in clinical workflow.

Methods:

After feeding this tool with relevant published clinical guidelines for these diseases, we evaluated the feasibility of the system regarding its behavior, ability, and potential, and made simulated cases and used this system to solve associated medical problems. The test and simulation questions were designed by a pulmonologist, and the appropriateness (focusing on accuracy and completeness) of AI responses were judged by three pulmonologists independently. The system was then deployed in an emergency department (ED) setting, where it was tested by medical staff (n=20) to see how it affected the process of clinical consultation. Test opinions were collected through questionnaire.

Results:

Most (58/84=66.7%) of the specialists’ ratings regarding AI responses were above average. The inter-rater reliability was moderate on accuracy (Intraclass correlation coefficient (ICC)=0.612, P<.001) and good on completeness (ICC=0.773, P<.001). When deployed in an ED setting, this system could respond with reasonable answers, enhance the literacy of personnel about these diseases. The potential to save the time spent in consultation did not reach statistical significance (Kolmogorov-Smirnov D=.223, P=.237>.05) across all participants, but indicated a favorable outcome if we analyzed only physicians’ responses.

Conclusions:

This system is customizable, cost-efficient, and accessible by clinicians and allied professionals without any computer coding experience in treating airway diseases. It provides convincing guideline-based recommendations, increases the staff’s medical literacy, and potentially saves physicians’ time spent on consultation. It warrants further evaluation in other medical disciplines and healthcare environments.