JMIR Publications

ABSTRACT

Background:

ChatGPT is increasingly relied upon as a study tool among medical trainees during the preclinical curricular phase, raising concern about its accuracy and reliability.

Objective:

The aim of this study is to compare ChatGPT 4o mini to UpToDate with the purpose of assessing for similarity.

Methods:

We queried a total of 150 preclinical-level questions: 30 biochemistry, 30 immunology, 30 microbiology, 30 pharmacology, and 30 pathology. ChatGPT was asked each question 5 times to account for stochasticity. Next, a text network analysis was performed using cosine comparisons of term frequency inverse-document frequency (TF-IDF) to gauge similarity between ChatGPT and UpToDate responses per question for each subject. A statistical reference (p = 0.05) for interpretation of TF-IDF values was generated using random text samples with same length distribution as the UpToDate responses. TF-IDF similarity of ChatGPT responses to overall subject category was also performed.

Results:

ChatGPT responses were most similar to UpToDate with regard to answering pharmacology questions (TF-IDF 0.3380.134). ChatGPT’s response similarity to UpToDate for the remaining subjects were 0.3210.142 for pathology, 0.296±0.120 for biochemistry, 0.2970.108 for microbiology, and 0.2750.102 for immunology. Reference TF-IDF scores of randomly generated text were 0.262, 0.279, 0.243, 0.267, and 0.281 for biochemistry, immunology, microbiology, pharmacology, and pathology respectively.

Conclusions:

The majority of ChatGPT responses are similar to UpToDate responses for preclinical questions across the subjects of biochemistry, immunology, microbiology, pharmacology, and pathology. Thus, ChatGPT may have a role in medical training during the preclinical curricular phase with the caveat that its utility may vary based on subject.