JMIR Publications

ABSTRACT

Background:

The increasing application of Generative Artificial Intelligence Large Language Models (LLMs) in various fields including Dentistry raises questions about their accuracy.

Objective:

This study aimed to comparatively evaluate the answers provided by four LLMs - Google's Bard, OpenAI's ChatGPT-3.5 and ChatGPT-4, and Microsoft's Bing to clinically relevant questions from the field of dentistry.

Methods:

The LLMs were queried with clinical dentistry-related questions. The LLMs’ answers were graded in a range from 0 (minimum) to 10 (maximum) points, against traditionally collected scientific evidence such as guidelines and consensus statements, as if they were exam questions posed to students, by two experienced faculty members. The scores were compared statistically to identify the best-performing model.

Results:

While ChatGPT-4 statistically outperformed the other LLMs, all models exhibited occasional inaccuracies, generality, outdated content, and a lack of source references. The evaluators noted instances where LLMs delivered irrelevant information, vague answers, or information that was not fully accurate.

Conclusions:

The study demonstrates that while LLMs hold promising potential as an aid in the implementation of evidence-based dentistry, their current limitations can lead to potentially harmful healthcare decisions if not used judiciously. Therefore, these tools should not replace the dentist's critical thinking and in-depth understanding of the subject matter. Further research, clinical validation, and model improvements are necessary for these tools to be fully integrated into dental practice.