JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly being explored for clinical decision support, but their reliability, consistency, and safety in dental clinical reasoning remain insufficiently understood. Conventional accuracy-focused benchmarks may not capture clinically important failures in reasoning and management.

Objective:

This study aimed to evaluate the accuracy, repeated-run consistency, task adherence, clinical reasoning, treatment appropriateness, and safety of an LLM across structured multiple-choice and open-ended dental scenarios.

Methods:

Ten clinically validated dental scenarios adapted from Odell's Clinical Problem Solving in Dentistry were transformed into 100 single-best-answer multiple-choice questions (MCQs) and 10 open-ended clinical reasoning tasks. ChatGPT (GPT-5) was evaluated with zero-shot prompts in new conversations. Each prompt was repeated 10 times; MCQ option order was randomized across repetitions. In total, 1000 MCQ responses and 100 open-ended responses were assessed using prespecified expert reference standards, an Script Concordance Test (SCT)-informed analytic framework, task-validity categories, repeated-run consistency measures, and an independent 4-level patient-safety scale.

Results:

MCQ accuracy ranged from 64% to 92% across cases, and mean modal consistency ranged from 0.65 to 0.93. Open-ended diagnostic accuracy ranged from 0% to 90%. Six cases had critical safety-error rates of at least 30%, including rates of 70% to 80% in Cases 8 to 10. The largest gaps between MCQ and open-ended diagnostic accuracy (a difference of at least 50 percentage points) occurred in Cases 6, 7, 8, 9, and 10, indicating that comparatively strong structured-task performance did not consistently translate into accurate or safe autonomous reasoning.

Conclusions:

Structured MCQ performance alone may substantially overestimate clinically meaningful LLM capability. Evaluation frameworks for dental AI should incorporate open-ended reasoning, repeated-run stability, task adherence, treatment appropriateness, and patient safety. The evaluated model was not sufficiently reliable for autonomous clinical decision-making; any educational or clinical use requires expert oversight and explicit safety safeguards.