Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Performance of three conversational generative AI models for computing maximum safe doses of local anesthetics: a comparative analysis.
ABSTRACT
Background:
Generative artificial intelligence (AI) is showing great promise as a tool to optimize decision-making across various fields, including medicine. In anesthesiology, accurately calculating maximum safe doses of local anesthetics (LAs) is crucial to prevent complications such as local anesthetic systemic toxicity (LAST). Current methods for determining LA dosage are largely based on empirical guidelines and clinician experience, which can result in significant variability and dosing errors. Generative AI models may offer a solution, as they could be capable of integrating all the relevant parameters and suggest adequate LA doses.
Objective:
This study aimed to evaluate the efficacy and safety of 3 generative AI models—ChatGPT, Copilot, and Gemini—in calculating maximum safe LA doses, with the goal of determining their potential utility in clinical practice.
Methods:
A comparative analysis was conducted using a 51-question questionnaire designed to assess LA dose calculation across 10 simulated clinical vignettes. The responses generated by ChatGPT, Copilot, and Gemini were compared to reference doses calculated using a scientifically validated set of rules. Quantitative evaluations involved comparing AI-generated doses to these reference doses, while qualitative assessments were conducted by independent reviewers using a 5-point Likert scale.
Results:
All 3 AI models—Gemini, ChatGPT, and Copilot—completed the questionnaire and demonstrated a basic understanding of LA dose calculation principles, but their performance in providing safe doses varied significantly. Gemini frequently avoided proposing any specific dose, instead recommending consultation with a specialist. When it did provide dose ranges, they often exceeded safe limits by 140±103% in cases involving mixtures. ChatGPT provided unsafe doses in 90% of cases, exceeding safe limits by 198±196%. Copilot's recommendations were unsafe in 67% of cases, exceeding limits by 217±239%. Qualitative assessments rated Gemini as "fair" and both ChatGPT and Copilot as "poor".
Conclusions:
Generative AI models like Gemini, ChatGPT, and Copilot currently lack the accuracy and reliability needed for safe LA dose calculation. Their poor performance suggests that they should not be used as decision-making tools for this purpose. Until more reliable AI-driven solutions are developed and validated, clinicians should rely on their expertise, experience, and a careful assessment of individual patient factors to guide LA dosing and ensure patient safety. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.