JMIR Publications

ABSTRACT

Background:

Clinical checklists are a list of items designed to reduce the errors of omission in clinical settings. To engineer clinical decision support systems (CDSS), checklists are manually extracted from clinical documents, causing the process to be time-consuming and susceptible to human errors, specifically when documents need updating. This can be addressed by automating the extraction process.

Objective:

To investigate the effectiveness of generic Large Language Models (LLMs) using various prompting techniques to extract checklist items from seven public neurotrauma documents.

Methods:

We selected seven clinical checklists used in neurotrauma: AO Trauma Asia Classification (ATAC), Pre-Arrival Trauma Checklist (PATC), Adult Trauma Triaging Tool (ATTT), Canadian CT Head Rule (CCHR), Canadian C Spine Rule (CCR), Nexus Criteria for C-Spine Imaging (NCCI), and Nexus Head CT Instrument (NHCI). We queried GPT-3.5-Turbo, GPT-4, GPT-4o, GPT-o1-mini and Claude 3.5 Sonnet using role-play, instruction following, chain-of-thought and few-shot prompting techniques and measured effectiveness through string similarity and cosine similarity.

Results:

Effectiveness varied greatly across LLMs and prompting techniques whereby the average string similarity ranged from 0% to 98.31%±1.20% and the average cosine similarity between 0.13±0.01 to 1.00±0.00. GPT-4 with Chain of Thought prompting demonstrated perfect similarity across more checklists (PATC, CCHR, NCCI). GPT-4o and GPT-o1-mini with instruction following demonstrated perfect similarity only the PATC checklist. GPT-3.5-Turbo and Claude 3.5 Sonnet using few-shot prompting failed to extract four of the seven checklists (ATAC, CCR, NCCI, NHCI).

Conclusions:

LLMs are limited in consistently extracting items from clinical checklists effectively despite the use of various prompting techniques. Clinical Trial: NA