JMIR Publications

ABSTRACT

Background:

Ultrasound examinations, while valuable, are time-consuming and often limited in availability. Consequently, many hospitals implement reservation systems; however, these systems typically lack prioritization for examination purposes. Hence, our hospital uses a waitlist system that prioritizes examination requests based on their clinical value when slots become available due to cancellations. This system, however, requires a manual review of examination purposes, which are recorded in free-form text. We hypothesized that AI language models could preliminarily estimate the priority of requests prior to manual reviews.

Objective:

This study aimed (1) to investigate potential challenges associated with using language models for estimating the priority of medical examination requests and (2) to evaluate the performance of language models in processing Japanese medical texts.

Methods:

We retrospectively collected ultrasound examination requests from the waitlist system at Keio University Hospital, spanning January 2020 to March 2023. Each request comprised an examination purpose documented by the requesting physician and a six-tier priority level assigned by a radiologist during the clinical workflow. We fine-tuned JMedRoBERTa, Luke, OpenCalm, and LLaMA2 under two conditions: (1) tuning only the final layer and (2) tuning all layers using either standard backpropagation or low-rank adaptation (LoRA).

Results:

We had 2335 and 204 requests in the training and test datasets post-cleaning. When only the final layers were tuned, JMedRoBERTa outperformed the other models (Kendall coefficient = 0.225). With full fine-tuning, JMedRoBERTa continued to perform best (Kendall’s coefficient = 0.254), though with reduced margins compared to the other models. The radiologists’ re-evaluation yielded a Kendall coefficient of 0.172.

Conclusions:

Language models can estimate the priority of examination requests with accuracy comparable to human radiologists. The fine-tuning results indicate that general-purpose language models can be adapted to domain-specific texts (i.e., Japanese medical texts) with sufficient fine-tuning. Further research is required to address priority rank ambiguity, expand the dataset across multiple institutions, and explore more recent language models with potentially higher performance or better suitability for this task.