JMIR Publications

ABSTRACT

Background:

Clinical notes are rich sources of clinical information and are routinely used in clinical practice to infer past diagnoses and other comorbidities. Large language models may simplify the task by extracting this information from clinical notes.

Objective:

To evaluate the performance of state-of-the-art LLMs on the task of labeling clinical notes.

Methods:

This IRB approved retrospective study comprised 380 clinical notes obtained from two institutions in the US. The notes were labeled manually for the presence or absence of 29 conditions (simplified Elixhauser comorbidity index) by two human readers – a radiology resident and a medical student under the supervision of a senior radiologist. Forty-six LLMs, including 4 closed-source commercial models from the GPT and Claude families, and 42 open-source LLMs from the Llama (13 models) and the Qwen (29 models) families were used to label each note. The GPT-o3-mini-high reasoning model was used to parse the free-text model responses to binary yes/no answers. Using the manual labels as the reference standard, the recall, precision and F1 scores for the models were evaluated. The effect of factors such as the length of the notes, and the family, size and release recency of the LLMs using generalized linear models.

Results:

The top performing LLMs included the GPT and Claude models, and the larger open-source models including Llama 3 70B, Llama 3.1 70B, Llama 3.1 405B, Llama 3.3 70B, Qwen 1.5 72B, Qwen 1.5 110B, Qwen 2 72B and Qwen 2 VL 72B, which achieved F1 scores above 70%. The most recent and higher-capacity releases of the LLMs have been improving mainly in their precision. The evaluation of 29 conditions showed a large variance on the performance depending on the condition.

Conclusions:

In this evaluation with 46 LLMs, larger open source LLMs such as the Llama 3.3 70B performed at par or better than the GPT and Claude models, achieving F1 scores above 70%. The performance of open-source models improved significantly with model size and recency. These results suggest that locally-run open-source LLMs can be potentially used to label clinical notes.