JMIR Publications

ABSTRACT

Background:

Large language models are increasingly being used for clinical decision support. Yet, a systematic evaluation of the performance and trustworthiness of such models in the intensive care unit (ICU) is lacking.

Objective:

This study aims to create an evaluation framework for large language models in the ICU based on clinical scenarios and evaluates the performance of Meditron-3, a 70-billion-parameter open-source medical large language model.

Methods:

Intensive care medicine specialists from a Swiss university hospital were recruited to create 200 clinical questions. The model used in this evaluation, Meditron-3, generated two independent answers per question. The same experts voted on the preferred answer and then rated it across 10 dimensions on a 5-point Likert scale: Alignment with Guidelines, Question Comprehension, Logical Reasoning, Relevance and Completeness, Harmlessness, Fairness, Contextual Awareness, Rater Confidence, Model Confidence, and Communication and Clarity. Interrater agreement was assessed using Fleiss' kappa, Krippendorff's alpha, and Gwet's AC1. Performance was stratified by interrater agreement quartiles, subspecialty, and clinical task type.

Results:

Eight experts produced 658 expert ratings and 788 answer evaluations. Vote agreement was fair (Fleiss' kappa 0.301, 95% confidence interval 0.23 to 0.38; Gwet's AC1 0.356, 95% confidence interval 0.28 to 0.43). Questions with high interrater agreement scored higher across all dimensions compared to questions with low agreement. Alignment with Guidelines correlated negatively with rater disagreement (Spearman rho -0.433, P < 0.001). The evaluated model scored highest on Communication and Clarity (mean 4.10, standard deviation 0.88) and Fairness (4.07, standard deviation 0.97), and lowest on Relevance and Completeness (3.06, standard deviation 1.12). Alignment with Guidelines was moderate (3.33, SD 1.09). Performance was consistent across subspecialties.

Conclusions:

A clinician-based evaluation framework can effectively benchmark large language models in the ICU and flag domain-specific short-comings. However, the fair interrater agreement highlights the inherent difficulty of evaluating large language model outputs in complex scenarios and underscores the need for standardized evaluation frameworks. The Meditron-3 model demonstrates acceptable performance in communication clarity and fairness but falls short in relevance, completeness, and guideline alignment for intensive care applications.