JMIR Publications

ABSTRACT

Background:

The interpretation of complete blood count (CBC) reports is a critical yet subjective task in the diagnosis of hematologic diseases. While large language models (LLMs) show promise for clinical decision support, their real-world performance and safety profiles remain insufficiently evaluated.

Objective:

To evaluate and compare three advanced LLMs—ChatGPT 5, Grok 4, and DeepSeek R1—in interpreting real-world CBC reports for hematologic diseases across multiple quality and task-specific dimensions.

Methods:

We retrospectively collected 100 CBC reports from patients with confirmed hematologic diseases at a tertiary hospital. Three LLMs interpreted these reports across five sequential tasks: analyzer alert processing, abnormal item identification, correlation analysis of abnormal items, preliminary diagnosis, and clinical management. Outputs were evaluated in a blinded manner by two junior and two senior laboratory professionals across six quality dimensions using 5-point Likert scales. Inter-rater reliability was assessed via intraclass correlation coefficients (ICC). Model performance was compared using Friedman tests, and errors were classified as either hallucinations or reasoning errors.

Results:

Across 100 report interpretations, DeepSeek R1 achieved superior performance (median score 4.0 [IQR 4.0–5.0] for junior evaluators; 5.0 [IQR 4.0–5.0] for senior evaluators) with excellent inter-rater reliability (junior ICC 0.817 [95% CI 0.804–0.830]; senior ICC 0.766 [95% CI 0.749–0.782]). Senior evaluators of three LLMs consistently assigned higher ratings than junior evaluators (p < 0.001). DeepSeek R1 outperformed the other models in five of six quality dimensions and across all clinical tasks (all p < 0.001). ChatGPT 5 demonstrated the highest concordance with gold-standard diagnoses (93%), whereas Grok 4 aligned most closely with initial clinical suspicions (96%) but demonstrated the lowest concordance with gold-standard diagnoses (89%). Notably, ChatGPT 5 exhibited 12 hallucination errors during analyzer alert processing; Grok 4 produced the highest proportion of unsafe outputs in clinical management (3.8%); and all models made unsupported inferences to varying degrees during the correlation analysis of abnormal items.

Conclusions:

DeepSeek R1 achieved the highest ratings for CBC interpretation, particularly among senior evaluators, reflecting near-expert performance. ChatGPT 5 demonstrated the highest concordance with gold-standard diagnoses, highlighting strong reasoning capabilities. However, all models exhibited distinct error patterns and performance heterogeneity, underscoring the necessity for human oversight and providing an evidence-based framework for safe LLM deployment in laboratory medicine.