JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly used for clinical information processing and reasoning, where numerical values must often be interpreted relative to explicit thresholds or reference ranges. Although classifying laboratory values as low, within range, or high is deterministic when the reference range is given, numerical comparison can constitute a basic component of more complex clinical reasoning. Previous studies have evaluated the overall accuracy of such tasks, but less is known about where errors occur relative to decision boundaries and how model scale, prompt formulation, and task-irrelevant clinical context affect performance.

Objective:

This study aimed to characterize LLM performance in clinical laboratory reference-range classification as a function of model scale, measurement position relative to reference-range boundaries, Japanese versus English prompt formulations, and the inclusion of laboratory test names and units.

Methods:

Of the 43 clinical laboratory items initially considered, 40 met the prespecified criterion and were included in the analysis. We synthetically generated 6429 measurements from the publicly available reference ranges of these 40 items, with LOW, IN, and HIGH values systematically positioned relative to the lower and upper boundaries. We evaluated 10 open LLMs from the Qwen3 (0.6B-32B) and Gemma 4 families (E2B-31B) under 8 prompt conditions combining Japanese or English prompts with the presence or absence of test names and units. Thinking mode was disabled. Predictions were determined from the first-step next-token probabilities of candidate LOW, IN, and HIGH tokens, and item-and-label macro accuracy was the primary outcome. No human participants or patient data were used; ethics approval was not required.

Results:

Performance increased substantially with model scale but differed between model families. Qwen3 macro accuracy was 0.350 at 0.6B, 0.397 at 1.7B, 0.863 at 4B, and 0.966-0.976 at 8B-32B. Gemma accuracy increased from 0.865 at E2B and 0.946 at E4B to 0.995 at 12B and 0.998 at 31B. Qwen models retained localized errors near the upper reference-range boundary even at larger scales; in Qwen3-32B, the mean IN-versus-HIGH log-probability difference crossed zero at a normalized position of approximately 0.96, before the stated upper boundary of 1.0. In contrast, position-dependent errors largely disappeared in Gemma 12B and 31B. Adding test names or units did not consistently improve performance and significantly reduced accuracy in several models (Holm-adjusted P<.05), with particularly pronounced effects in Gemma E2B and E4B. Differences between Japanese and English prompt conditions varied by model and label, with no consistently superior prompt language.

Conclusions:

LLM reference-range classification cannot be fully characterized by overall accuracy or model scale alone. Error localization near numerical boundaries and sensitivity to additional clinical context differed substantially across model families and sizes. Controlled evaluation of such basic numerical operations may complement task-level benchmarks when assessing LLMs for more complex clinical information processing and reasoning.