JMIR Publications

ABSTRACT

Background:

Unstructured electronic health records (EHRs) hinder intestinal infection monitoring. Large language models (LLMs) enable automated symptom extraction. However, their clinical validation is limited by a lack of systematic multi‑model comparisons, unclear prompting strategies, and privacy risks of cloud‑based models (e.g., data leakage, cross‑border transfer).

Objective:

To systematically evaluate the performance of locally deployed open-source LLMs across four model families in extracting intestinal symptoms from unstructured EHR chief complaints under different prompting strategies.

Methods:

From a city‑wide healthcare information platform in Wuhan, China, we randomly selected 1,000 chief complaints from outpatient records of intestinal clinics, infectious disease departments, pediatrics, and fever clinics. Six symptoms related to intestinal infectious diseases (IIDs), including diarrhea/bloody/mucoid stools, vomiting, abdominal pain, fever, nausea, and rash were manually annotated as a gold-standard dataset. Twelve locally deployed open-source LLMs across four families, namely Gemma3 (1b, 4b, 12b), Qwen3 (1.7b, 8b, 14b), Deepseek-r1 (1.5b, 7b, 14b), and Llama (Llama2-Chinese: 7b, 13b; Llama3.1: 8b) were evaluated on the symptom extraction task using the gold-standard dataset. Three prompting strategies (norole, zeroshot, fewshot) were tested. Performance metrics included accuracy, precision, recall, F1-score, specificity, balanced accuracy, and inference time. Statistical comparisons employed Friedman tests for global differences, followed by Wilcoxon signed-rank and Mann-Whitney U tests with Bonferroni and FDR corrections for pairwise comparisons.

Results:

Among four families, Qwen3 models showed higher F1‑scores and balanced accuracy, with Qwen3‑1.7b achieving a macro‑averaged F1‑score of 0.85 under zeroshot prompting and Qwen3‑8b reaching 0.89 under norole prompting, while Gemma3 demonstrated robust performance in the small to medium scale. Symptom-wise, models agreed more on frequent symptoms such as diarrhea and fever, whereas greater variability was observed for rarer symptoms like rash and nausea. The effect of prompting strategy varied across models, with no single strategy consistently outperforming the others. Although some pairwise differences reached statistical significance (P <.05), the absolute gains in F1-score were small.

Conclusions:

This study provides a systematic comparison of several open-source LLMs on a structured intestinal symptom extraction task. Among LLM families, Qwen3 models offer a favorable balance between accuracy and efficiency, making them suitable for resource-constrained scenarios.