Previously submitted to: JMIR Medical Informatics (no longer under consideration since Sep 07, 2026)
Date Submitted: Sep 6, 2026
Open Peer Review Period: Sep 7, 2026 - Sep 7, 2026
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Collaborative Weak-Supervision Annotation With Multiple Large Language Models and BERT for Hospital Patient Service Question Categorization: Development and Validation Study
ABSTRACT
Background:
Hospital patient-service questions are short, colloquial messages in which one entry often bundles several requests with unclear business boundaries, making manual labeling costly. As internet hospitals and online patient-portal platforms scale, hospitals need automatic classification for triage, ticket routing, and risk detection.
Objective:
To develop and evaluate a multi–large language model (LLM) weak-supervision annotation pipeline combined with domain-adapted BERT (bert-base-chinese) fine-tuning that constructs a 20-category hospital patient-service label system and a binary outpatient-service classifier from routinely collected, de-identified consultations.
Methods:
We performed retrospective text mining on 145,128 de-identified consultations from a single tertiary internet hospital (de-identification GB/T 39725-2020; IRB approval 2026-E0760). Three LLMs (GLM, Qwen, LLaMA) independently labeled each text; full agreement was treated as a high-confidence signal and disagreement was routed to a fourth model (DeepSeek) and to human review. Twenty business labels were built through model proposal, semantic merging, and human revision. We fine-tuned bert-base-chinese for a binary outpatient-service task and a 20-class task, and evaluated accuracy, F1, AUC-ROC/AUC-PR, agreement (Fleiss Kappa, ICC), and a temporally held-out external-validation set of 4,135 later consultations.
Results:
Of 145,128 valid texts, 57.2% had full tri-model agreement (Fleiss Kappa 0.6205; ICC 0.6329). After review, 136,979 samples (94.4%) received labels. The binary task reached internal accuracy 0.9422/F1 0.9175 and external 0.9453/0.9123. The 20-class task reached internal accuracy 0.9032/macro-F1 0.8141 and external 0.8730/0.7554. Inter-model disagreement flagged unclear category boundaries and served as a quality-control signal for routing, monitoring, and governance.
Conclusions:
A multi-LLM weak-supervision pipeline with BERT yields a checkable 20-category label system and a stable binary outpatient-service classifier from real-world consultations. Multi-class generalization is limited by minority and boundary classes, and cross-institution validation remains pending.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.