Previously submitted to: JMIR AI (no longer under consideration since Apr 27, 2026)
Date Submitted: Apr 18, 2026
Open Peer Review Period: Apr 27, 2026 - Apr 27, 2026
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating Locally Deployed Gemma 3 and GPT-OSS as Privacy-Preserving Alternatives to Cloud-Based Large Language Models on a National Medical Board Examination
ABSTRACT
Background:
Cloud-based large language models (LLMs), including GPT, Gemini, and Claude, have demonstrated near-human performance in many medical areas. However, their reliance on internet-based inference raises concerns regarding patient privacy, data security, and the potential leakage of sensitive clinical information. On-device LLMs may address these concerns by enabling offline inference on local hardware.
Objective:
To evaluate whether locally deployed LLMs can serve as practical, privacy-preserving alternatives to cloud-based models using a national medical board examination as the assessment benchmark.
Methods:
We evaluated two families of locally deployed models, Google Gemma 3 (1B, 4B, 12B, and 27B parameters; vision-enabled from 4B onward) and GPT-OSS-20B, using 1,200 multiple-choice questions from the Taiwan Pulmonary Specialist Board Examinations, including 1,156 text-only and 44 text-and-image items across 26 categories. A cloud-based GPT-4 Turbo model served as a reference. Models were queried locally via Ollama. Accuracy was analyzed by year and category using repeated-measures analysis of variance with Tukey-adjusted pairwise comparisons.
Results:
GPT-OSS-20B achieved the highest overall accuracy (58-78 correct answers per 100 questions) and significantly outperformed all Gemma 3 variants (p < 0.001), while Gemma 3-27B ranked second. No statistically significant difference was observed be-tween GPT-OSS-20B and GPT-4 Turbo after Tukey adjustment. Larger models showed improved accuracy but longer inference time.
Conclusions:
These findings suggest that selected open-weight LLMs deployed on-device can approach the performance of cloud-based models in structured medical examinations, with trade-offs among accuracy, multimodal capability, and computational efficiency.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.