JMIR Publications

ABSTRACT

Objective:

The objective was to assess the viability and usefulness of a locally deployed large language model (LLM) in decision support on pharmacist-led outpatient prescription evaluation.

Methods:

The open-source Qwen3-14B model was implemented in hospital intranet server with the help of the Ollama framework. Structured knowledge base based on drug package inserts was used as main reference that allowed lightweight knowledge augmentation through an exact-match injection. Two-phase crossover design was used: two pharmacists independently evaluated 213 outpatient prescriptions in both unaided and AI-assisted conditions, resulting in paired independent and collaborative evaluation results per prescription. Results of plain AI review and knowledge-augmented AI review models on the same 213-prescription test set were compared to determine the hallucination rates. The reference standard was built as a consensus among experts, i.e., two experienced supervising pharmacists would independently assess all the prescriptions, and disagreements were resolved upon a decision of a deputy chief pharmacist. The accuracy, sensitivity, specificity and the efficiency of review were compared across groups.

Results:

The overall accuracy of the human-AI collaborative group was 97.2% which is much greater than that of the pharmacist-alone group (82.6% ) (P < 0.001). Sensitivity reached 98.4% in the collaborative group versus 54.8% in the pharmacist-alone group and the false-negative rate had dropped from 45.2% to 1.6% (P<0.001). There was no significant difference in specificity among the groups (P=0.289). Augmenting knowledge minimized the hallucination rate of the model from 19.7 percent to 4.7 percent, an absolute reduction of 15 percentage points, which can be expressed as a 76.2 percent relative decrease. Mean time of review per prescription was 51.9 percent lower in collaborative group (as descriptive estimate). Conclusion: A locally deployed LLM, operating through the Ollama framework and supported by a structured drug knowledge base, markedly enhances both the accuracy and efficiency of pharmacist-led prescription review without exposing patient data beyond the hospital network. Such strategy gives healthcare facilities a safe, realistic route towards smart pharmacy decision support.