JMIR Publications

ABSTRACT

Background:

Hospital new drug inclusion reviews require time-intensive synthesis of regulatory information, clinical evidence, comparative effectiveness and safety, and institutional formulary considerations. Large language models may support this process, but their performance and limitations within pharmacist-led formulary workflows remain insufficiently evaluated.

Objective:

This study aimed to develop a standardized pharmacist-led, artificial intelligence (AI)-assisted workflow for hospital new drug inclusion review and evaluate its scoring agreement with historical expert assessments, domain-level score differences, AI output inaccuracies, operational efficiency, and pharmacist acceptability.

Methods:

We conducted a retrospective paired simulation study of 161 new drug applications reviewed at the University of Hong Kong–Shenzhen Hospital from January 2023 to December 2025. DeepSeek V3.2 was used within a standardized four-stage workflow incorporating structured prompts, historical evidence cutoffs, and pharmacist verification. AI-simulated and historical expert scores were compared using Pearson correlation and a two-way mixed-effects, single-measure, absolute-agreement intraclass correlation coefficient (ICC[A,1]). Domain-level differences were evaluated using paired tests with Bonferroni correction. Verified AI output inaccuracies were classified using a predefined four-category taxonomy. Operational measures were derived separately from system logs and a post-implementation questionnaire completed by 17 clinical pharmacists.

Results:

AI-simulated total scores were strongly correlated with historical expert scores (r=0.875, P<.001), with good absolute agreement (ICC[A,1]=0.863, 95% CI 0.801–0.904). After Bonferroni correction, AI scores were lower for clinical recommendation (mean difference 1.46, P=.005) and safety (mean difference 1.02, P=.002), while the other five domains showed no significant differences. At least one verified AI output inaccuracy occurred in 93 of 161 reviews (57.8%); evidence distortion/outdatedness was the most frequent category (39/161, 24.2%), followed by factual fabrication (32/161, 19.9%), calculation deviation (29/161, 18.0%), and logical association error (25/161, 15.5%). System logs showed a mean AI processing and draft-generation time of 35.2 (SD 11.3) minutes. In the pharmacist questionnaire, the most commonly selected overall review-time categories were 16–24 hours for traditional manual review and 2–4 hours for the AI-assisted workflow; 15 of 17 pharmacists (88.2%) supported routine implementation, although further optimization was considered necessary by some respondents.

Conclusions:

The pharmacist-led AI-assisted workflow showed good agreement with historical institutional scoring and was associated with perceived reductions in overall review time, but verified inaccuracies remained frequent. High scoring agreement should therefore not be interpreted as evidence of factual or evidentiary accuracy. AI is better positioned as a structured drafting and standardization tool than as an autonomous evaluator, with pharmacist verification remaining essential for evidence interpretation, scoring, and formulary decision support.