JMIR Publications

ABSTRACT

Background:

In recent years, the implementation of artificial intelligence (AI) in health care is progressively transforming medical fields.However, there remains a gap between technological potential and practical application, necessitating the establishment of a scientific evaluation system.Despite some existing research beginning to conduct clinical application assessments of generative AI dialogue systems, these efforts are largely limited to testing individual models on single tasks, lacking horizontal comparative analysis across multiple dialogue models and validation of continuous decision chains in real clinical scenarios.As generative artificial intelligence (AI) systems play an increasingly extensive role in the field of Medicine and Pharmacy, we need more research to explore this area.

Objective:

To systematically evaluate and compare the performance of eight mainstream generative AI systems, both domestic and international, across four core clinical pharmacy practice scenarios: medication consultation, medication education, prescription review, and case analysis with pharmaceutical care. This study aims to quantitatively assess their capabilities in addressing common clinical pharmacy practice problems.

Methods:

Assessment questions were systematically extracted from medication consultation clinic records, real clinical cases, and clinical pharmacist standardized training examination databases. Three researchers tested eight different generative AI systems on the same day using standardized "inquiry prompts." A double-blind scoring design was employed, with six experienced clinical pharmacists with extensive clinical backgrounds evaluating the AI responses on a 0-10 scale across six dimensions: accuracy, rigor, applicability, logical coherence, conciseness, and universality. Statistical analysis used one-way analysis of variance (ANOVA) to compare score differences between systems, with multiple comparison tests for significant results, and intraclass correlation coefficient (ICC) calculations to assess inter-rater consistency.Systematic descriptive evaluations of the AI-generated responses were also conducted.

Results:

DeepSeek-R1 demonstrated the best overall performance across all four task categories. Qwen, GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro performed slightly inferior to DeepSeek-R1. Doubao and Kimi showed inconsistent performance, while ERNIE Bot performed the poorest. Comprehensive evaluation indicated that responses from existing generative AI systems still have certain limitations and should be used as clinical reference tools rather than independent clinical decision-making bases. Inter-rater consistency analysis showed good agreement (ICC>0.75) in evaluating medication consultation, medication education, and prescription review. However, the lowest consistency level (ICC=0.70) was observed in assessing the conciseness of case analysis and pharmaceutical care, reflecting significant cognitive differences among raters regarding evaluation standards for these complex issues.

Conclusions:

The DeepSeek-R1 model demonstrates significant potential as a supportive decision-making tool in clinical pharmacy practice. However, overall, the current generative AI systems still require systematic improvement and refinement in their ability to handle multidimensional complex clinical pharmacy problems. Clinical Trial: none