We collected 48 common questions from 4 categories of clinical pharmacy work scenarios:

The study used a standardized experimental design, with 3 researchers using identical “inquiry prompts” to question 8 generative AI dialogue systems during the same time period. All models were tested using their publicly available versions as of February 20, 2025; the results reflect a performance snapshot restricted to this timepoint. Each chatbot received 48 inquiry prompts, generating a total of 384 independent response samples. The evaluation was conducted by 6 clinical pharmacists who had successfully obtained clinical pharmacist training certificates after standardized training and had more than 5 years of clinical pharmacy work experience. The evaluation encompassed six dimensions: (1) accuracy, (2) rigor, (3) applicability, (4) logical coherence, (5) conciseness, and (6) universality. These clinical pharmacists’ professional domains covered all disease types relevant to the questions. A double-blind scoring mechanism was implemented for independent evaluation to avoid subjective bias and ensure objectivity and fairness in the assessment process.

Standardized prompting instructions: all questions were input to the model using a standardized format. The core instruction template was to act in the role of a clinical pharmacist. Based on the latest clinical guidelines and evidence-based principles, answer the following question (Specific question description). For prescription review tasks, the following emphasis was added: determine whether this prescription contains errors and provide your rationale. For case analysis and pharmaceutical care tasks, the following emphasis was added: analyze the pharmacotherapy plan for this case and develop a pharmaceutical care plan addressing the following 4 aspects “Indication, Efficacy, Safety, and Adherence.” The complete list of all 48 standardized prompting instructions used in this study is provided in Multimedia Appendix 1.

Questions were collected using stratified sampling, drawing from the following sources: common medication inquiries at a medication consultation clinic in a large Grade A Class 3 hospital in China, real clinical cases, the theoretical assessment question banks for standardized clinical pharmacist training programs of the Chinese Medical Association (CMA) and the Chinese Hospital Association (CHA), the China Medication Therapy Management (MTM) pharmacist training program, and the China Pharmacist Skills Competition. Specific steps are as follows:

Question bank construction: questions from the 5 sources above were categorized into 4 scenarios: medication consultation, medication education, prescription review, and case analysis and pharmaceutical care.

Stratified sampling:

Question screening: the initially screened questions were independently reviewed by 2 senior clinical pharmacists. This process ensured the questions aligned with the assessment objectives, possessed clinical relevance, were clear and unambiguous, and excluded duplicate or outdated items. Ultimately, 48 questions were finalized for inclusion in the study.

Scoring standards were constructed based on evidence-based medical references, including drug package inserts, the latest clinical guidelines, and the Micromedex database. Clinical pharmacists conducted quantitative assessments of response contents according to standard answers and scoring criteria, with accuracy levels represented on a 0‐ to 10-point scale. Unified scoring rules were established, with each question having a maximum score of 10 points, with deductions as follows:

Evaluators were required to select at least one scoring option during the assessment process.

One-way ANOVA was used to compare score differences among the 8 generative AI dialogue systems across 4 task categories (medication consultation, medication education, prescription review, and case analysis with pharmaceutical care). Data satisfied the assumptions of normality through Shapiro–Wilk tests (P>.05) and homogeneity of variance through Levene tests (P>.05), followed by Tukey Honestly Significant Difference (HSD) multiple comparison analysis to identify significant differences. Interrater consistency was calculated using the intraclass correlation coefficient (ICC) through a 2-way random effects model, with ICC>0.75 indicating good consistency.

Following quantitative scoring, the research team (3 investigators) conducted a systematic content review of all 384 AI-generated responses to identify recurrent strengths, typical error patterns, critical limitations, and notable variations across models. The review process followed the steps mentioned below.

Initial screening: investigators independently reviewed all responses, documenting exemplar responses that demonstrated exceptional performance or significant deficiencies in accuracy, rigor, applicability, logical coherence, conciseness, or generalizability.

Theme identification: investigators consolidated preliminary observations and identified recurring themes through discussion (eg, “Errors in Special Device Instructions,” “Lack of Localized Medication Recommendations,” “Failure to Identify Critical Contraindications,” “Overly Technical or Jargon-Rich or Verbose Language,” and “Ignoring Conflicting Gender-Specific Diagnostic Criteria”).

Case selection: for each identified theme, representative response examples demonstrating the issue or strength were selected across different models. Priority was given to cases where quantitative assessment revealed performance variations and where the response content clearly illustrated the qualitative concern.

Content extraction and verification: key content was abstracted from selected cases to ensure descriptions accurately reflected the original response meaning (with critical verbatim excerpts cited where necessary). Final qualitative cases and their analyses were confirmed by consensus within the research team.

These qualitative findings supplement the quantitative results, providing deeper insight into model performance variations and potential risks across different clinical scenarios. They are reported and analyzed in detail in the Discussion section. The methodology flowchart is shown in Figure 1.

This study constitutes a noninterventional evaluation of outputs from publicly accessible generative AI systems. It does not involve direct participation, intervention, or interaction with human participants (patients or healthy volunteers).

According to the policy of the Ethics Committee of Fuwai Central China Cardiovascular Hospital, this type of research (performance evaluation of publicly available AI systems using deidentified question banks and simulated scenarios) is qualified for exemption from formal ethics review application. The study design strictly followed the principles of the Declaration of Helsinki concerning nonbiomedical research.

This study exclusively used rigorously deidentified question texts as input stimuli for the AI systems. It did not involve the use, storage, or analysis of any raw patient data containing personally identifiable information. Therefore, additional informed consent for this secondary analysis was not required.

All question texts input into the AI systems were based on simulated scenarios or deidentified, generic inquiries, containing no real, personally identifiable patient information. The AI-generated output texts produced during the study contained only clinical pharmacy knowledge-related responses and similarly did not involve any private personal data. All research data (question bank, AI responses, and scoring sheets) were encrypted during storage and transmission and were accessible only to authorized researchers.

The 6 clinical pharmacists participating in the assessment received market-standard honoraria commensurate with their professional contribution.

This study did not use any images containing personally identifiable information (eg, faces, unique physical characteristics, and personal details). All results presentation figures (eg, Figures 1-3) were generated based on aggregated statistical data and anonymized ICC values, posing no risk of individual identity disclosure.

AI applications in health care demonstrate enormous development potential while facing numerous challenges [11]. This study, through constructing a 6-dimensional evaluation system, systematically compares the application effects and limitations of current mainstream conversational AI platforms, providing important evidence for further optimization of intelligent pharmaceutical systems.

This study systematically evaluated the comprehensive performance of 8 mainstream generative AI systems across 4 core clinical pharmacy scenarios: medication consultation, medication education, prescription review, and case analysis with pharmaceutical care. The principal findings are as follows.

While the scoring criteria in this study focused on explicit errors (eg, omission of contraindications), they did not detect potentially hazardous information embedded within otherwise correct responses. For instance, an AI model correctly identified a drug-drug interaction but ambiguously advised to “monitor during use” without explicitly stating the need for immediate discontinuation. In pediatric dosing advice, a model correctly recommended a drug dose but failed to emphasize the necessity of weight-based adjustment. Such errors carry a risk of clinical misinterpretation. Therefore, future research may need to develop algorithms for detecting latent risks, such as real-time validation based on databases like the FDA Adverse Event Reporting System (FAERS).

This study conducted a systematic evaluation and comparative analysis of the application efficacy of 8 mainstream domestic and international generative AI systems across 4 core clinical pharmacy practice scenarios by constructing a 6-dimensional evaluation system. The results demonstrate that DeepSeek-R1 outperformed other models in overall performance, exhibiting particularly significant advantages in handling complex case analysis and pharmaceutical care tasks. However, all models exhibited limitations, prominently manifested in lagging knowledge base updates (eg, incorrect instructions for special devices and lack of localized recommendations), insufficient complex decision-making reasoning capabilities (eg, failure to identify critical medication contraindications and special regulatory requirements), sensitivity to prompt instructions, and overly technical terminology in outputs. Interrater reliability analysis revealed substantial disagreement in evaluating the conciseness dimension for complex tasks such as case analysis.

Based on these findings, this study concludes that while current generative AI systems demonstrate significant potential for efficiency gains and value as decision-support tools in clinical pharmacy, their responses still contain non-negligible errors and limitations, particularly at high-risk decision points. Therefore, at this stage, they should be strictly positioned as auxiliary reference tools for clinical pharmacists, not as independent bases for clinical decision-making. Future development should focus on overcoming key bottlenecks, including achieving dynamic knowledge updating and localization adaptation, enhancing reasoning capabilities in complex scenarios, improving prompt robustness and output interpretability, and establishing continuous evaluation mechanisms. Ultimately, safe, reliable, and patient-centered intelligent pharmaceutical care systems should be built through interdisciplinary collaboration integrating evidence-based medicine, ethical norms, and technological innovation.