The MedNLP-CHAT Japanese corpus consists of 226 English-translated patient question-chatbot answer pairs, with 100 allocated for training and judged by experts (gold standard), and 126 for testing [12]. Each pair is annotated with objective labels, namely, medical risk, ethical risk, and legal risk.

Objective labels consist of binary values, where “true” indicates the presence of risk or an inappropriate chatbot response. These annotations are accompanied by explanatory justifications that specify the issues with the response. Conversely, a value of “false” indicates that the response is considered appropriate and has no identifiable risk. Subject matter experts were assigned to determine these values based on these descriptions [12]:

The following are the patient question-chatbot answer pairs labeled by experts, indicating risks in chatbot responses and their reasons in italics:

Four system experiments were conducted to evaluate the proposed approach using GPT-4o, identify the top-performing system, and perform an ablation study (Table 2). Each experiment includes an initial assessment (MA1), a verification assessment (MA2), and a final (consensus) assessment (MA3), tailored to each system.

The baseline system established a foundation for assessing different systems and identifying key factors that significantly affect performance. We implemented several improvements to address identified limitations, particularly in MA2 and MA3: enhanced prompt design, embedding-based search, and RAG.

A thorough evaluation of the baseline system revealed that the final assessment prompt required refinement. An enhanced prompt was developed to address this issue. In addition, recent studies indicate that embedding-based search and RAG enhance performance by enabling the model to use retrieved context more efficiently when validating QA pairs [22]. The RAG system was also evaluated using external evidence from non-Japanese sources as a benchmark to assess its replicability across countries (Multimedia Appendix 4).

Furthermore, to reduce potential prompt-related bias and ensure fair comparisons across systems, prompt design was standardized. All systems used a common base prompt structure that included task instructions, QA inputs, output format, and risk definitions. Variations between systems were limited to components involved in system configuration, such as the integration of contextual evidence or RAG (Table 2). For the ablation study, systems adhered to the same prompt structure and were designed modularly, allowing the controlled removal and addition of specific components while keeping all other elements fixed. No system-specific prompt tuning was conducted beyond these predefined configurations. Lastly, all systems used fixed prompt structures, with the output generation parameter temperature set to 1. This configuration introduces nondeterministic output variation, such that outputs may vary across repeated runs even when the same prompts are used (Multimedia Appendix 2).

Experiments were evaluated using the macro F₁-score (from the sklearn Python library [Python Foundation]), which reflects balanced performance across risk domains, and joint accuracy, a strict exact-match metric in which a QA pair is counted as correct only when all risk classifications are correct and aligned with the gold standard. These metrics served as the primary indicators of performance, along with CIs for differences in macro F₁-score and joint accuracy computed between systems (RAG vs non-RAG) and across MA phases, regardless of the risk domain (Multimedia Appendix 5). Additionally, we reported accuracy, precision, and recall (Multimedia Appendix 6).

For statistical analysis, the 95% CIs for the macro F₁-score and joint accuracy were estimated using a nonparametric bootstrap. Specifically, the test set (n=126) was resampled with replacement 1000 times, and the 2.5th and 97.5th percentiles were used as the lower and upper bounds of the CIs.

For comparisons between MA phases and between RAG and non-RAG systems (baseline or enhanced prompt), paired performance differences (Δ) were evaluated using the same nonparametric bootstrap. For each bootstrap sample, Δ was computed as the difference between metrics evaluated on the same resampled instances. Differences were considered statistically significant when the 95% CI for Δ excluded 0. The formulas are given below:

Paired macro F₁-score difference (Δ) for MA is defined as follows:

Paired macro F₁-score difference (Δ) for RAG vs non-RAG is defined as follows:

Beyond conducting a risk assessment of chatbot responses and complementing the results, we performed a preliminary clinical AI risk assessment focusing on commonly misclassified chatbot responses across systems. This assessment aimed to evaluate the potential severity of harm to patients if the chatbot responses are followed.

For scope, the clinical AI risk assessment was limited to the medical risk domain, considering only false-positive (FP) and false-negative (FN) cases, and comprised a subset of 5 QA pairs (Multimedia Appendix 7).

With the help of a medical expert, we defined the assessment protocol, guided by the International Organization for Standardization (ISO) 31000:2018 [24], adapted for clinical AI risk assessment. In this study, we define:

The assessment was conducted by a single health care professional—a nurse with clinical experience in patient care and familiarity with patient safety. The annotator was asked to review each commonly misclassified chatbot response in the medical domain and assign a severity level using a 5-point scale (ranging from 1, the lowest, to 5, the highest), based on the definitions provided in the assessment guidelines (Multimedia Appendix 8). The 5 severity levels were interpreted as follows: 1=minimal (discomfort, inconvenience, or an ambiguous term with safe escalation); 2=minor (incomplete wound care and possible minor infection or missed injury); 3=moderate (harm that may require urgent care); 4=major (harm that could lead to emergency admission with a high chance of patient deterioration); and 5=severe (life-threatening or irreversible harm, including omitting vital information without escalation). These categories were defined with the help of the health care professional to reflect increasing potential consequences to patient safety if the chatbot response were followed. Additionally, the annotator was asked to include an explanation for the assigned severity level.

This study did not require participants to undergo any physical or mental interventions, nor did it involve experiments on human participants. As this research did not use any personally identifiable information at any stage, it was exempt from institutional review board approval in accordance with the Ethical Guidelines for Medical and Health Research Involving Human Subjects established by the Japanese government. The patient question-chatbot answer pairs used were publicly available from the NTCIR-18 MedNLP-CHAT corpus [12]. Therefore, this study poses no ethical concerns regarding patient privacy or informed consent.

We evaluated each system using the proposed framework to identify areas with the most substantial improvements. Specifically, we focused on the effectiveness of the MA approach, MPAs, and overall performance across systems and risk domains. The macro F₁-score, joint accuracy, and paired macro F₁-score difference (Δ) were used as evaluation metrics (Table 3 and Multimedia Appendix 5).

We examined the effect of the MA approach to assess how performance changed across the 3 assessments. Across systems, the transition from MA1 to MA2 produced the largest gains. For the paired macro F₁-score difference, all 4 systems improved substantially, with increases ranging from +0.176 to +0.214. The transition from MA2 to MA3 yielded smaller but consistent gains for the enhanced prompt, embedding-based, and RAG systems, while the baseline showed no additional benefit. This suggests that most performance gains are achieved in MA2, with MA3 providing incremental refinement (Multimedia Appendix 5).

In addition to the MA approach, MPA was integrated into the systems, specifically in MA2. Agents were generated with explicit professions (eg, neurologist, ophthalmologist) or as role-based LLM-specialized agents, using credible Japanese sources and EK incorporated into the RAG system.

As shown in the paired bootstrap estimates reported in Table S3 in Multimedia Appendix 5, RAG yielded a paired macro F₁-score increase of +0.037 (95% CI 0.003-0.074) relative to the baseline and +0.054 (95% CI 0.010-0.100) relative to the enhanced prompt. By contrast, improvements in joint accuracy over the baseline were not evident (95% CI –2.9% to 7.7%), whereas gains over the enhanced prompt were small but notable (95% CI 0.3% to 10.6%). These results emphasize the importance of MA2, where MPA and EK are integrated to ensure contextually grounded evaluations.

In summary, the experiment shows that the RAG system achieved the highest average macro F₁-score of 0.800 (95% CI 0.733-0.852) among all systems, along with a joint accuracy of 76 (60.3%) correct predictions across all risk domains out of 126 QA pairs (95% CI 51.6%-68.3%). Per-domain macro F₁-scores were 0.673 (medical), 0.905 (ethical), and 0.821 (legal), indicating improvements relative to the NTCIR-18 MedNLP-CHAT shared task report in the ethical (+0.252) and legal (+0.096) risk domains, while the medical (+0.070) risk domain remained lower than the ethical and legal domains. Additionally, the RAG system’s performance in the medical domain is moderate, with precision and recall of 0.732 and 0.714, respectively. The lower recall suggests a higher rate of FNs, indicating that some unsafe responses were misclassified as safe. By contrast, the ethical domain demonstrates both high precision (0.885) and recall (0.929), reflecting strong overall performance with few FPs and FNs. The legal domain also exhibits high precision (0.862) but relatively lower recall (0.792), indicating a more conservative pattern with a higher incidence of FNs (Multimedia Appendix 6). BERT models with the same split scored lower, with a macro F₁-score of 0.480 (95% CI 0.427-0.532) for BERT (bert-base-uncased) and 0.521 (95% CI 0.470-0.570) for BioClinicalBERT, as shown in Table 3 and Multimedia Appendices 5 and 10.

Lastly, we conducted a preliminary experiment to assess the framework’s applicability using non-Japanese external evidence (Multimedia Appendix 4). The results showed that the framework achieved 72 (57.1%) correct predictions out of 126 QA pairs, compared with 76 (60.3%) correct predictions when using Japanese sources. By contrast, RAG (non-Japanese sources) alone performed poorly, with only 25 (19.8%) correct predictions. The small difference between results obtained using Japanese and non-Japanese sources suggests that the framework’s performance may be influenced more by the structured MA and MPA reasoning process than by reliance on country-specific knowledge sources. The slightly higher performance observed with Japanese sources may reflect closer alignment with the dataset’s annotation guidelines and standard care, as well as the ethical and legal context.

While the proposed approach demonstrates significant potential for risk estimation in chatbot responses, this study has several limitations.

First, the dataset was limited to 226 QA pairs, which were originally in Japanese and released in English by the NTCIR-18 MedNLP-CHAT task organizers. Both the training and test data were manually translated into English by professional translators, and no additional translation was performed during the experiment. We conducted a limited validation step using a machine translation model on a subset of the original Japanese dataset, which indicated that it may generate misleading phrases and terminology [26]. In addition, the test set (n=126) limits the statistical power and generalizability of the results, as ethical (true=8) and legal (true=18) risks are relatively rare and may be sensitive to small changes in the data; therefore, observed improvements should be interpreted with caution. Although the proposed approach shows promising results in the ethical and legal risk domains, the limited number of positive cases prevents strong conclusions about robustness.

Second, the expert generation phase represents an abstraction inspired by primary care referral workflows, in which patient queries are initially triaged before referral to an appropriate specialist. It is intended for experimental evaluation and should not be interpreted as a substitute for clinical decision-making. As specialized agents are generated by an LLM based solely on its internal knowledge, without evidence-based verification, the resulting role assignments, specialization descriptions, and reference information may contain inaccuracies or hallucinations. In addition, the generated specialist definitions were not always limited to general descriptions of the specialty and could incorporate details from the individual patient query. As these case-specific definitions were passed into the verification assessment phase (MA2), this behavior may have introduced case-relevant context beyond pure role assignment. This should be considered when interpreting the framework’s performance. We also conducted a preliminary clinical AI risk assessment in accordance with ISO 31000:2018 guidelines [24]. Only 1 subject matter expert annotator performed the assessment; therefore, interannotator agreement was not measured, and the results may have been influenced by subjective bias.

Third, GPT-4o was the only model used, as other domain-specific models, such as Google’s Med-PaLM 1 and 2, were not publicly available. As an alternative, we tested strong supervised text classifiers such as BERT and BioClinicalBERT; however, their performance was inferior to GPT-4o, and they lacked the capability to implement the proposed approach. Thus, conducting a comparative analysis with a specialized LLM would be important for more accurately benchmarking the framework’s performance. Comparisons between zero-shot LLMs and supervised BERT-based models may not represent a fully competitive supervised baseline, as these models were trained on a small labeled dataset (n=100), which inherently limits their performance. To mitigate this constraint, we applied stratified k-fold cross-validation on the training set and selected the optimal learning rate. The observed performance differences should be interpreted with caution, as they largely reflect differences in data availability and modeling approaches between zero-shot LLMs and supervised classifiers trained on limited labeled data, rather than a definitive indication of model superiority. In addition, the formal experiments used a fixed prompt structure and did not include systematic reproducibility analyses, such as prompt randomization or repeated-run stability testing. Predictions were generated with temperature=1, a nondeterministic output setting, and lower temperature configurations, which may affect output stability, were not systematically evaluated. Accordingly, the reported macro F₁-scores and joint accuracy values may vary across repeated runs. Future work should test different temperature settings, including lower-temperature configurations, and evaluate the reproducibility of the proposed framework more systematically.

Fourth, the scope of the EK used for evidence-based risk estimation was limited to Japanese data sources, including Wikipedia (using keywords such as “Health in Japan”), the Japanese Law Translation Database, and the Ministry of Health, Labour and Welfare (Multimedia Appendix 3). While this focus limits the immediate generalizability of our findings, it does not constrain the approach’s core methodology. The approach can be adapted to other countries, as demonstrated by preliminary experiments using non-Japanese (United States and European Union) data sources, which indicate its usefulness and replicability (Multimedia Appendix 4).

In addition, retrieval diagnostics (eg, the proportion of queries) were not logged or retained during the experiments. Future work should include retrieval logging to enable quantitative evaluation of retrieval relevance and system behavior. However, the improved performance observed with external evidence in the ablation study indicates that the retrieved documents provided valuable contextual information for the verification process. While MA3 relies on the reasoning outputs generated during MA2 rather than directly reevaluating retrieved documents, potential error propagation from earlier reasoning stages cannot be completely ruled out. Future work may explore hybrid designs that combine weighted reasoning with top-k evidence reevaluation to determine whether direct access to raw evidence at MA3 improves robustness.

Lastly, we did not establish new criteria for medical and ethical risks; instead, we relied on the NTCIR-18 MedNLP-CHAT annotation guidelines and labels, which are designed for expert assessment by health care professionals (Multimedia Appendix 11). This approach helps maintain consistency and comparability with existing systems in the NTCIR-18 MedNLP-CHAT shared task.

While the proposed approach demonstrates notable performance improvements over existing systems, it is limited by its reliance on the general capabilities of the LLM and the current scope of the RAG system. Future research should explore the following areas:

First, to validate the approach’s robustness and ensure its suitability for real-world clinical settings, future studies should incorporate larger, more diverse, natively sourced multilingual datasets, as well as include low-resource languages.

Second, future research should explore human-annotated or curated mappings between patient questions and relevant specialists, along with verified definitions and sources, to mitigate the risk of hallucination and improve system reliability. In addition, because the generated specialist definitions may include case-relevant context beyond pure role assignment, future work should conduct a sensitivity analysis comparing case-specific and case-agnostic definitions.

Third, beyond Med-PaLM, developing and validating a specialized LLM for medical risk identification should include the following: (1) understanding medical terminologies by integrating medical-related dictionaries such as Wikipedia medical terms, a publicly available dataset containing 6000 medical terms and explanations [27]; the Manbyo Dictionary, a large-scale dictionary of disease names in which data on symptoms and disease names are extracted from electronic medical records and discharge summaries written by medical staff [28]; and the Hyakuyaku Dictionary, a large-scale drug name dictionary that contains drug-related terms extracted from medical documents and generic names from the KEGG (Kyoto Encyclopedia of Genes and Genomes) DRUG Database [29-65], to enhance semantic and linguistic understanding; (2) enriching the dataset by expanding it with more true-labeled medical risk cases using counterfactual generation to balance risk representation while maintaining medical plausibility; (3) using LLMs to mimic real-world medical scenarios involving professional diagnosis or debates; (4) incorporating enhanced RAG for improved reasoning by integrating similar cases from electronic medical records or electronic health records using SOAP (Subjective, Objective, Assessment, and Plan) notes or other structured medical documentation formats; and (5) enabling a medical agent to handle a patient case and provide a final decision. Collectively, these steps would enable the models to better understand medical language, enrich the dataset, align reasoning with grounded support, and provide a final consensus supported by an appropriate medical agent.

Fourth, developing detailed criteria in the medical and ethical domains would be helpful for evaluating these risks, as well as expanding EK sources and medical-related dictionaries to assess chatbot response risks.

Overall, this study represents a vital step toward establishing technical and safety standards for the next generation of medical AI chatbots.

This study aimed to design, develop, and evaluate a synergistic MA and MPA approach for estimating medical, ethical, and legal risks in chatbot responses using a patient question-chatbot response corpus from the MedNLP-CHAT shared task.

Using the MA-MPA framework, we evaluated the performance of iterative assessments and role-based LLM specialized agents across various system configurations. The results demonstrate that the proposed framework improves risk estimation performance compared with existing systems, especially when combined with EK integrated through a RAG system, as measured by the average macro F₁-score metric.

Specifically, we observed notable domain-specific improvements in macro F₁-score for ethical risk (+0.252) and legal risk (+0.096), while medical risk showed a modest improvement of +0.070. The MA approach showed the largest gains during the transition from MA1 to MA2. The MPA approach performed better when integrated with MA and EK (RAG system), achieving an average macro F₁-score of 0.800 across risk domains.

By contrast, the ablation study showed that MA only achieved higher joint accuracy than the RAG system, whereas RAG achieved a higher average macro F₁-score. These findings indicate a metric-specific trade-off rather than consistent superiority across all metrics. Domain-level FN and FP analyses help explain this discrepancy. In the medical risk domain, MA only produced more FN than RAG, indicating that it missed more unsafe responses, whereas RAG produced more FP, suggesting that the addition of MPA and EK made the system more conservative by increasing sensitivity at the cost of overflagging safe responses. In the legal domain, the RAG system did not consistently improve error balance relative to MA only, indicating that the added components did not uniformly improve legal risk estimation.

Taken together, these findings suggest that the MA-MPA framework with EK improves balanced overall performance, as reflected in the average macro F₁-score, especially in the ethical risk domain; however, the medical risk domain remains challenging and may require more precise reasoning and calibration.

The scope of this study is limited to binary risk classification on a small benchmark dataset and evaluation with a single general LLM. Results within this scope show that a synergistic approach (MA and MPA), combined with EK, improves risk estimation.