This retrospective study was approved by the institutional review board of the Dental Hospital of Yonsei University (IRB 2-2024-0075). All procedures were part of standard clinical care, and the committee waived the need for informed consent. To maintain privacy and confidentiality, all data were fully anonymized before analysis. This study did not involve participant compensation.

This retrospective analysis included records of patients who visited the outpatient clinics of the Departments of Advanced General Dentistry (AGD) or Conservative Dentistry between January 2011 and December 2022. Eligible participants were adults aged 19 years or older, who had a recorded chief complaint at the initial visit, and were diagnosed with pulpal or periapical disease by a specialist. Among those who underwent pretreatment periapical radiography with corresponding clinical and radiographic records, 100 cases without previous endodontic treatment were randomly selected. The demographic and clinical characteristics of the included cases are summarized in Table 1. Chief complaints recorded in the clinical records reflect the patient-reported symptom history, whereas objective examination findings reflect the clinical status at the time of presentation.

Four LLMs were evaluated, namely ChatGPT (version 4.0, OpenAI Inc), Gemini (version 1.5 Pro, Google), Bing (Microsoft), and Clova X (version Hyper Clova X, Naver Corporation). All LLM queries were conducted between November 30, 2024, and December 7, 2024. Four prompt types were created by combining 2 variables—language (Korean vs English) and role (doctor vs patient): (1) Korean-doctor, (2) Korean-patient, (3) English-doctor, and (4) English-patient (Multimedia Appendix 1). All LLMs were queried using platform-default generation settings, without any manual adjustments. Clinical cases were provided in a narrative clinical record format, including patient age, sex, chief complaint, reason for visit, and clinical symptoms. Radiographic images were not directly provided to either the LLMs or the human evaluators. However, because radiographic interpretations are commonly documented in routine clinical information, summarized radiographic descriptors could be included within the chart text. Screening performance was compared with that of human evaluators using Korean-doctor prompts reflecting the clinical background of the evaluators. All treatment plan responses were reformatted into a narrative format before evaluation to ensure assessor blinding to the source of the response. English prompts were generated from the original Korean prompts using Google Translate (Google LLC) and reviewed by a domain expert for accuracy. To ensure reproducibility, prompt structure and generation settings were standardized. Persona prompts were adopted to reflect differences in clinical communication between clinicians and patients. A zero-shot prompting design was used to allow standardized evaluation without example-based guidance, thereby improving comparability across models and evaluator groups.

A conservative dentistry specialist with more than 20 years of clinical experience initially reviewed and selected suitable clinical cases for this study based on clinical data and radiographic assessments. The same specialist established the reference standard diagnosis using the full clinical and radiographic records and was blinded to the LLM-generated outputs and the diagnostic assessments of all evaluators. Twelve evaluators then independently assessed the cases using the same clinical information in the patient records provided to the LLMs. The evaluators included 3 AGD specialists with more than 5 years of experience, 3 AGD residents with more than 2 years of experience, 3 endodontic residents with more than 2 years of experience, and 3 senior dental students. Subsequently, the same evaluators assessed the clinical validity and relevance of the treatment plans generated by each LLM.

Symptom-based screening accuracy was evaluated on a 0 to 2-point scale according to pathological concordance and the criteria of the ICD-10 (International Statistical Classification of Diseases, Tenth Revision) [17]. Two points were assigned for diagnoses fully consistent with the reference standard, whereas 1 point was assigned for identification of the correct disease category with minor discrepancies in diagnostic specificity or terminology (eg, correct classification as pulpal disease with an imprecise subtype). No points were assigned for a complete mismatch (Multimedia Appendix 2). Because the doctor prompts requested ICD-10–based diagnoses while the reference standard used American Association of Endodontists terminology, differences in terminology between the 2 systems were handled within the existing assessment criteria, where responses identifying the correct disease category with a minor terminological discrepancy were scored as 1, and those fully consistent with the reference diagnosis were scored as 2. For screening performance analyses, the ordinal scores were binarized, with scores of 1 and 2 classified as positive and a score of 0 classified as negative. Diagnostic metrics were calculated using a one-versus-other scheme, in which pulpal cases (58/100) served as positives and periapical cases (42/100) as negatives for pulpal disease metrics, with the reverse applied for periapical disease metrics. For the human evaluator groups, responses from the 3 evaluators within each group were combined before calculating diagnostic metrics, resulting in an effective observation count of 174 per group for pulpal disease and 126 for periapical disease; 95% CIs reflect this aggregated count. The corresponding 2×2 contingency cell counts (true positive, false negative, false positive, and true negative) for all groups are provided in Multimedia Appendix 3. The clinical validity and relevance of LLM-generated treatment plans were assessed using a 5-point Likert scale [18,19]. Higher scores reflected greater clinical appropriateness and concordance with evidence-based practice. Evaluators independently rated the realism and evidence-based nature of each plan within a real-world clinical context; they were blinded to the LLM outputs (Multimedia Appendix 4).

To analyze differences between groups, the Kruskal-Wallis test was used, with post hoc pairwise comparisons conducted using the Dunn test with Bonferroni correction. Interrater agreement among the 4 evaluator groups was assessed using the intraclass correlation coefficient (ICC). Screening sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and 95% CIs of the LLMs and clinicians were analyzed and compared for each disease. Diagnostic metrics and corresponding 95% CIs were calculated using the MedCalc Diagnostic Test Evaluation Calculator [20]. All statistical analyses were performed using SPSS (version 27, IBM Corp). Statistical significance was set at P<.05.

The case selection and diagnostic classification process is shown in Figure 1. The text-based screening performance of LLMs, measured on a 0 to 2-point scale, was compared across prompt types (doctor vs patient) and languages (Korean vs English; Figure 2). ChatGPT achieved the highest concordance on Korean-doctor prompts (mean 0.98, SD 0.82), with a significant within-model difference between languages (P<.001). Conversely, Clova X recorded the lowest agreement on English-patient prompts (mean 0.23, SD 0.63), with a significant within-model difference (P=.004).

Across all evaluator groups, ChatGPT demonstrated higher mean ranks for treatment plan validity than other LLMs, whereas Clova X showed the lowest mean ranks (Figures 3 and 4). Statistically significant differences in both outcomes were observed among LLMs within each evaluator group.

Symptom-based assessment scores on the 0 to 2-point scale were compared between LLMs and human evaluators using the Kruskal-Wallis test. AGD specialists achieved the highest mean rank (mean 1.10, SD 0.71; 466.81), followed by endodontic residents (mean 0.97, SD 0.79; 425.49), ChatGPT (mean 0.98, SD 0.82; 425.21), and AGD residents (mean 0.94, SD 0.78; 416.07). Senior dental students showed intermediate performance (mean 0.90, SD 0.71; 410.20), whereas Bing (mean 0.77, SD 0.90; 361.55), Gemini (mean 0.74, SD 0.79; 357.28), and Clova X (mean 0.68, SD 0.74; 341.40) demonstrated lower mean ranks (Table 2). There was a significant difference in symptom-based assessment scores among the groups (χ²₇=25.75; P<.001).

Interrater agreement among the 4 evaluator groups, measured by the ICC, was lowest for AGD specialists (ICC 0.579, 95% CI 0.413‐0.704; P<.001). Moderate agreement was observed for AGD residents (ICC 0.774, 95% CI 0.685‐0.841; P<.001), endodontic residents (ICC 0.793, 95% CI 0.711‐0.854; P<.001), and senior dental students (ICC 0.750, 95% CI 0.651‐0.824; P<.001; Table 3). Post hoc pairwise comparisons using the Dunn test with Bonferroni correction showed that AGD specialists scored significantly higher than Clova X, Gemini, and Bing (Multimedia Appendix 5).

Tables 4 and 5 present the diagnostic metrics for pulpal and periapical diseases, respectively. AGD specialists showed the highest diagnostic accuracy among all groups for both pulpal (0.70) and periapical diseases (0.65). Among the LLMs, ChatGPT demonstrated the highest accuracy for pulpal disease (0.65, 95% CI 0.55‐0.74), whereas Clova X showed the lowest accuracy (0.47, 95% CI 0.37‐0.57). For periapical disease, the overall diagnostic performance of LLMs was lower, with ChatGPT achieving an accuracy of 0.57 (95% CI 0.47‐0.69).

Distribution analysis of diagnostic scores demonstrated that AGD specialists had the highest proportion of correct diagnoses at 48% (48/100), followed by endodontic residents at 43% (43/100), AGD residents at 41% (41/100), and senior dental students at 33% (33/100; Figure 5). Among the LLMs, ChatGPT produced the highest proportion of correct responses at 32% (32/100), surpassing that of senior dental students but remaining lower than that of specialists and residents. Regarding incorrect responses, Bing had the highest proportion at 54% (54/100), followed by Clova X at 48% (48/100). An exploratory subgroup analysis comparing patients aged younger than 60 years and 60 years or older demonstrated no significant difference in diagnostic accuracy between the groups (P=.051).

This study compared the performance of LLM-generated outputs with that of clinicians and senior dental students, evaluating the clinical validity and relevance of LLM-derived treatment plans. AGD specialists achieved the highest symptom-based screening accuracy and sensitivity. Among the LLMs, ChatGPT consistently showed the highest performance, with screening accuracy and diagnostic metrics comparable to those of AGD and endodontic residents. However, its relatively low sensitivity for pulpal disease diagnosis represents an important safety limitation, indicating that LLM-generated outputs should not be used for independent definitive diagnosis without expert supervision.

The 0 to 2 points scoring system enabled standardized comparison across LLMs and human evaluators. However, partially correct responses included a broad range of diagnostic differences, from minor imprecision to clinically meaningful misclassification. Therefore, the findings should be interpreted with caution, as this simplified scoring may not fully reflect the clinical impact of specific diagnostic errors. Although AGD specialists demonstrated high screening accuracy, their interrater agreement was relatively low, likely reflecting variability in expert interpretation of ambiguous or incomplete clinical information. This variability does not indicate poorer diagnostic ability but highlights the complexity of decision-making under limited clinical information [21,22]. LLMs did not achieve specialist-level performance; however, ChatGPT showed potential as a supportive tool for less experienced clinicians. Post hoc pairwise analysis further showed that AGD specialists scored significantly higher than Clova X, Gemini, and Bing. In contrast, no statistically significant difference was detected between ChatGPT and any of the human evaluator groups. These results are consistent with our overall finding that ChatGPT performed relatively better than the other LLMs evaluated, while overall performance remained modest. Nonetheless, hallucinations were observed, including fabricated clinical details and treatment plans inconsistent with established guidelines (Multimedia Appendix 6).

Although recent studies have investigated the use of LLMs in conservative dentistry, most have focused on educational settings, patient inquiries, or multiple-choice examinations [12-16] (Table 6). In contrast, this study used real-world data under multiple prompting scenarios. Zero-shot prompts and personas were applied to reflect practical clinical use, where optimized prompting cannot be assumed [23].

Outputs varied substantially by language and prompt type, underscoring the importance of prompt design in clinical contexts [24,25]. Consistent with prior medical studies, role-specific prompts improved clinical relevance, whereas vague prompts reduced consistency [26,27]. The student group rated the LLM-generated treatment plans more positively than experienced clinicians, likely reflecting differences in error recognition ability based on clinical experience. Tangadulrat et al [28] reported similar findings in medicine, where medical students tended to trust ChatGPT’s results more than physicians did. These results suggest that clinical experience is critical for evaluating AI-generated outputs. As summarized in Table 6, this study differs from prior work.

Recent advances in multimodal LLMs, such as GPT-4V and Gemini Pro Vision, demonstrate improved diagnostic performance when radiographic images are incorporated into diagnostic assessments [29,30]. In this study, radiographic images were not directly reviewed, although summarized radiographic findings were available in clinical records. Given the indispensable role of radiographic examination in endodontic diagnosis, limited access to image interpretation may partly explain the lower sensitivity observed for LLMs compared with specialists. Incorporating radiographic image analysis into future dental AI systems may improve diagnostic performance in clinical practice.

Previous studies in conservative dentistry have mainly examined the use of LLMs in educational contexts and multiple-choice examinations. Previous studies by Arılı Öztürk et al [16], Abdulrab et al [12], and Künzle and Paris [31] consistently demonstrated that more advanced models, such as ChatGPT-4 and ChatGPT-4o, showed higher accuracy and stability than earlier versions or other models. Durmazpinar and Ekmekci [32] found that ChatGPT-4o outperformed students in multiple-choice questions. In addition, Nguyen et al [33] observed that ChatGPT performed well in text-based questions but showed lower accuracy in image-based questions.

Similar findings have been reported in studies addressing patients’ clinical inquiries. Aljamani et al [15], Baris and Baris [34], and Büker and Mercan [13] showed that several models achieved high accuracy and quality in answering patients’ questions related to endodontic treatment. However, these studies consistently highlighted important limitations, including higher reading levels, variability in reliability, and the potential for misinterpretation without expert supervision.

This study had several limitations. First, a zero-shot prompting design was used, and follow-up questions were intentionally excluded, which limited the evaluation of iterative clinical reasoning [35]. Consequently, this study could not assess whether multiturn prompting, which allows iterative clarification and the inclusion of additional information, would alter diagnostic accuracy or reasoning performance.

Second, although the English prompts were reviewed by a domain expert, translation nuances may have influenced model interpretation. Notably, per (+) was used instead of the standard English shorthand for percussion sensitivity, and conservative dentist was used instead of the internationally recognized term endodontist. The study was also limited to English and Korean prompts, which may have restricted the generalizability of the findings to other linguistic or cultural contexts.

Third, the analysis relied on clinical record information. Radiographic images were not directly reviewed by LLMs or human evaluators, although summarized radiographic interpretations were available in routine documentation. Diagnostic performance may, therefore, have been underestimated. Accordingly, direct comparability between the reference standard and test conditions was limited. Therefore, it remains unclear whether diagnostic failures reflect limited reasoning capability or simply reflect missing diagnostic data. A single specialist determined the reference diagnosis, and pre-2017 cases were not reclassified under updated criteria. In addition, no formal sample size calculation was performed, as the sample size was determined by the available eligible cases. This may have limited statistical power for some comparisons. Furthermore, 139 of 312 eligible cases were excluded due to multiple concurrent symptoms, which may have resulted in a less clinically diverse set of included cases. Therefore, the diagnostic performance of both LLMs and human evaluators may have been higher than that observed in routine clinical practice, where patients frequently present with overlapping symptoms.

Fourth, doctor and patient prompts differed in both persona and the amount of clinical information provided, as doctor prompts included objective examination findings and radiographic descriptors, while patient prompts relied solely on subjective symptoms. Therefore, it remains unclear whether the observed performance differences were attributable to persona framing or to the difference in available clinical data. Additionally, the doctor prompts requested ICD-10–based diagnoses, while the reference standard used AAE terminology. Where the 2 classification systems do not correspond one-to-one in certain categories, model responses may have been more likely to receive a score of 1 rather than 2, potentially underestimating performance under doctor-prompt conditions.

Fifth, the Kruskal-Wallis test assumes independent samples; however, because all groups assessed the identical set of 100 clinical cases, this assumption is not fully satisfied. This limitation should be considered when interpreting both Table 2 and the post hoc pairwise comparisons in Multimedia Appendix 1. Additionally, the one-versus-other classification scheme assigns each case to a single disease category. It may not fully capture clinically linked dual diagnoses, potentially classifying biologically comprehensive responses as false positives. Furthermore, pooling responses across 3 evaluators within each group may constitute pseudoreplication, potentially narrowing the reported 95% CIs in Tables 4 and 5. In addition, scores of 1 and 2 were both classified as positive in the binary diagnostic performance analysis. Although this approach was intended to reflect clinically meaningful partial agreement, grouping partially correct diagnoses as positive outcomes may have inflated the reported sensitivity and accuracy metrics (Multimedia Appendix 7).

Finally, hallucination and interpretation biases were observed, but qualitative error classification was not performed. Future studies incorporating qualitative analyses and multimodal inputs are warranted for better characterization of safety risks and improved clinical applicability. Despite these limitations, this study provides a comprehensive evaluation using real-world clinical data under diverse prompting conditions.

Future research should evaluate multimodal LLMs integrating text and imaging data under identical clinical conditions. In addition, advanced prompting strategies—including chain-of-thought and few-shot prompting—should be examined to enhance structured clinical reasoning and diagnostic accuracy. Approaches that constrain model outputs using established clinical guidelines or retrieval-augmented generation may help mitigate hallucinations. Moreover, interactive conversational frameworks that enable clarification of missing information may also enhance clinical reasoning and reliability. From a clinical implementation perspective, a human-in-the-loop workflow may be more appropriate than independent model use. Given the relatively high specificity but limited sensitivity observed, LLMs may be better suited for generating differential diagnosis lists rather than for definitive screening. In this workflow, clinicians review model-generated suggestions and integrate them with clinical examination and radiographic findings before making final decisions.

Compared to other LLMs, ChatGPT provides more stable and reliable responses in endodontic diagnosis, demonstrating potential as a supportive tool for clinicians. However, limitations such as hallucinations, low sensitivity, and variability depending on user expertise remain significant challenges to its use as an independent diagnostic aid. For safe and effective clinical application, the development of standardized prompt templates, the implementation of comprehensive user training, and continued clinical supervision will be essential.