The study was designed as a multidisciplinary assessment of ChatGPT-generated treatment recommendations of 6 cases of chronic lymphedema, ranging from ISL stage I to ISL stage 3 of the upper and lower limb (Multimedia Appendix 1). To ensure consistency, all cases were submitted to GPT-4o (GPT-4 Omni) on December 18, 2024, at noon Central European Time by the first authors (AF and CEE). Each case was entered into a distinct, separate chat session to limit session history bias. Only the initial response was considered, and the option to regenerate the response was not used. The responses obtained from GPT-4o were compiled into a PDF and were subsequently evaluated by a panel of resident doctors (considered inexperienced), board-certified doctors (considered experienced), and physiotherapists or advanced practice nurses (APNs) using a 9-point agreement scale [19].

The clinical vignettes were designed with the intention of incorporating the underlying etiology of lymphedema through past medical history, current patient symptoms, and findings from clinical examination. Cases were prepared to be as realistic as possible and to reflect real clinical scenarios. Patients were clearly categorized into the 3 ISL stages, with corresponding indocyanine green studies illustrating the lymphatic patterns consistent with the stage described in each clinical vignette [10]. Furthermore, the cases were illustrated with images adapted from Principles and Practice of Lymphedema Surgery by Cheng et al [20]. A standardized prompt was used for each case and is provided in Multimedia Appendix 1.

The evaluators had access to the original prompts, all 6 clinical cases, and the corresponding treatment recommendations generated by ChatGPT (Multimedia Appendix 1). The web-based questionnaire for subsequent evaluation was developed and distributed using the Research Electronic Data Capture (REDCap; Vanderbilt University) platform [21]. The DISCERN tool, a validated questionnaire for assessing the quality of written treatment information, was adapted to a 9-point agreement scale and then used for the quantitative evaluation of each generated response [19].

In accordance with the instructions provided in the DISCERN handbook [19], questions 1 and 2 (“Are the aims clear?” and “Does it achieve its aims?”) were excluded from the evaluation as they were found to be irrelevant to the defined research question of this study. Two additional questions were included in the evaluation of each case: one concerning correct staging (question 1) and the other regarding overall final agreement with the treatment option recommended by ChatGPT (question 16). In summary, each response was evaluated using 16 questions rated on a 9-point agreement scale, where 1 represents complete disagreement and 9 represents complete agreement. Finally, 3 concluding questions were added to evaluate the overall agreement regarding the treatment recommendation by ChatGPT and the attitude regarding AI use as a clinical decision tool (Multimedia Appendix 2). In addition, each participant was given the option to offer feedback. These open-text comments were reviewed descriptively by the first authors to provide contextual insight alongside the quantitative results. No formal qualitative analysis or coding approach was applied. All available comments were read in full, and relevant observations are summarized in Multimedia Appendix 3.

The survey was conducted from January 7 to March 18, 2025. Resident doctors without prior experience in lymphedema management were recruited internationally through the same broader professional networks as the expert panel, primarily via participating centers in the LYMPH trial, as well as international resident and student networks. Experienced, board-certified doctors specializing in lymphedema treatment were identified and contacted, most of whom had previously participated in the international lymphedema consortium through existing collaborations [22]. A total of 64 board-certified doctors were invited via email to take part in the study, receiving a survey link along with a comprehensive explanation of its objectives and requirements (response rate 24/64, 37.5%). Additionally, 30 physiotherapists and lymphedema APNs specializing in lymphedema treatment were invited (response rate 10/30, 33.3%) through the same consortium and a contact list provided by the Lymphödem Vereinigung Schweiz (a Swiss patient organization) [22,23]. As resident recruitment was conducted via mass email and a preexisting student group, the response rate of invited resident doctors could not be calculated.

Statistical analysis was conducted using R software (version 4.4.3; R Foundation for Statistical Computing) [24]. The primary end point was the scoring of agreement and reliability of ChatGPT-generated treatment recommendations stratified by experience groups. Consequently, participants were categorized into 3 groups: resident doctors, board-certified doctors, and physiotherapists or APNs. For each question, group-wise means and SDs were calculated. The Levene test was used to confirm homogeneity of variances prior to conducting 1-way ANOVA. Depending on the result of the Levene test, either standard ANOVA (for equal variances) or Welch ANOVA (for unequal variances) was used to test for differences between groups. Where applicable, an “overall” group comprising all participants was included for descriptive purposes only and was excluded from inferential statistical testing. A 2-sided P value of ≤.05 was considered statistically significant.

This study was conducted in accordance with the Declaration of Helsinki. On the basis of the fictional nature of the cases and the absence of human subject data, the local ethics committee (Ethikkommission Nordwest- und Zentralschweiz) deemed that no formal ethics approval was required. Participation in the web-based survey was voluntary. All participants were informed about the purpose of the study, and informed consent was implied through completion of the survey. Participants had the option to discontinue the survey at any time without any consequences. All data were collected anonymously. Data protection and confidentiality were ensured in accordance with applicable regulations. No compensation was provided. No identifiable individuals are shown in any figures or supplementary materials.

Staging and diagnostic accuracy (question 1; Q1) were generally rated high (mean 7.3, SD 1.5), with board-certified doctors assigning significantly lower scores than resident doctors (P<.001; Table 1). Similarly, the relevance of the recommendations (Q2) was rated highly (mean 7.5, SD 1.4), with board-certified doctors rating significantly lower compared to resident doctors or physiotherapists or APNs (P<.001; Figure 1; Table 1). A detailed overview of scoring per case is provided in Multimedia Appendix 4.

The clarity of source citation (Q3) and the dating of sources (Q4) were rated very low across all groups, with mean scores of 3.2 (SD 2.5) and 2.7 (SD 2.1), respectively (Figure 2). Interestingly, resident doctors gave both questions significantly lower ratings than the other groups (P<.001). Interestingly, Q5 (objectivity or neutrality of ChatGPT) received an average score of 5.8 (SD 1.8), with no significant differences between groups.

The quality of additional resources (Q6) was rated badly, with a mean score of 3.5 (SD 2.3) across all cohorts. Similarly, the communication of uncertainty (Q7) received moderate scores of 4.1 (SD 2.1), with residents giving the lowest scores across both questions (P<.001).

ChatGPT’s performance in communicating treatment options and possible risks was generally rated positively, with Q8 (how the treatment would work), Q9 (benefits of treatment), and Q10 (potential risks) all receiving favorable scores (mean 6.3, SD 1.8; mean 6.8, SD 1.6; and mean 6.6, SD 1.9, respectively). In contrast, Q11 (consequence of no treatment) consistently scored low (mean 3.5, SD 2.2). Q12 assessed whether the provided information addressed the impact of the treatment on patients’ quality of life. Ratings were moderate (mean 4.7, SD 2.4), with residents rating the responses significantly lower than other groups. Q13 (ChatGPT mentioning that there may be more than one treatment choice) was generally rated positively (mean 6.8, SD 1.9).

Support for shared decision-making (Q14) was rated moderately (mean 5.3, SD 2.3). The overall impression of the quality of responses (Q15) and overall agreement with treatment recommendations (Q16) both received high ratings (mean 6.5, SD 1.6 and mean 6.8, SD 1.8, respectively), with physiotherapists or APNs rating the answers the lowest (P<.001).

Concluding question (CQ) 1 assessed trust in ChatGPT’s responses, CQ2 assessed willingness to use ChatGPT in future clinical work, and CQ3 assessed whether participants would recommend it for clinical decision-making. Resident doctors rated all 3 concluding questions (CQ1-CQ3) significantly higher than board-certified doctors and physiotherapists or APNs. CQ2 showed a similar pattern, with resident doctors at a mean of 6.5 (SD 1.4), board-certified doctors at 5.3 (SD 2.2), and physiotherapists or APNs at 5.3 (SD 2.3). Finally, for CQ3, which assessed whether participants would recommend AI as a clinical decision-making tool, residents again scored highest, followed by board-certified doctors and physiotherapists or APNs (Figure 3; Table 2). All survey comments are listed in Multimedia Appendix 3.

The overall high level of agreement with the ChatGPT-generated treatment recommendations (Q16) indicates broad acceptance across all cases and groups (Table 1). Resident doctors expressed a more positive opinion toward ChatGPT, a tendency that is presumably attributable to their lesser clinical experience and greater openness to digital technologies in medical practice, indicating that the evaluation of ChatGPT may also be influenced by generational factors (Figure 3) [25,26]. Importantly, however, this favorable perception was not uncritical. Residents assigned significantly lower scores than board-certified physicians for source transparency (Q3-Q4; Figure 1). The lower ratings may reflect stricter expectations regarding explicit source citation, differences in grading thresholds, or a reduced ability to recognize implicit sources that experienced clinicians may infer based on prior knowledge [27]. As the study design does not allow discrimination between these explanations, no conclusions can be drawn regarding the underlying cause of this difference. Nevertheless, several resident doctors reported that ChatGPT already serves them as a useful orientation tool, particularly in situations with limited prior familiarity with a condition or treatment options. This underscores the tool’s potential as a rapid decision support or educational aid, as it only uses a few seconds to generate an extensive response [18]. It also highlights the need to integrate the critical use of AI tools into medical education. As residents showed greater willingness to use ChatGPT, targeted training could help prevent misconceptions and ensure these tools complement, rather than replace, clinical reasoning.

However, board-certified doctors demonstrated a lower level of agreement with treatment recommendations provided by ChatGPT, possibly due to their greater clinical experience and higher expectations regarding therapeutic nuances. This may, in part, stem from the perception that AI systems are not yet capable of providing fully personalized treatment algorithms. This viewpoint was supported in the open comments section (Multimedia Appendix 3), where experienced clinicians repeatedly pointed out that well-established surgical interventions, such as LVA and VLNT, were insufficiently addressed. In this study, LVA and VLNT were only briefly mentioned in cases 2, 3, 4, and 6 and only as potential options if compression therapy and liposuction were deemed insufficient. Reconstructive procedures such as LVA and VLNT are mentioned in both the German AWMF (S2k) guidelines and the ISL consensus document [10,11]. While the AWMF (S2k) guidelines suggest considering surgical options only after 6 months of conservative treatment without any lasting beneficial effect, the ISL guidelines refer to microsurgical interventions without specifying any time frame regarding previous complex decongestive therapy [10,11]. However, recent literature suggests that these procedures may also be used earlier or even as a preventative measure, thus challenging the traditional time threshold [28,29]. Given that key papers and guideline discussions on LVA and VLNT were already available well before October 2023, their limited presence in the responses is unlikely to be explained by the model’s knowledge cutoff alone [30]. A more plausible explanation is that ChatGPT defaults to what it has seen most often: conservative management pathways that are widely taught and described, and a generally “play it safe” tendency when asked broad clinical questions [31]. In other words, the model may not be unaware of surgery; rather, it may just not surface it unless the prompt explicitly asks for surgical options or provides more detail about operative candidacy [12]. Nevertheless, newer ideas such as prophylactic LVA may still be underrepresented due to the recency of the evidence. Future work should test whether prompting specifically for surgical options changes the balance and completeness of the recommendations. These findings also emphasize the necessity of incorporating temporal limitations when using AI tools in clinical decision-making processes [7,8,20,28-30,32,33]. Furthermore, the strong emphasis on lymphedema management using conservative treatment options may result from the broader and more established evidence base in this area, which has developed over decades and is therefore more prominently represented in ChatGPT’s training data.

Physiotherapists or APNs expressed greater skepticism regarding ChatGPT’s treatment recommendations, as highlighted in the open comments section. The recommendations were regarded as being too generalized and lacking personalization. It was emphasized that effective lymphedema care necessitates ongoing, individualized assessment, a component that the AI recommendations did not adequately address.

One of the key characteristics of ChatGPT is its reliance on user input—responses are generated solely based on the prompt it receives [12]. As a result, the specificity and clarity of the input significantly affect the quality, depth, and relevance of the output [12]. Shorter prompts tend to produce shorter, less detailed responses. Citations were completely absent in ChatGPT’s responses and consistently rated poorly by participants (Q3; Table 1), demonstrating that the absence of sources significantly limited the reliability of AI-generated treatment recommendation and, consequently, did not earn the physicians’ trust. Furthermore, the absence of information regarding additional sources (Q6), the lack of mention of areas of uncertainty (Q7), and the omission of potential consequences of forgoing treatment (Q11) were also rated negatively (Table 1).

As an LLM, ChatGPT does not inherently verify the information it provides, but instead generates content based on linguistic patterns, which may influence the accuracy of treatment recommendations [12]. Notably, previous studies emphasized the need to explicitly ask for citations, as the model may otherwise omit references or produce so-called “hallucinations,” meaning fabricated or incorrect references [34,35]. As our prompt did not explicitly request sources, thus reflecting the expected real-world use of ChatGPT, the absence of citations was expected. However, this highlights 2 key limitations. First, when users lack the background knowledge to formulate precise prompts, the model’s effectiveness is significantly reduced. Second, users currently have no possibilities to verify the origin or quality of the information provided. Previous studies have shown that such uncritical reliance on unverifiable information has the potential to result in inefficient or inappropriate treatment decisions and, consequently, potentially harm the patient [36]. It is also important to emphasize that ChatGPT does not collect information but rather generates responses based on statistical patterns in its training data, which further limit its interpretive reliability.

ChatGPT may also pose security risks [12]. Of particular concern is the use of false or deliberately misleading prompts, the aim of which is to manipulate the model into generating inaccurate responses [12]. To ensure patient safety, any clinical implementation would require strict regulatory oversight. A more realistic, and potentially safer, alternative could involve embedding ChatGPT within a clinical decision support system using prevalidated clinical data [37]. In this scenario, ChatGPT would function as a predictive model, assisting in decision-making within a strictly regulated and supervised data environment.

This is the first exploratory, cross-sectional study evaluating the agreement, reliability, and decision-making value of AI using LLMs in generating treatment recommendations for lymphedema management in a global, multidisciplinary, and interprofessional panel. Strengths of this study include the sizeable expert panel of 67 doctors and lymphedema therapists from 34 different institutions across 11 countries. A potential limitation of this study is that evaluators were aware that responses were generated by ChatGPT, which may have introduced either positive bias (novelty or expectation effect) or negative bias (preexisting skepticism toward AI). The results should therefore be interpreted as clinicians’ perceptions of AI-generated recommendations rather than a purely objective measure of model performance. The lack of human-generated reference standard represents an important limitation, as it prevents the interpretation of absolute DISCERN scores; these should instead be understood in a relative context across professional groups. Although the 6 vignettes were deliberately designed to reflect common clinical presentations across the ISL stage spectrum across both extremities, they cannot capture the full heterogeneity of lymphedema. Consequently, generalizability to all clinical contexts is limited, and future studies should evaluate a larger and more diverse set of cases. Another limitation of this study is the use of an adapted assessment tool based on the DISCERN criteria. Although the underlying framework is validated, modifications to the item set and scoring scale may have altered the original psychometric properties of the instrument. As resident recruitment was conducted via large email and international student group, the response rate of invited resident doctors could not be calculated. Moreover, all analyses were based on a single, time-stamped response per case and were not controlled for random seed values or multirun variability, as this would have substantially increased survey burden. Furthermore, because the study was conducted using the standard ChatGPT web interface rather than the application programming interface, we were unable to access or control key inference parameters such as temperature, top-p, or system prompts. As a result, these factors were not reported. Finally, the study did not assess how variations in prompt design might affect the consistency or accuracy of the generated recommendations. Additionally, participants did not generate the prompts themselves, which may limit user satisfaction to the generated responses.

ChatGPT has the capacity to provide relevant basic knowledge for lymphedema management and may be particularly useful for less experienced clinicians. However, it has not yet succeeded in replacing expert consultation and offers only limited personalization. Additionally, its constraints regarding source transparency and prompt dependency highlight the need for cautious and context-aware integration into clinical practice.