A total of 33 clinicians (including 2 nonphysicians) were recruited from 3 hospitals (University Hospital Ghent, AZ Sint-Lucas Ghent, and AZ Oudenaarde) and 2 general practitioner circles (Huisartsenkring Schelde en Leie and Huisartsenvereniging Ghent). Two physicians did not complete the evaluation due to technical login issues (n=1) or time constraints (n=1). Half of the group (n=16, 51.6%) were hospital-based specialists, followed by general practitioners (n=13, 42%), with the remainder consisting of other health care professionals (n=2, 6.5%; Table 1). All participants evaluated 30 summaries from the respective medical discharge letters. More details on the number of years of clinical practice for each participant type can be found in Appendix 1.2 in Multimedia Appendix 1.

A workshop on user interface and user experience design was conducted on October 23, 2024, at Ghent University Hospital. The primary objective was to explore how an evaluation tool for LLM-generated summaries should be structured from a physician’s perspective. A total of 26 participants attended, representing all participating organizations. Feedback collected during the workshop was synthesized and shared with technical implementers to finalize the design of the evaluation tool. The user interface was tested by 2 individuals, one data scientist (MD) and one physician (HR). A subsequent test session involved 5 additional technical participants and 3 physicians. These sessions focused on usability, clarity, and alignment with clinical workflows.

The evaluation phase timeline is illustrated in Figure 1. A total duration of 6 weeks was allocated for the completion of the evaluation. From April 1 to April 30, 2025, initial testing of the evaluation tool was conducted by 2 physicians and 1 data scientist. A pilot demonstration was held for the Data Science Institute from Ghent University Hospital on April 29, 2025. Following feedback from this session, a slide-based walkthrough was developed to clarify procedural steps prior to demonstrating the tool. An online presentation for (local) coordinators at participating institutions took place on May 5, 2025; participants received a manual with screenshots and keyboard shortcuts. All coordinators evaluated the tool over a week and reported no major issues.

A broader online demonstration for all participants occurred on May 15, 2025, and was announced via newsletter and email. This session included login instructions for the secure platform and a comprehensive walkthrough of the evaluation tool. For those unable to attend, a recording of the session was made available. On May 19, 2025, a follow-up feedback session addressed questions, which were subsequently compiled into a frequently asked questions document distributed to all users. The formal evaluation phase spanned from May 15 to June 15, 2025, with an extended period continuing until June 30, 2025. Subsequently, a brief demonstration was presented by a general practitioner during the FRAIT symposium on October 22, 2025.

Several educational resources were provided for physicians, including an extensive manual with sections such as examples of potential questions, a live demonstration with an available recording, and a dedicated question-and-answer session. Additionally, during the evaluation phase, support was offered via email and telephone during working hours.

The evaluation was conducted using a standardized prompt derived from the FRAIT questionnaire [2]. This prompt was designed to capture essential clinical and structural elements of discharge summaries across predefined categories: general, medical history, investigations, hospital course, medication, follow-up, and conclusion. An overview of the number of questions per category is provided in Appendix 2.1 Multimedia Appendix 2.

Thirty expert-curated synthetic Dutch medical discharge letters were generated according to the protocol outlined in Appendix 3 in Multimedia Appendix 3. For the purpose of this study, we used the term synthetic clinical letters to describe synthetically modified clinical documents derived from real patient records that were systematically transformed to prevent reidentification while preserving clinical structure and plausibility. This process included complete removal of all direct and indirect personal identifiers, temporal shifting of dates, and targeted modification of clinical content (eg, laterality, diagnoses not relevant to the clinical scenario, and physiological and laboratory parameters) where deemed necessary. To minimize the risk of medical logic drift introduced by shifting dates, all synthetic letters were manually generated by 1 physician and independently reviewed by a senior physician to ensure that clinical decisions, treatments, and guideline-based care remained historically plausible despite the temporal adjustments. The resulting letters do not correspond to any real individual but retain realistic clinical narratives suitable for evaluation purposes. All data were processed to eliminate any risk of patient reidentification, and the approach was reviewed and approved by the relevant institutional committees (reference ONZ-2024‐0273 and ONZ-2024‐0304). These letters served as the source material for summarization and subsequent evaluation. The LLM used for this task was GPT-4o, configured with a temperature setting of 0 to minimize interpretative variability. In this phase, one single prompt was used to be able to evaluate the same results.

The evaluation questions were organized into 4 categories (Figure 2). Layout questions (n=61) were binary and assessed whether the specific sections requested in the prompt were present in the generated summary (for instance, evaluators were asked: “Show the General section in the summary. Is the requested section included?”). Additionally, these questions assessed the format in which sections should be presented, such as bullet points or plain text. Content questions (n=33) examined the correctness and completeness of individual items. Responses could be yes or no, with the latter requiring further classification as incomplete, too much or irrelevant information, or incorrect information or hallucination. When answering no, evaluators were required to indicate the erroneous content in either the source or the summary. An example question was “Show the Admission Date. Is the content correct? If not, indicate what is incorrect.” In addition, the evaluation framework comprised 4 distinct categories of items: layout, content, global scoring, and an open question. Layout items assessed compliance with prompt-specified structural requirements and were scored as binary (yes or no). For each layout item, participants were able to view the original prompt and asked to evaluate whether the LLM-generated summary adhered exactly to the requested formatting and structure (eg, presence of specific section headings, use of bold formatting, bullet lists with a maximum number of items, or mandated section titles such as “medication”). Strict adherence was required for layout items to reflect real-world prompt-based use of LLMs in clinical documentation workflows.

Content items evaluated the correctness and completeness of the generated summaries relative to the original clinical text and were also scored binary (yes or no). When content was rated as incorrect (no), participants were required to assign one or more predefined error categories: missing information (clinically relevant information present in the source text but absent from the summary), irrelevant details (information present in the source text but not requested or inappropriate for the section, including excessive or misplaced detail), or hallucinations (information not present in the original clinical text).

The global score question (n=1) asked evaluators to provide a global rating of the summary quality on a 4-point scale: very bad, bad, good, and very good. Finally, an open question (n=1) invited evaluators to add comments, provide suggestions, or note blind spots in the original letter that were not captured by the prompt.

This structured approach ensured a comprehensive assessment of both structural fidelity and semantic accuracy. Figure 2 provides an overview of the evaluation question types. The set of evaluation questionnaire, including all layout and content items, is provided in Appendix 2.2 in Multimedia Appendix 2.

A sensitivity analysis was performed comparing results from all raters with those obtained exclusively from the physician cohort. Further details can be found in Multimedia Appendix 4.

Responses to the open-ended question were analyzed manually and summarized thematically to identify recurring topics and suggestions.

This study involved the evaluation of LLM-generated summaries of synthetically modified discharge letters. All procedures complied with institutional and national regulations governing the secondary use of clinical data for research.

The workshop generated numerous suggestions for enhancing the FRAIT evaluation tool. A comprehensive list of implemented features is provided in Appendix 5.1 in Multimedia Appendix 5. Examples include keyboard shortcuts for navigation, progress indicators, autosave functionality, and text highlighting. Figure 3 illustrates the main components of the evaluation interface, while additional screenshots and examples of the content question user interface are included in Multimedia Appendix 5.

Across all layout-related questions, 88% of responses were rated as “yes,” with the highest proportion observed in the conclusion section. For content-related questions, 78% of responses were affirmative, with the medical history category achieving the highest score. In contrast, the medication section demonstrated the lowest positive response rate and the highest proportion of hallucinations. Overall, hallucinations accounted for 3% of content-related responses. When content questions were answered “no,” missing information was the predominant reason, particularly in the conclusion section. The general section contained the largest proportion of irrelevant details (Table 2).

Nearly 70% of participants rated summaries as good or very good. A detailed visual overview is presented in Appendix 6 in Multimedia Appendix 6. Figure 4 displays an ordered heatmap of global scores (1-4) by participant and discharge letter.

Mixed effect ordinal regression results are summarized in Table 3. The proportion of “yes” responses for content questions was significantly associated with higher global scores (β=.079; P<.001), whereas layout performance showed no significant effect. Participant specialty did not exhibit a significant influence on scoring.

Consensus for layout questions was high, with a median of 96.77% (IQR 90.22‐100.00). Content questions showed lower agreement (median 83.87%, IQR 67.74‐96.77). The overall consensus of both categories was 93.55% (IQR 80.65‐100.00; Appendix 7 in Multimedia Appendix 7). When examining the items in detail, consensus levels were substantially lower for the no categories (Appendix 7 in Multimedia Appendix 7, Figure 7.2 in Multimedia Appendix 7).

Category-level analysis showed that the medication section had the lowest consensus for both layout and content, while the conclusion section achieved perfect agreement for layout but lower for content (Table 4).

Additional information can be found in Appendix 7 (Figure 7.3 and Figure 7.4) in Multimedia Appendix 7, which presents the consensus for each summary and question.

Pairwise Cohen κ values ranged from 0.07 to 0.56, with a median of 0.36 (IQR: 0.29‐0.43), indicating generally low agreement among raters (Figure 5; Appendix 8.1 in Multimedia Appendix 8). Participant Z31 demonstrated notably lower agreement compared to others. A Dunn post hoc test confirmed significant differences for this participant (Appendix 8.2 in Multimedia Appendix 8).

The ICC for all questions combined was 0.945 (95% CI 0.942‐0.948). Layout questions achieved slightly higher ICC (0.961, 95% CI 0.958‐0.963), while content questions treated as binary yielded 0.908 (95% CI 0.899‐0.916). Excluding participant Z31 did not produce a significant difference in the ICC (Appendix 8.3 in Multimedia Appendix 8), so we continued including this participant in our analysis.

The ICC for global scores was 0.776 (95% CI 0.649‐0.875). Fleiss κ for content questions was lower (0.221, 95% CI 0.205‐0.237), with category-specific values ranging from 0.198 to 0.240 (Table 5).

Further category-specific ICC analysis (Table 6; Appendix 8.4 in Multimedia Appendix 8) confirmed patterns observed in descriptive consensus measures. Interrater reliability was excellent across most sections, with median ICCs around 0.92 for the majority of items. The highest reliability was observed in the medical history, follow-up, and conclusion sections, where ICCs reached up to 0.96. The medication section showed comparatively lower, but still substantial, reliability for content (ICC=0.83). Notably, the conclusion section had a lower ICC for layout (0.65), indicating more variability among raters in assessing structural aspects of this section.

Further inspection of the medication section showed that lower agreement among raters was partly associated with ambiguous prompt wording, particularly instructions such as “show temporary medication.” These ambiguities led evaluators to apply different interpretations of what information should be included in the summary, contributing to reduced consensus even in cases where the model output itself was not incorrect. This indicates that both the characteristics of the model output and the clarity of the prompt influenced the observed agreement patterns in this section.

The PI revealed substantial differences between layout and content items. Layout questions exhibited a high degree of prevalence skew, with an overall PI of 0.768 and a mean PI of 0.856 (SD 0.211). Several categories showed PI values above 0.90, indicating that layout-related items were almost always rated identically (typically “yes”), resulting in limited variability across raters. Content items demonstrated more balanced response distributions, with an overall PI of 0.550 and a mean PI of 0.622 (SD 0.316). Category-specific PI values ranged from 0.430 (medication) to 0.720 (medical history), reflecting greater heterogeneity in item difficulty and a broader range of rater judgments.

A detailed overview of PI values per category is presented in Table 7.

These findings confirm that layout items were highly homogeneous, while content items displayed more meaningful variability. Importantly, the high PI values for layout items help explain the discrepancy between the relatively low median Cohen κ and the high ICC: skewed response distributions can suppress κ, whereas ICC remains elevated because it is driven by between-item variance rather than categorical balance.

Finally, responses to the open-ended question were analyzed qualitatively and summarized by topic (Multimedia Appendix 9).

This study examined expert evaluation and interrater agreement in the assessment of LLM-generated discharge summaries, using the FRAIT framework, to assess structural fidelity, semantic accuracy, and usability.

Regarding the evaluation of structural integrity, layout adherence was consistently high, whereas content accuracy, particularly for medication details, remains a critical challenge. These findings align with prior research on clinical text summarization, which identifies medication and follow-up instructions as frequent sources of error and hallucination [9,10]. Although layout adherence was 88%, the remaining 12% reflects structural inconsistencies that may hinder automated processing and pose challenges for electronic health record integration. Even infrequent deviations from required section formatting can disrupt downstream workflow steps, indicating that layout performance should be considered a potential vulnerability rather than a strength.

The strong association between the positive response rate for content and global scores underscores the importance of semantic integrity in clinical documentation. Evaluators prioritized correctness over format, reinforcing FRAIT’s emphasis on clinically meaningful metrics rather than purely structural criteria. Future iterations should therefore incorporate automated validation of medication-related information. This also aligns with the context-aware evaluation by Agrawal et al [6].

Interrater reliability analysis revealed high ICC values for global scores but low pairwise agreement for individual questions, especially in complex categories such as medication. Our PI findings confirm that the discrepancy between item-level κ and overall ICC is partly attributable to prevalence skew in layout items, rather than true inconsistency in evaluator behavior. This suggests that while evaluators share similar global impressions, item-level judgments are subject to interpretation variability, a challenge also noted in previous discharge summary studies. A deeper analysis of the medication section revealed that ambiguous prompts (eg, “show temporary medication”) contributed to low agreement.

The medication section warrants particular attention, as it exhibited the highest hallucination rate (7.8%) among all content categories. A deeper review showed, however, that most of these hallucinations were related to start, stop, or change medication status rather than incorrect drug content. For example, in one case, vitamin D was listed on admission but not repeated in the discharge medication; the model labeled it as “unchanged,” which some raters considered a hallucination because no corresponding entry was present. This section also showed the lowest positive response rate and weakest interrater consensus, reflecting both its clinical complexity and the fact that some prompt instructions were interpreted differently by different raters. These findings highlight that LLM-generated medication summaries cannot be relied upon without human validation. When using this model and prompt configuration, human-in-the-loop verification remains essential to ensure clinical safety, and future implementations should include targeted safeguards, such as automated checks for omissions and medication-related inconsistencies. This highlights the need for clearer, context-specific prompts to optimize both LLM output and evaluation consistency, as suggested by Borse et al [11]. Follow-up workshops were organized to refine prompt design and ensure unambiguous interpretation, which is essential for improving interrater agreement.

The PI analysis clarifies the discrepancy between the moderate median Cohen κ (0.36) and the high ICC (0.945). Layout items showed consistently high PI values, indicating minimal variability and near-uniform “yes” responses. Such skewed distributions suppress κ (the prevalence paradox [12]) while inflating ICC, which is driven by between-item variance rather than categorical balance. Content items showed lower and more variable PI values, reflecting a more balanced mix of responses and greater item-level complexity. This explains why κ was lower for content judgments: raters disagreed more frequently on clinically nuanced items, although overall rating patterns remained consistent enough to keep ICC high. Together, these findings show that the high ICC partly reflects the high prevalence of affirmative responses in structural items rather than purely strong interrater consistency. Reporting PI alongside κ and ICC therefore provides a more accurate basis for interpreting agreement patterns.

Usability enhancements generated during the workshop, such as keyboard shortcuts, progress indicators, and autosave, reflect user-centered design principles and are expected to improve efficiency and reduce evaluator fatigue. These improvements support FRAIT’s goal of creating a scalable, reproducible evaluation environment for clinical LLMs.

An additional observation concerns infrastructure and security considerations. Although these aspects were outside the scope of this article, they represented the largest portion of the project’s budget. Establishing a secure, cloud-based environment was essential to ensure compliance with data protection standards and enable safe collaboration. This underscores the resource-intensive nature of deploying LLMs in clinical settings and highlights the importance of planning for robust infrastructure early in implementation.

While our findings confirm earlier reports of promising outcomes for LLMs in clinical documentation [9,10,13], infrastructure and human factors remain critical determinants of success. Addressing these challenges will be essential for scaling LLM-based solutions in health care.

A key limitation of this study is the exclusive use of a single standardized prompt and a single LLM (GPT-4o). Although GPT-4o is recognized for its advanced capabilities, our findings should not be generalized to other model families, including open-weight systems or medical-specific models, nor to alternative prompting strategies. While this controlled design was necessary to ensure consistent comparison across evaluators, it also restricts the generalizability of our findings. LLM performance evolves rapidly, and outputs can vary substantially across models, model versions, and prompt formulations. As a result, this study should be interpreted primarily as a validation of the FRAIT evaluation framework, rather than a definitive benchmark of LLM summarization performance.

As our task is clinical summarization, it is plausible that smaller, domain-specific models trained on medical corpora could outperform general-purpose models on factuality, medication handling, and terminology precision while also reducing hallucination risk and computational overhead. The growing adoption of domain-specific models, such as Meditron and HuatuoGPT, which have demonstrated strong performance in clinical contexts [1], suggests that specialized models may offer advantages in higher positive response rates and interpretability that general-purpose models cannot match. Furthermore, larger general-purpose LLMs may introduce unnecessary complexity and increase the risk of hallucinations compared to smaller, clinically optimized models, as noted by Lehman et al [14]. These considerations highlight the need for comparative studies involving multiple LLMs to better understand the generalizability, safety, and optimal balance between model size and specialization in health care applications. We did not test such models here. Therefore, our results may underestimate the attainable accuracy and safety for targeted deployments.

Our evaluation used expert-curated synthetic discharge letters, developed to maintain clinical plausibility while omitting direct identifiers. The original section structure and general layout were largely maintained; however, the transformation process may have reduced certain real-world irregularities such as incomplete phrasing, extraneous noise, or ambiguous references that commonly occur in routine documentation. This distribution shift may have resulted in inflated performance metrics, particularly regarding layout adherence, compared to those expected from more variable and unstructured real-world notes, and may similarly overstate content accuracy. Therefore, our results should be regarded as an upper bound established under controlled conditions, rather than indicative of field performance.

The free-text explanations accompanying “no” responses could not be systematically analyzed because many raters did not follow the intended instructions for marking errors. As a result, the raw annotations were inconsistent in format and therefore unsuitable for reliable qualitative or quantitative reporting. Although the rater pool included clinicians from different specialties, it may not fully capture the diversity of clinical perspectives. Variability in interpretation between GPs and hospital physicians, especially for nuanced categories such as medication details, could have influenced agreement scores and overall evaluations. Ensuring broader representation across specialties and experience levels would strengthen the reliability of future assessments.

Certain sections of the discharge summaries, such as the conclusion, contained limited data. This imbalance restricted the robustness of evaluation for those components and may have skewed the overall usability and accuracy metrics. Future research should aim for more balanced datasets to enable thorough analysis across all content categories. Furthermore, as multiple layout categories exhibited PI values exceeding 0.90, certain reliability measures, especially κ, should be interpreted with caution because high prevalence can restrict their capacity to distinguish genuine agreement from systematic response patterns.

Analysis of pairwise Cohen κ identified one rater whose agreement consistently differed from that of the other evaluators. A post hoc review indicated that delays in initiating and completing the evaluation tasks may have contributed to this variability. This finding highlights the importance of voluntary participation and adequate protected time for expert evaluation to support data quality integrity in future studies. Notably, overall interrater reliability remained high despite this variability, suggesting that the evaluation approach was robust under routine clinical conditions.

A reliable measure of time-on-task could not be obtained because participants were free to pause or take breaks while completing evaluations. Although the platform recorded start and end times, these time stamps do not reflect actual working time and therefore cannot be used to assess efficiency. This limits our ability to draw conclusions about the workflow impact of the evaluation process.

The findings of this study have several practical implications for the integration of LLMs into clinical workflows. While LLMs show promise in assisting with documentation tasks, they should not replace expert review. Human oversight remains essential to ensure patient safety and compliance with clinical standards. Automated quality assurance mechanisms, particularly those targeting omissions and hallucinations in critical areas, such as medication and follow-up instructions, will be vital for real-world deployment.

Prompt design emerged as a key factor influencing evaluation consistency. Ambiguous or poorly defined prompts contributed to variability in interrater agreement and model output quality. Future research should therefore prioritize the development of clear, context-specific prompts and explore adaptive prompt optimization strategies to enhance reliability.

Model refinement is another important area for future work. Comparative evaluations of multiple LLMs, including domain-specific models, are needed to identify configurations that balance positive response rate, interpretability, and computational efficiency. Additionally, longitudinal studies should assess the real-world impact of LLM-assisted documentation on clinical workflows, patient outcomes, and resource allocation. These investigations will provide critical insights into the scalability and sustainability of LLM integration in health care.

To ensure standardization, the evaluation was restricted to a single medical prompt, despite participants having previously developed personalized prompts in a separate workshop [2]. As emphasized by Verma et al [8], the value of summarization is maximized when outputs are tailored to the specific needs of different clinical roles and contexts. Customized summaries, adapted for specialists, general practitioners, or administrative staff, ensure that each user receives the most relevant and actionable information for their responsibilities. This approach not only improves usability but also supports safer and more effective integration of AI tools into diverse health care environments.

Building on recent insights from Croxford et al [7], future research should explore the potential of LLMs not only as tools for generating clinical documentation but also as evaluators, or “judges,” of their own outputs. They propose that context-aware LLMs, when properly calibrated and validated, could assist in the assessment of clinical summaries by providing rapid, scalable, and consistent feedback on accuracy, completeness, and relevance. Integrating LLMs as judges within expert-driven frameworks such as FRAIT may enhance the efficiency of evaluation processes, support continuous quality improvement, and facilitate benchmarking across diverse clinical scenarios. However, this approach will require rigorous validation to ensure that automated judgments align with clinical standards and expert consensus and that potential biases or limitations are systematically addressed.

The FRAIT framework demonstrates strong potential for evaluating LLM-generated discharge summaries, offering a structured approach that balances usability and clinical relevance. While layout fidelity was consistently high, content accuracy, particularly for medication details, remains a critical challenge. Interrater reliability findings highlight the need for clearer prompts and standardized evaluation criteria to reduce variability. Usability enhancements developed through co-creation with clinicians further support FRAIT’s scalability and practical adoption. Overall, LLMs show promise as an aid in clinical documentation, but robust validation mechanisms and expert oversight are essential to ensure safety and accuracy. Future work should focus on prompt optimization, model refinement, and automated quality checks to advance reliable integration of LLMs into health care workflows.