Accessibility settings

This paper is in the following e-collection/theme issue:

Letters to the Editor (47) Electronic Health Records (1796) New and Emerging Methods and Tools in Surgery and Anesthesiology (76) Artificial Intelligence in Surgery and Perioperative Medicine (97) Clinical Decision Support for Anesthesiology (31) Generative Language Models Including ChatGPT (1644) AI Language Models in Health Care (837)

Published on in Vol 14 (2026)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/112521, first published .
Doctor filling out medical history questionnaire with patient present

Authors' Reply: Interpreting Agreement in AI-Assisted American Society of Anesthesiologists (ASA) Classification

Authors' Reply: Interpreting Agreement in AI-Assisted American Society of Anesthesiologists (ASA) Classification

Authors of this article:

Bo-Han Lin1, 2 Author Orcid Image ;   Cheng-Wei Lu2, 3 Author Orcid Image ;   I-Ching Hou1 Author Orcid Image ;   Tzu-Yu Lin2, 3 Author Orcid Image ;   Chin-E Liu2 Author Orcid Image ;   Chong-Sun Khoi2, 4 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    1College of Nursing and Smart Care Research Center, National Yang Ming Chiao Tung University, Taipei City, Taiwan

    2Department of Anesthesiology, Far Eastern Memorial Hospital, No 21, Section 2, Nanya S Road, Banqiao District, New Taipei City, Taiwan

    3Department of Mechanical Engineering, College of Engineering, Yuan Ze University, Taoyuan City, Taiwan

    4Graduate School of Biotechnology and Bioengineering, College of Medicine and Nursing, Yuan Ze University, Taoyuan City, Taiwan

    Corresponding Author:

    Chong-Sun Khoi, MD, PhD



    We thank the author for the thoughtful comments [1] on our study [2]. We provide supplemental analyses to clarify American Society of Anesthesiologists (ASA) classification agreement and the incremental contribution of Taiwan National Health Insurance (NHI) data.

    First, the original analysis used the unweighted Cohen κ to evaluate the exact agreement between AI-generated and clinician-assigned ASA classifications in routine preanesthetic documentation. This statistic treats any category mismatch as a disagreement while accounting for agreement expected by chance. We acknowledge that the absence of weighting should have been explicitly stated in the Methods.

    Recognizing the ordinal nature of ASA classification, we additionally calculated the linearly weighted Cohen κ, which was 0.893 with NHI data and 0.560 without NHI data.

    We also examined patient-level cross-tabulations. With NHI data, among 57 clinician-assigned ASA III patients, AI classified 50 as ASA III and 7 as ASA II; among 6 ASA IV patients, AI classified 4 as ASA IV and 2 as ASA III. Without NHI data, among clinician-assigned ASA III patients, AI classified 44 as ASA III, 12 as ASA II, and 1 as ASA IV; among ASA IV patients, AI classified 2 as ASA IV, 3 as ASA III, and 1 as ASA II. Among these 63 patients, all 9 disagreements with NHI data involved adjacent categories. Without NHI data, 16 of 17 disagreements involved adjacent categories, and 1 ASA IV patient was classified as ASA II.

    Second, we estimated uncertainty in the difference in unweighted κ using 10,000 patient-level paired bootstrap replicates. In each replicate, 600 patients were sampled with replacement, retaining each patient’s clinician-assigned ASA classification and both AI classifications together. Both unweighted κ estimates and their differences were recalculated in each replicate, accounting for the dependence between conditions.

    The published unweighted κ estimates remain 0.883 (95% CI 0.841-0.946) with NHI data and 0.518 (95% CI 0.412-0.691) without NHI data. The observed difference was Δκ=0.365, with a supplemental 95% percentile bootstrap CI of 0.292-0.440, derived from the 2.5th and 97.5th percentiles of the bootstrap distribution of paired differences. This interval quantifies the precision of the estimated increase and excludes zero, supporting higher overall agreement when NHI data were included.

    These supplemental results clarify both disagreement patterns and uncertainty in the NHI comparison. Nevertheless, adjacent-category discrepancies may still be clinically relevant. The predominance of ASA II patients and small ASA III to IV subgroups limit category-specific interpretation; neither unweighted nor weighted κ establishes equivalent performance across ASA categories. Comparisons with previous interrater studies also require caution because of differences in rater design, clinical setting, case mix, and κ weighting. We appreciate the opportunity to clarify these findings.

    Acknowledgments

    The authors thank the Department of Anesthesiology and the information technology team of Far Eastern Memorial Hospital for their support in system development, data integration, and clinical workflow implementation. The authors also thank the clinical staff involved in preanesthetic consultation and perioperative documentation.

    Generative AI tools, ChatGPT (GPT-5.6 Sol; OpenAI), were used to assist with language editing, manuscript formatting, and response-to-reviewer preparation. All scientific content, data interpretation, statistical results, and final manuscript decisions were reviewed and approved by the authors. No generative AI tool is listed as an author.

    Funding

    This study received no external funding.

    Data Availability

    The data analyzed in this study are not publicly available because they contain sensitive clinical information derived from hospital electronic medical records and Taiwan National Health Insurance cloud data. Deidentified aggregate results are presented in the manuscript. Requests for access to additional deidentified data may be considered by the corresponding author and Far Eastern Memorial Hospital, subject to institutional review board approval, institutional data governance policies, and applicable privacy regulations.

    Authors' Contributions

    Conceptualization: BHL, CSK

    Formal analysis: BHL

    Methodology: BHL

    Writing – original draft: BHL, CSK

    Writing – review & editing: BHL, CWL, ICH, TYL, CEL, CSK

    Conflicts of Interest

    None declared.

    1. Emirkadı H. Interpreting agreement in Al-Assisted American Society of Anesthesiologists (ASA) classification. JMIR Med Inform. 2026;14:e111973. [CrossRef]
    2. Lin BH, Lu CW, Hou IC, Lin TY, Liu CE, Khoi CS. GPT-4o-powered preanesthetic AI: development and validation study. JMIR Med Inform. Aug 25, 2026;14:e83893. [CrossRef] [Medline]

    ‎
    ASA: American Society of Anesthesiologists
    NHI: National Health Insurance

    Edited by Andrew Coristine; This is a non–peer-reviewed article. submitted 22.Sep.2026; accepted 02.Oct.2026; published 21.Oct.2026.

    Copyright

    © Cheng-Wei Lu, I-Ching Hou, Tzu-Yu Lin, Chin-E Liu, Chong-Sun Khoi. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 21.Oct.2026.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.