In clinical medicine, anesthesia is a fundamental component across a wide range of procedures. However, it is associated with several postoperative complications, among which postoperative nausea and vomiting (PONV) remains one of the most common and clinically significant. PONV can disrupt recovery, increase the risk of secondary complications, prolong hospitalization, and negatively impact patients’ perceived quality of life [1].

Accurate identification and stratification of PONV risk factors are essential for guiding preventive strategies. The overall incidence of PONV is estimated at 27.7%, underscoring its clinical importance [2]. Traditional risk assessment approaches primarily rely on static patient characteristics, such as sex, age, smoking status, and prior history of PONV or motion sickness [3,4]. However, these factors alone may be insufficient to capture the complexity of perioperative risk.

Although PONV includes both nausea and vomiting, these outcomes differ in mechanisms and clinical implications. Therefore, this study focuses specifically on postoperative vomiting within 24 hours (POV 24h) as a clearly defined and objectively measurable endpoint. Existing prediction tools, such as Apfel’s simplified risk score, report moderate accuracy (55%‐80%) [5], but are largely based on fixed patient characteristics and do not account for the temporal availability of perioperative information [6]. As a result, their applicability in real-time clinical decision-making remains limited.

Despite these advances, current prediction models face several key methodological limitations. First, most models do not explicitly distinguish between variables available at different clinical time points, potentially introducing information leakage by incorporating variables not available at the intended prediction time point, thereby limiting real-world clinical deployment. Second, perioperative factors—such as intraoperative physiological changes and anesthetic exposure—are often underrepresented despite their clinical relevance. Third, unstructured clinical text, including medical history and procedural notes, is rarely incorporated into predictive modeling due to integration challenges.

To address these gaps, we propose a temporally structured prediction framework that separates preoperative and perioperative modeling tasks. In addition, we explore the use of large language models (LLMs) in a structured framework that separates feature construction from interpretability to transform unstructured clinical text into structured representations and to support post hoc clinical reasoning, with the primary goal of generating structured explanations and supporting transparent post hoc review, rather than substantially improving predictive performance.

To evaluate this framework, we conducted a large-scale study using 33,460 anesthesia records, incorporating both structured (eg, physiological parameters, surgical, and anesthetic information) and unstructured data (eg, medical history and procedural notes). The objective of this study was to develop and internally validate predictive models for POV 24h. Two models were constructed based on the availability of clinical data: a preoperative model using variables available before anesthesia induction, and a perioperative model using variables available up to the end of surgery. Furthermore, we investigated whether LLM-based processing of unstructured clinical text could be performed while maintaining methodological rigor and supporting clinically grounded reasoning.

To overcome the limitations of conventional models, recent studies have increasingly turned to machine learning (ML) and deep learning (DL) techniques for PONV prediction [7]. These methods are well-suited for analyzing high-dimensional clinical datasets and capturing complex, nonlinear relationships that often elude traditional statistical models [8,9]. Although most prior studies focus on the composite outcome of PONV, vomiting can be assessed as a distinct and clinically measurable postoperative outcome within a defined time frame [10]. Therefore, this study specifically focuses on POV 24h.

Among these approaches, gradient boosting models like LightGBM (Light Gradient Boosting Machine) have been widely used to identify key PONV predictors and assist anesthesiologists in postoperative risk evaluation [11]. Common predictors highlighted across ML studies include sex, smoking status, type of surgical procedure, and use of opioids [12]. DL further advances prediction by modeling intricate interactions among features, with comparative analyses showing that algorithms such as logistic regression, support vector classifiers (SVCs), and AdaBoost frequently outperform traditional scoring systems [13]. Notably, TabNet, a DL architecture tailored for tabular data, leverages instance-level feature selection and attention mechanisms to improve efficiency and model transparency [14].

Despite these advances, the opaque nature of many ML and DL models presents a barrier to clinical adoption. To address this, explainable AI (XAI) techniques such as Shapley Additive Explanations (SHAP) have been introduced to illuminate the decision-making process of complex models. SHAP enables clinicians to assess the influence of individual features on prediction outcomes [15-17]. For instance, Jiang et al [18] demonstrated that incorporating traditional Chinese medicine (TCM) metrics alongside glycosylated hemoglobin levels substantially enhanced the interpretability of ML models in diabetic neuropathy. Similarly, in PONV research, SHAP has revealed previously unrecognized high-dimensional risk factors linked to delayed but clinically significant postoperative nausea and vomiting (CIPONV) [19].

Preoperative assessment and optimization have long been recognized as critical components in surgical care [20,21]. More recently, LLMs such as ChatGPT have been increasingly explored as tools to enhance the interpretability and usage of unstructured clinical text. LLMs have shown promise in preoperative evaluations and have even demonstrated alignment with expert consensus on pharmacologic strategies for PONV management [22-24]. However, current LLM implementations tend to function independently of structured patient data and often overlook patient-specific factors such as age, sex, and clinical history [22,24-26].

Past PONV research has laid a strong foundation for risk factor identification, offering essential insights for future feature engineering and data preprocessing efforts [27,28]. Nonetheless, in today’s fast-paced clinical environments, real-time decision-making is increasingly entrusted to frontline providers. Moreover, ML and DL systems should serve as enhancements to, and not replacements for, human clinical decision-making, especially in real-time environments where clinicians must rapidly interpret patient data [29].

Recent studies emphasize that beyond predictive performance, model interpretability and calibration are critical for clinical deployment. For instance, explainable tree-based models such as ExtraTrees have shown that even models with moderate discrimination require careful evaluation of calibration and feature contributions before clinical adoption, highlighting the importance of aligning model outputs with real-world clinical distributions and decision thresholds [30]. In addition, systematic reviews of clinical AI applications have identified persistent challenges in interpretability, bias, and insufficient validation, particularly in surgical and perioperative decision-making [31]. These concerns further underscore the need for greater transparency and trust in AI systems, as emphasized in broader discussions on AI deployment [32,33].

Taken together, future research should not only focus on improving predictive accuracy but also prioritize interpretable modeling, robust validation (including calibration), and transparency-aware model design, thereby fostering greater clinician confidence and facilitating real-world implementation.

We propose a clinically grounded framework for predicting POV 24h using ML, deep learning (DL), and LLMs. The framework comprises three core modules: data preprocessing, model development, and interpretability analysis (Figure 1). Traditional ML and DL models serve as predictive baselines, while LLM-based techniques support the structured transformation of unstructured clinical data and support post hoc interpretation within the modeling framework.

The primary endpoint was POV 24h, excluding nausea-only events. For each surgical case, multiple postoperative interviews were aggregated at the surgery level. A case was labeled positive (1) if any documented vomiting event occurred within 24 hours, and negative (0) if all available interviews indicated no vomiting. Surgeries without a valid 24-hour follow-up were excluded from the main analysis.

To reflect real-world clinical decision-making, we defined two prediction tasks based on the timing of feature availability: (1) a preoperative model using variables available before anesthesia induction, and (2) a perioperative model using variables available up to the end of surgery. Only variables available prior to each prediction time point were included in model training. Postoperative variables were excluded from all prediction models. To minimize the risk of temporal leakage, all variables were classified according to temporal availability prior to model construction (Multimedia Appendix 1). Variables unavailable at the intended prediction time point were excluded from the corresponding prediction task.

This retrospective study was approved by the Institutional Review Board of Kaohsiung Medical University Hospital (approval number: KMUHIRB-E(I)-20200040). The requirement for informed consent was waived due to the deidentified and retrospective nature of the data. The study was conducted in accordance with the principles of the Declaration of Helsinki.

The dataset was obtained from the hospital's electronic medical records system. Data collection and processing were conducted following institutional review board (IRB) approval (IRB No: KMUHIRB-E(I)-20200040).

The dataset used in this study was obtained from a collaborating medical center and comprises both structured and unstructured data. Structured data includes physiological measurements, anesthetic information, and surgical procedure types. Unstructured data consists of narrative text, such as patient medical history and surgical records.

Structured data preprocessing was performed to ensure data quality and consistency prior to model development. An important characteristic of this dataset is that the hospital information system encodes the absence of an event or measurement as a value of 0 rather than as a conventional missing value (not a number [NaN]). For example, an intraoperative drug dosage of 0 indicates that the drug was not administered, and an airway-equipment size of 0 indicates that the corresponding procedure was not performed. Consequently, the structured feature set contained no conventional missing (NaN) values: across all 78 perioperative model features and the full study cohort (n=33,460), the NaN rate was 0%. A median- and mode-imputation step (for numeric and categorical variables, respectively) was nonetheless retained within the unified modeling pipeline as a safeguard for prospective deployment, where sporadic true missing values may occur; in the present study dataset this step was not triggered because no NaN values were present. All preprocessing procedures were refitted independently within each training fold before being applied to the corresponding validation data, ensuring that any imputation parameters would be derived exclusively from training data and preventing information leakage.

It is therefore important to distinguish these structurally coded zeros from genuine missing data. To characterize their distribution, we summarized the per-variable zero rate across the cohort. Most numeric features were near-completely recorded, whereas a small number of procedure-specific measurements (eg, airway-equipment sizes and selected intraoperative drug dosages) showed high zero rates that reflect non-performance of the corresponding procedure rather than missing information. A detailed per-variable zero-rate summary is provided as a supplementary file (Multimedia Appendix 2). Alternative imputation strategies, including K-Nearest Neighbors (KNN) imputation, were explored during preliminary assessment but did not materially affect downstream model performance and were therefore not adopted in the primary pipeline.

Extreme values were handled using percentile-based capping to reduce the influence of outliers while preserving clinically meaningful variation. Antiemetic administration was retained as a perioperative variable rather than excluded. As this variable may reflect clinicians’ responses to perceived risk and thus introduce potential confounding, prediction tasks were explicitly defined based on temporal availability, with variables included only when available at the time of prediction. The potential confounding effect of this variable was considered during the interpretation of model performance.

The study protocol was reviewed and approved by the Institutional Review Board (IRB) of Kaohsiung Medical University Hospital (approval number: KMUHIRB-E(I)-20200040). All data used for analysis were fully deidentified in accordance with institutional data protection and ethical standards. Direct identifiers (eg, patient names, national identification numbers, and dates of birth) were completely removed. For indirect identifiers (eg, admission time, hospital stay, and diagnosis codes), data masking or generalization techniques were applied to minimize the risk of reidentification.

LLM-based methods for unstructured clinical text processing were implemented using a fixed model version with deterministic settings (temperature=0) to ensure reproducibility. All inputs to the LLM were strictly limited to deidentified text. Direct identifiers (eg, patient names, identification numbers, and exact dates) were completely removed prior to processing. When a commercial API was used, only deidentified text was transmitted through secure, encrypted channels in accordance with institutional data governance policies. No direct identifiers or personally identifiable information were transferred to external services. Alternatively, LLM processing could be conducted within a controlled or self-hosted environment. This design supports reproducible and secure use of LLM technologies within the clinical data pipeline while maintaining compliance with data protection regulations.

Handling unstructured clinical text requires approaches beyond conventional categorical encoding. We implemented a concept-based framework in which narrative text was normalized and mapped to predefined clinical concepts using LLM-assisted semantic standardization. Ordinal encoding was applied separately to ordered categorical variables, such as American Society of Anesthesiologists (ASA) physical status, by assigning ranked numerical values, enabling machine learning models to capture ordinal relationships effectively. However, more complex narrative text requires semantic interpretation that cannot be fully represented through predefined ordinal or categorical structures alone.

To extract clinically meaningful signals from medical history, surgical notes, and diagnostic descriptions, we implemented a keyword-based feature construction approach guided by predefined clinical concepts derived from prior literature and domain knowledge. The LLM was used solely to standardize semantically related expressions (eg, synonyms and linguistic variations) into consistent representations aligned with predefined clinical concepts. The identified concepts were subsequently mapped to structured features using a predefined scoring scheme, ensuring consistent and reproducible feature construction. The overall preprocessing pipeline for unstructured clinical text is illustrated in Figure 2.

Beyond keyword-based feature construction, we further incorporated Question Answering (QA) Chain, an LLM-based framework built on LangChain, as a post hoc explanation module to support clinical interpretation of model outputs. QAChain operates after model prediction by leveraging structured inputs, including model outputs and derived clinical features, to generate context-aware explanations. With a deterministic setting (temperature=0), the model produces consistent, structured explanations aligned with clinical reasoning and grounded in patient context.

For example, it may generate: “The patient received high-dose opioids and underwent prolonged general anesthesia, both of which contribute significantly to PONV risk.” Such outputs provide interpretable narratives that connect model predictions with clinically relevant factors. The QAChain-based explanation process is illustrated in Figure 3. The full prompt template, system instructions, and configuration details are provided in Multimedia Appendix 3 to ensure reproducibility and transparency.

We developed predictive models for POV 24h using a temporally structured dual-task framework aligned with clinical decision-making. Two independent prediction tasks were constructed to prevent information leakage: (1) a preoperative model using variables available before anesthesia induction, and (2) a perioperative model using variables available up to the end of surgery. Only features available prior to each prediction time point were included in the corresponding model.

For structured data, three ML algorithms were implemented: logistic regression, LightGBM, and XGBoost (Extreme Gradient Boosting). Logistic regression served as an interpretable baseline model, while gradient boosting models (LightGBM and XGBoost) were used as primary predictive approaches due to their strong performance on tabular clinical data and their ability to capture nonlinear relationships and feature interactions.

Unstructured clinical text features derived from the preprocessing pipeline were incorporated into the model as structured inputs using a predefined keyword-based scoring system. This approach enabled integration of clinically relevant information from narrative data while maintaining methodological rigor and preventing information leakage.

Model training and evaluation were conducted under a temporal split design to simulate real-world deployment, in which models trained on earlier data were evaluated on unseen future cases. Model performance was assessed using discrimination (receiver operating characteristic–area under the curve and precision-recall-area under the curve), calibration (Brier score and calibration plots), and clinically relevant operating metrics. Classification thresholds were selected using Youden’s J statistic, and decision curve analysis was performed to evaluate clinical utility. All performance metrics were reported with 95% CIs derived from bootstrap resampling.

To enhance transparency and support clinical interpretability, we incorporated both model-based and LLM-assisted explanation approaches. First, model-level interpretability was assessed using SHAP. SHAP values were used to quantify the contribution of individual features to model predictions and to derive global feature importance rankings based on the mean absolute SHAP values across samples. These analyses enabled the identification of key predictors associated with POV 24h.

Second, the keyword-based scoring system provided interpretable representations of unstructured clinical text, enabling examination of text-derived risk factors within the model.

Finally, we implemented QAChain, an LLM-based post hoc explanation module, to generate context-aware natural-language interpretations of model outputs. QAChain was configured with deterministic settings (temperature=0) to ensure reproducibility. Importantly, this module was not used for predictive modeling, but solely to provide human-readable explanations aligned with clinical reasoning.

Together, these approaches provide complementary perspectives on model behavior by combining quantitative feature attribution with qualitative explanation, thereby supporting clinical understanding and potential real-world application.

The study population consisted of patients who underwent anesthesia for surgical procedures at Kaohsiung Medical University Hospital between July 2019 and September 2022. The initial dataset included 79,919 anesthesia records comprising 127 variables and 95,304 postoperative interview records comprising 81 variables. These datasets were merged at the surgery level using a unique surgery identifier, resulting in a combined dataset with 208 variables.

Cohort construction was performed based on predefined inclusion and exclusion criteria to ensure data quality and clinical consistency. Patients younger than 18 years, duplicate records, and cases with invalid or inconsistent timestamps were excluded. Patients with an ASA physical status of 5 or 6 were also excluded due to their extreme clinical risk profiles. In addition, cases without valid postoperative follow-up within 24 hours were excluded from the primary analysis. The cohort selection process is illustrated in Figure 4. (Multimedia Appendix 4)

Prophylactic antiemetic use was retained as a variable and further examined through sensitivity and subgroup analyses. After cohort construction, the final study population consisted of 33,460 surgical cases. Among these, 3607 cases (10.8%) experienced postoperative vomiting within 24 hours (POV 24h), while 29,853 cases (89.2%) did not.

The primary endpoint was defined as POV 24h. For each surgical case, postoperative interview records across multiple time windows were aggregated at the surgery level. A case was labeled as positive if vomiting occurred at any time within the 24-hour postoperative period.

Given the class imbalance in POV 24h outcomes, model evaluation focused on metrics that reflect both discrimination and clinical relevance, including receiver operating characteristic – area under the curve, precision-recall area under the curve, and operating characteristics such as sensitivity and positive predictive value.

We evaluated three ML models (logistic regression, LightGBM, and XGBoost) for POV 24h. Logistic regression served as an interpretable baseline, while LightGBM and XGBoost were used to capture nonlinear relationships and feature interactions in tabular clinical data. The Apfel score was included as a clinical reference baseline.

All models were evaluated using a temporal split protocol, in which models were trained on earlier cases and tested on subsequent, unseen cases. Two prediction settings were defined according to clinical workflow: a preoperative model (using variables available before anesthesia induction) and a perioperative model (using variables available up to the end of surgery).

Table 1 summarizes model performance using only structured variables. Across both prediction settings, discrimination was consistent across model families. Gradient boosting models generally achieved slightly higher receiver operating characteristic-area under the curve and precision-recall-area under the curve than logistic regression, while logistic regression showed lower Brier scores, indicating better calibrated probability estimates. The receiver operating characteristic curves for the preoperative and perioperative prediction models are shown in Figure 5, demonstrating similar discrimination performance across the two settings, with only a marginal improvement when perioperative variables were included.

The transition from preoperative to perioperative prediction yielded only marginal improvements in discrimination metrics. Notably, changes in threshold-dependent metrics such as F₁-score were inconsistent across models, reflecting differences in operating characteristics rather than overall predictive capability. This suggests that a substantial proportion of predictive information is already available prior to anesthesia induction, with intraoperative variables providing limited incremental value in overall model discrimination.

The relatively small performance differences between model types further indicate that, under the current feature set, model selection plays a secondary role compared to the availability and timing of clinical information. The Apfel score, a rule-based clinical model based on four binary predictors, showed lower discrimination and F₁-scores than ML models.

Calibration curves for both preoperative and perioperative models are presented in Figure 6. Overall, predicted probabilities showed reasonable agreement with observed event rates, although some deviation from perfect calibration was observed at higher predicted risk levels. This is consistent with the Brier score results reported in Table 1, which show that logistic regression demonstrated slightly better calibration performance. Clinical performance metrics at the Youden-optimal threshold are summarized in Table 2. The between-model differences in calibration carry practical implications for the intended clinical use. Because the framework is intended to support selective antiemetic prophylaxis—where action depends on whether a patient’s estimated risk exceeds an institution-specific threshold—well-calibrated probabilities are as important as discrimination. Logistic regression produced the lowest Brier scores and the closest agreement between predicted and observed risk, whereas the gradient boosting models, despite marginally higher discrimination, tended to overestimate risk at the highest predicted-probability levels. This suggests that, when absolute risk estimates rather than rank ordering alone are used to guide prophylaxis, either the better-calibrated logistic regression model or a post hoc recalibration of the gradient boosting models (eg, Platt scaling or isotonic regression) would be preferable.

We further evaluated the contribution of unstructured clinical text by incorporating deterministic text-derived scores as structured model inputs. Table 3 presents model performance after including text-derived features. Across models, the inclusion of text-derived features resulted in modest changes in discrimination metrics, with slight improvements observed in several model and task settings. The magnitude of improvement was limited, indicating that structured perioperative variables account for the majority of the predictive signal.

Text-derived features appear to provide complementary information, enriching the representation of clinical context without substantially altering overall model performance.

Decision curve analysis was performed to evaluate the predictive models’ clinical utility across a range of threshold probabilities (Figure 7). For both preoperative and perioperative prediction tasks, the machine learning models demonstrated positive net benefit across clinically relevant threshold probabilities, supporting the potential value of model-guided selective prophylaxis strategies compared with treat-all or treat-none approaches.

To enhance transparency and support clinical interpretation, we adopted a multi-level interpretability approach integrating model-based feature attribution, text-derived feature representation, and LLM-assisted explanation. SHAP was used to quantify feature contributions at the model level, the deterministic keyword-based scoring system provided traceability for unstructured clinical text, and QAChain generated context-aware, human-readable explanations of model predictions. Together, these approaches offer complementary insights into model behavior and support clinically meaningful interpretation.

To interpret the model’s internal decision-making mechanism, SHAP analysis was performed on the XGBoost model, which achieved the best overall predictive performance. Figure 8 presents the combined SHAP summary and dependence plots for the preoperative model. The summary plot ranks features according to their global importance based on mean absolute SHAP values, while the dependence plots further illustrate the direction and magnitude of each feature’s effect on the predicted PONV risk.

The most influential features include female sex, total intravenous anesthesia (TIVA), red blood cells, general anesthesia, age, ASA status, and selected surgical specialties. Among these, several variables exhibit clear, clinically meaningful directional patterns. Female sex shows a strong positive association with predicted PONV risk, with consistently higher SHAP values observed in female patients. In contrast, TIVA exhibits a substantial protective effect, with markedly negative SHAP values indicating reduced predicted risk. Age demonstrates an inverse relationship, where younger patients are associated with higher predicted risk, as reflected by decreasing SHAP values with increasing age.

Continuous variables, such as red blood cells, show a positive trend, indicating that higher values are associated with a higher predicted risk. Similarly, general anesthesia is associated with higher prediction outputs compared to nongeneral anesthesia approaches. Other variables, including ASA status and surgical specialties, exhibit more complex or nonlinear patterns, suggesting potential interactions or confounding effects within the model. These features contribute to prediction but require cautious interpretation.

Overall, the SHAP analysis indicates that demographic characteristics, anesthetic strategies, and perioperative physiological variables jointly drive model predictions, with several features demonstrating consistent and interpretable directional effects aligned with clinical knowledge. Similar patterns were observed in the perioperative model (Figure 9), with additional contributions from intraoperative variables such as anesthesia duration and opioid use.

A structured summary of the top features, including their direction and clinical interpretation, is provided in Table 4.

To connect the preprocessing pipeline with model interpretability, we examined how text-derived features constructed from LLM-assisted normalization and concept-based mapping contribute to model predictions.

The LLM was applied to selected unstructured clinical text fields, including medical history descriptions, surgical notes, and diagnostic narratives. These fields contain heterogeneous linguistic expressions, such as synonyms, abbreviations, and mixed-language representations, which cannot be reliably standardized using simple rule-based methods. Using a deterministic configuration (temperature=0), the LLM was employed to normalize semantically equivalent expressions into consistent representations aligned with predefined clinical concepts (eg, smoking status, prior PONV, and laparoscopic procedures). Importantly, the LLM was not used to generate new features or discover additional concepts, and no outcome-related information was used in this process. Only text fields available prior to the prediction time point were included, ensuring temporal consistency and preventing information leakage.

Clinically relevant concepts were predefined based on established literature and domain knowledge. Following normalization, clinical narratives were mapped to structured features using deterministic and rule-based procedures. Three types of text-derived features were constructed: (1) concept-based binary indicators (cx_text_*), representing the presence or absence of clinically meaningful conditions; (2) aggregated scoring features (eg, history_score, surgery_score, diagnosis_score, and total_text_score), derived from predefined weighting schemes; and (3) count-based features based on keyword frequency. While all final features are constructed deterministically, the LLM enhances the consistency and robustness of concept extraction from unstructured text.

Within the XGBoost model, SHAP analysis (Figure 9) shows that text-derived contributions are concentrated in a small number of concept-based features. In particular, cx_text_smoker and cx_text_ponv_history show substantially higher contributions than most other text-derived features, indicating that clinically meaningful concept indicators account for the majority of the text-derived signal in the model. In comparison, composite scoring variables (eg, history_score, surgery_score, diagnosis_score) show relatively limited contribution, while total_text_score provides a moderate but secondary contribution.

In addition to concept-based features, certain procedure-specific text-derived features (eg, port-a-catheter implantation) also appear among the top contributors, reflecting localized clinical patterns captured from surgical narratives that are not fully represented by predefined concept categories.

To further examine how the model uses different representations of clinical text, we grouped text-derived features into concept-based, aggregated-score, and count-based categories and visualized their SHAP contributions (Figure 9). This analysis shows that the model primarily relies on concept-based indicators, with aggregated- and count-based representations playing a secondary role.

Overall, these findings indicate that LLM-assisted semantic normalization enables consistent extraction of clinically meaningful concepts, which the model subsequently leverages as structured features. This design supports integrating unstructured clinical text into predictive modeling while preserving interpretability and reproducibility. Additional supporting analyses of text-derived features, including concept hit rates, postoperative vomiting rate comparisons, signal strength, the coverage funnel, and concept-level statistics, are provided in (Multimedia Appendix 5).

To evaluate the interpretability and reliability of the proposed framework, we conducted a comprehensive analysis of QAChain, an LLM-based post hoc explanation module that generates context-aware, human-readable interpretations of model predictions.

At the case level, QAChain consistently produced clinically meaningful explanations aligned with patient-specific risk profiles. For high-risk cases, explanations frequently emphasized well-established risk factors such as opioid exposure (eg, intraoperative fentanyl), prolonged anesthesia duration, and general anesthesia. For instance, explanations often highlighted that high-dose opioid administration and extended anesthesia duration jointly contribute to increased PONV risk. In contrast, for low-risk cases, QAChain tended to emphasize protective or mitigating factors, such as the use of TIVA and shorter procedural duration. For intermediate or borderline cases, QAChain generated balanced explanations that incorporated both risk-enhancing and risk-reducing factors, reflecting nuanced and context-sensitive reasoning.

To quantitatively assess whether QAChain explanations align with model behavior, we evaluated their alignment with SHAP-based feature attributions, which serve as a proxy for model-derived feature importance. Specifically, for each case, the top-k features ranked by SHAP values were compared with the features explicitly referenced in the corresponding QAChain explanation. Alignment was defined as the proportion of SHAP top-k features that were mentioned in the generated explanation.

As illustrated in Figure 10, QAChain explanations show high consistency with model-derived feature importance at the surface level. Most clinically relevant features—such as intraoperative fentanyl, TIVA, anesthesia duration, patient age, and female sex—are frequently reflected in QAChain explanations, achieving near-perfect alignment rates (≈99%‐100%). For example, high-frequency features such as female sex (n=2863) and anesthesia duration (n=2023) exhibit alignment rates above 99%, indicating that QAChain reliably captures dominant predictive signals even in large-scale clinical data.

However, this alignment metric captures lexical feature mention rather than direct causal faithfulness. Therefore, the results should be interpreted as surface-form concordance between QAChain explanations and SHAP-derived feature importance, rather than definitive evidence of full mechanistic fidelity. Features with lower alignment, such as oral fentanyl use, likely reflect semantic abstraction, where QAChain generalizes specific variables into broader clinical concepts (eg, opioid exposure) rather than consistently preserving fine-grained distinctions. Additionally, variables such as missing values (“nan”) and categorical surgical department labels are rarely mentioned, suggesting that QAChain selectively prioritizes clinically interpretable and semantically meaningful features in its natural-language explanations.

Overall, these findings indicate that QAChain produces stable, clinically coherent explanations that are meaningfully related to model attributions, while allowing semantic abstraction and human-readable reasoning.

To ensure reproducibility, QAChain was configured with deterministic settings (temperature=0), resulting in highly consistent outputs across repeated runs with identical inputs. Furthermore, QAChain explanations were generated from structured, engineered features derived from the modeling pipeline, without direct access to raw unstructured text or outcome variables. This design reduces the risk of information leakage and helps ensure that explanations remain aligned with the model’s input space.

Collectively, these findings indicate that QAChain produces stable explanations that are internally consistent with model-derived signals and are intended to enhance the interpretability of model predictions. Because no formal clinician-based evaluation was performed, this interpretability benefit cannot be objectively confirmed at this stage; the explanations should therefore be regarded as a useful interpretability reference to be validated in future clinician-centered studies.

This study presents a temporally structured framework for POV 24h that integrates structured clinical variables, text-derived features, and multilevel interpretability methods. By explicitly separating preoperative and perioperative prediction tasks, the framework reflects real-world clinical decision-making and avoids information leakage.

The results show that predictive performance is largely driven by structured clinical variables available before anesthesia induction, with intraoperative variables and text-derived features providing complementary but limited additional contributions. These findings highlight the importance of early-stage risk stratification and suggest that most predictive information is already available before surgery.

From an interpretability perspective, the proposed framework combines SHAP-based feature attribution with QAChain-generated explanations to provide both quantitative and human-readable insights into model behavior. Rather than serving as a standalone explanation method, QAChain functions as a translation layer that converts structured feature contributions into clinically interpretable narratives. The observed alignment between QAChain explanations and model-derived feature importance indicates meaningful consistency with dominant predictive signals, while allowing for semantic abstraction and contextual reasoning. This consistency supports the intended interpretability contribution of the explanations, although it does not by itself constitute an objective, clinician-based validation of interpretability or usability.

Overall, this work demonstrates a practical, transparent approach to integrating machine learning and large language models into clinical prediction tasks. By bridging structured modeling and natural-language explanation, the proposed framework provides multi-level explanations intended to enhance interpretability and support more transparent clinical decision-making. Because clinician-based evaluation was beyond the scope of this study, this interpretability benefit should be regarded as a promising reference rather than an objectively validated outcome, and warrants confirmation in future clinician-centered studies. Before broader clinical adoption can be considered, the models also require prospective, multi-center external validation; the present results should therefore be regarded as evidence of internal feasibility rather than established clinical applicability.