We conducted a retrospective review of medical records for 2637 patients who underwent nephrectomy between July 2012 and June 2019. Ultimately, the study included 1559 eligible patients. The patient data were sourced from a prominent tertiary hospital known for its comprehensive services and ranked among the top 60 nationwide in terms of overall performance. Patients were followed up for a duration ranging from 3.0 to 62.8 months until December 2019, with the primary focus being on the development of CKD as a long-term outcome. The patient data were randomly stratified into training, validation, and test sets, using Python’s stratified random sampling method, maintaining a ratio of 76.5:8.5:15. Internal validation was performed through 10-fold cross-validation, involving the partitioning of the training and validation sets into 10 subsets. A majority of 9 of these subsets were used for model training, and the remaining 1 was dedicated to model evaluation. The exclusion criteria for this study encompassed the following characteristics: (1) patients younger than 18 years of age or with hospitalization duration <24 hours, (2) patients with inadequate serum creatinine (Scr) monitoring interval, (3) patients with anatomical kidney malformations, (4) patients undergoing renal cyst unroofing or donor nephrectomies, (5) patients with pre-existing CKD or undergoing dialysis prior to nephrectomy, and (6) patients lacking essential features such as Scr.

This study received approval from the Ethics Committee of the Affiliated Hospital of Qingdao University (approval QDFY WZ 2018-9-13). Informed consent was waived due to the retrospective nature of the data and the large number of patients involved, making it impractical to seek consent from each patient. All data were deidentified. No compensation was provided to the participants as the study did not involve direct participant interaction.

Clinical and demographic data were extracted through the application of natural language processing and parsing methods on structured information within the electronic health record. Preoperative complete blood counts, coagulation markers, blood chemistry analyses, urine tests, and echocardiography were performed within 3 days of admission. Comorbidities were defined based on the International Statistical Classification of Diseases, Tenth Revision. Comprehensive data on concomitant medications were meticulously collected, with particular attention to instances where these medications were administered prior to the occurrence of kidney injury. The surgical details encompass the surgical approach (laparotomy, laparoscopy, and da Vinci surgery), procedure type (RN and PN), duration of the surgery, pathological findings, maximum excision diameter, and blood loss.

The primary outcome of our study was postoperative AKI. The secondary outcomes were AKD and CKD. AKI was defined as an increase in Scr to ≥0.3 mg/dL within 48 hours or ≥1.5 times the baseline value within 7 days, following the 2012 Kidney Disease Improving Global Outcomes guideline [18]. According to the 2017 Acute Disease Quality Initiative, AKD was defined as persistent renal damage or renal dysfunction for a duration of 7 to 90 days after exposure to an AKI initiating event [8]. CKD was defined as abnormalities of kidney structure or function for at least 3 months [19]. Based on the diagnostic criteria for AKI and AKD, patients exhibited three distinct trajectories of renal function following kidney injury: (1) AKI recover, if Scr returned to baseline value within 7 days (AKI without AKD); (2) AKD with AKI, if stage 1 or greater AKI persisted for ≥7 days after an AKI initiating event (continuous AKI progressing to AKD); and (3) subacute AKD, if Scr levels increased slowly but lasted more than 7 days (AKD without AKI). The final classification consisted of four categories: (1) no kidney disease (NKD), (2) AKI recover, (3) AKD with AKI, and (4) subacute AKD.

Baseline Scr was defined as the most recent Scr level measured before nephrectomy. The diagnosis time of AKI, AKD, and CKD was established when patients first met the respective diagnostic criteria. All patients underwent at least 3 Scr tests, including 2 during hospitalization and 1 at the first follow-up. If elevated Scr levels did not return to baseline, additional tests were conducted once a week during hospitalization or at the next follow-up. The estimated glomerular filtration rate (eGFR) was calculated by using the Chronic Kidney Disease Epidemiology Collaboration creatinine formula [20].

The Light Gradient-Boosting Machine (LightGBM) algorithm was used to construct predictive models. LightGBM, a tree-based gradient-boosting framework, adeptly manages high-dimensional and extensive datasets [21]. By integrating gradient-based 1-side sampling and exclusive feature bundling, LightGBM effectively mitigates overfitting and notably outperforms the computational speed and memory use of Extreme Gradient-Boosting and stochastic gradient-boosting techniques [22]. In our comparative analysis, we trained various ML models, including LightGBM, Gradient-Boosting Machine, k-nearest neighbors, multilayer perceptron, logistic regression (LR), naive Bayes, random forest (RF), and support vector machine, using the same dataset and applying consistent imputation and scaling techniques. We initially used the default hyperparameters of each ML algorithm to establish our models. Subsequently, we conducted manual parameter tuning by grid search to optimize the performance. The process of parameter optimization was facilitated through 10-fold cross-validation, aiding in the identification of the most suitable hyperparameter configurations [23].

For discerning significant features that influenced the algorithm and ensuring the appropriateness of the optimal model, we used SHAP and LIME to interpret the model from both global and instance-based perspectives. SHAP values, rooted in the Shapley value from coalitional game theory, quantify the influence of each feature variable on the target outcome, elucidating the derivation of a sample’s predicted result [24]. LIME uses local surrogate models for explaining individual predictions. Its core method perturbs an input instance to generate interpretable samples, upon which a linear model approximates the complex model’s decision-making process near the instance [25]. The SHAP summary plots exhibit the relative significance of individual features in predictions, along with their corresponding positive or negative impact directions. The SHAP interaction plots reveal the interactions among multiple features and illustrate how their combined influence impacts model predictions. We separately used the top 10 features from the AKD and CKD models and created SHAP dependence plots through pairwise combinations to elucidate the influence of individual features on the model’s predictions and the correlations among them. Additionally, we highlighted features with significant correlations in Figure S5 in Multimedia Appendix 1. The SHAP force plots and LIME bar plots were used to clarify individualized forecasts, demonstrating each feature’s contribution to the prediction of individual samples. Finally, we used AKD and CKD as outcomes and applied the PC algorithm to construct DAGs, facilitating the exploration of potential associations and causal pathways among the top 20 features [26,27].

A web-based calculator for predicting AKD and CKD among those patients was developed using the “Streamlit” application in terms of the optimal model. Streamlit, an open-source Python framework, aids developers in swiftly constructing web-based and responsive applications [28]. To improve the user-friendliness of the web calculator, this study implemented 2 panels: one for inputting model parameters and acquiring AKD or CKD probabilities and another for providing a model introduction.

Features with a missing proportion exceeding 15% (n=234) are removed, while those with missing proportions less than 15% (n=234) are imputed using an RF model. Using LR to calculate the required sample size with CKD as the outcome, we determined that a minimum of 1171 patients is necessary to attain a statistical power of 90% for detecting an effect size of 0.10 at a 2-side α=.05. Categorical features were presented using frequencies and percentages, while continuous features were presented as mean (SD) or median (IQR). Comparative analyses were performed to assess patient characteristics between individuals with and without CKD as well as among various trajectories of renal function postkidney injury. Quantile-quantile plots were generated to visually inspect the distribution patterns of continuous features. The independent 2-tailed t test was used for normally distributed continuous features, the Mann-Whitney U test for nonnormally distributed continuous features, and the Pearson chi-square test for categorical features. We used a weight rebalancing technique to adjust the weights of both the majority and minority classes in the training dataset [29]. The validation dataset underwent balancing, whereas the test datasets remained unaltered to assess model performance with representative data. The scikit-learn Python library (Python Software Foundation) includes a built-in parameter called “class weight” or “weights” for LR, RF, LightGBM, support vector machine, and k-nearest neighbors. For AKD, the class weight was set to 3.3; and for non-AKD cases, it was set to 0.6. Similarly, the class weight for CKD was set to 10.0; and for non-CKD cases, it was set to 0.5. In the case of the naive Bayes classifier, we established a prior probability of .50 for each class to achieve group balance, and we adjusted class weights in the multilayer perceptron classifier by modifying the loss function’s weights. The area under the receiver operating characteristic curve (AUROC) was used for optimal model selection. The model underwent evaluation through graphical techniques, encompassing the receiver operating characteristic curve and decision curve analysis, in addition to quantitative metrics such as AUROC, average precision, precision, recall, accuracy, F₁-score, Brier score loss, and Matthew correlation coefficient. A P value of less than .05 was considered as significant (2-tailed). Python programming language (version 3.9.13 and integrated development environment Visual Studio Code 1.81.1) was applied to our analysis.

The entire study process is illustrated in Figure 1. Among the participants, 1131 (72.6%) underwent RN, and 1152 (73.9%) underwent laparotomy. The incidence rates of AKI, AKD, and CKD were 21.7% (330/1559), 15.3% (238/1559), and 10.6% (165/1559), respectively. In total, there were 451 (28.9%) patients who developed acute or subacute kidney dysfunction (AKI or AKD criterion), with 117 (7.5%) meeting both AKI and AKD criteria, 121 (7.8%) developed subacute AKD, and 213 (13.7%) experienced recovery from AKI. The quantile-quantile plots show that features including blood loss, Scr, and operative duration exhibit skewed distributions, potentially due to the distinct condition of nephrectomy patients (Figure S1 in Multimedia Appendix 1). Increased CKD rates were observed in older patients (mean age 69, SD 9.6 vs mean age 58, SD 12.3 years), male patients (n=118, 12.9% vs n=47, 7.3% in female patients), those who underwent RN (n=143, 12.6% vs n=22, 5.1% in PN), AKD with AKI (n=42, 35.9% vs n=32, 26.4% in subacute AKD, n=24, 11.3% in AKI recovery, and n=67, 6% in NKD), and individuals with 1 or more chronic complications such as hypertension, diabetes mellitus, and coronary heart disease. The demographic and clinical characteristics of the patient cohort, both within different groups and as a whole, are detailed in Table 1 and Table S1 in Multimedia Appendix 2.

A comprehensive set of over 90 features was served as features for both AKD and CKD and were integrated into the ML models. Among the assessed ML models, the LightGBM model demonstrated superior performance (Figure S2 in Multimedia Appendix 1 and Tables S2 and S3 in Multimedia Appendix 2). In the test set, LightGBM achieved the highest AUROC of 0.97 for AKD and 0.96 for CKD prediction. The F₁-scores, 0.75 for AKD and 0.70 for CKD, indicate a balanced trade-off between precision and recall. Additionally, Brier score loss was maintained at 0.05 for both AKD and CKD predictions, demonstrating the model’s impressive calibration. To create a user-friendly web-based calculator, we simplified the model by reducing the number of input features. The inclusion of the top 10 and top 5 features for the AKD and CKD models, respectively, negligibly affected the LightGBM model’s AUROC (achieving 0.94 vs 0.97 for AKD prediction and 0.94 vs 0.96 for CKD prediction, as detailed in Figure S2 in Multimedia Appendix 1 and Table 2). Notably, it outperformed all other ML algorithms in terms of AUROC (Tables S4 and S5 in Multimedia Appendix 2), while maintaining an optimal balance between precision, recall, and error rates (both false positives and negatives). Subsequently, we used the LightGBM model for result interpretation and the development of a web-based calculator. Comprehensive insights into performance metrics and visualizations are provided in Figures S2 and S3 in Multimedia Appendix 1, Table 2, and Tables S2-S5 in Multimedia Appendix 2.

The SHAP summary plots of the LightGBM models are depicted in Figure 2. Operative duration, hemoglobin (Hb), blood loss, urine protein, and hematocrit were the top 5 features associated with predicted AKD. Baseline eGFR, pathology, trajectories of renal function, age, and total bilirubin were the top 5 features associated with predicted CKD. The SHAP interaction plots visually elucidate the interplays among the top 10 features in both the AKD and CKD models (Figure S4 in Multimedia Appendix 1). The SHAP dependence plots offer detailed insights into the correlations among the top 10 features, as depicted in Figures S6 and S7 in Multimedia Appendix 1, with representative examples showcased in Figure S5 in Multimedia Appendix 1. For instance, the influence of AKI grade on the probability of AKD varies across Hb levels. Among patients with lower Hb levels, higher AKI grades are associated with a significant increase in the risk of AKD. Conversely, this correlation is less pronounced in patients with higher Hb levels. For patients presenting with a baseline eGFR below 80, postoperative complications, such as AKD with AKI, subacute AKD, or AKI recover, markedly elevate the risk of developing CKD. This observation underscores the significance of encompassing factors like trajectories of renal function within a comprehensive clinical framework, particularly one that integrates a patient’s eGFR.

Sample individualized predictions with their explanations are shown in Figure 3. The AKD and CKD models, respectively, present the top 10 and top 5 features. We selected 4 random samples from the test set and analyzed them using both the SHAP and LIME algorithms. For example, Figure 3D presents an individualized explanation for a case where the actual and predicted outcomes are both CKD. The notably high predicted probability for CKD (P=.97) primarily stemmed from several incremental factors, including a low baseline eGFR level (39.36 mL/min/1.73 m²), postoperative complications of AKD with AKI, clear cell pathology, and a history of antibiotic use, despite a normal white blood cell level (3.94×10⁹/L). The SHAP force plot revealed minor deviations in the top 5 features for predicting this patient, highlighting the greater significance of γ-glutamyl transferase over albumin-globulin (AG) ratio.

DAG is a type of causal diagram comprising nodes representing features and arrows representing causal relationships between the features. Since the importance of features does not necessarily reflect causality, we designated only AKD and CKD as end points (nodes with only inward-pointing arrows) in the DAGs, without designating source nodes (nodes with only outward-pointing arrows). Given that analyzing all features (over 90) would lead to an excessively complex causal structure, we limited the analysis to the top 20 features for AKD and CKD. During the investigation of AKD as the outcome, we observed direct links from features such as AKI grade, operative duration, systolic blood pressure, Hb, antibiotic, baseline eGFR, urine protein, and hematocrit to AKD, indicating potential direct causality (Figure S8 in Multimedia Appendix 1). All these features, except for antibiotics, are among the top 10 features for AKD prediction. We discovered that the trajectories of renal function, pathology, and baseline eGFR exhibit potential direct causal relationships with CKD, and they also rank among the top 5 features in the CKD model (Figure S9 in Multimedia Appendix 1). Age did not exert a direct effect on CKD but influenced it indirectly through its impact on pathology and baseline eGFR. Despite AG and procedure being within the top 10 features for CKD, our analysis did not reveal a causal link to CKD, suggesting that while there was a correlation between AG or procedure and CKD, they were causally independent.

Since the LightGBM model proved to be the most effective in our study, we developed a web-based calculator using the “Streamlit” application to predict both AKD and CKD with this model. Restricting the LightGBM model to only the top 10 and top 5 features did not diminish predictive performance for the AKD and CKD models (AUROC: 0.94 vs 0.97 in AKD prediction and 0.94 vs 0.96 in CKD prediction). For ease of use, we constructed a web-based calculator using the top 10 and top 5 features to predict AKD and CKD, respectively. You can access this calculator at Streamlit [30].

Our exploration into the use of ML techniques to predict and elucidate outcomes in patients undergoing nephrectomy was instigated by an amplified emphasis on the long-term renal functional prognosis, the accessibility of intricate data within the electronic health record system, and the maturation of interpretable predictive models. Among patients who underwent nephrectomy, 28.9% (n=451) developed AKI or AKD. Specifically, 7.5% (n=117) of patients developed AKD in conjunction with AKI, 13.7% (n=213) experienced recovery from AKI, and 7.8% (n=121) developed subacute AKD. The incidence rate of CKD was 10.6% (n=165). We formulated a diverse array of ML models with a focus on AKD and CKD prognosis. Among these models, LightGBM exhibited the most robust predictive prowess, achieving an AUROC of 0.97 for AKD prediction and 0.96 for CKD prediction. Our research used SHAP values and the LIME algorithm to interpret the decision-making process from both global and instance-based perspectives. Additionally, we used DAG to further visualize the potential causal relationships between features and outcomes. In consideration of clinical applicability, we further developed a web application that uses the final prediction model to estimate AKD and CKD risks.

Assessment of renal injury risk following nephrectomy has predominantly concentrated on AKI and CKD, with limited attention to the recovery of renal function within 7-90 days post-AKI and its enduring consequences [5,31-33]. Our prior research has unveiled discernible distinctions in the predicted risk factors between AKI and AKD. Specifically, AKD is associated with a notably elevated risk of de novo CKD development when contrasted with AKI [34]. This study encompassed all patients who were hospitalized, with no specific subgroup analysis conducted for those undergoing nephrectomy. Hu et al [35] initially developed a predictive model for postnephrectomy AKD, using an LR model to assess predicted risk factors associated with renal injury within 3 months following nephrectomy. Nevertheless, this study did not differentiate between AKI recover, subacute AKD, and AKD with AKI. Furthermore, it did not investigate the long-term outcomes for patients experiencing these distinct renal function trajectories. In our study, we found a significant association between trajectories of renal function and the onset and progression of CKD. Specifically, the coexistence of AKD with AKI led to a CKD incidence rate of 35.9% (n=42), nearly 1.5 times higher than that observed in patients with subacute AKD (AKD without AKI). Meanwhile, the CKD incidence rate was 11.3% (n=24) for individuals who had recovered from AKI and 6% (n=67) for those with normal kidney function. Among patients with kidney injury, nearly one-third experienced subacute AKD, which did not meet the criteria for either AKI or CKD diagnosis. These individuals are frequently overlooked in the early stages due to the modest changes in renal function they exhibit; however, their risk of developing CKD is significantly elevated when compared to both patients with NKD and those who have recovered from AKI. As such, the presence of AKD serves as a critical link between AKI and CKD, aiding in the assessment of declining renal function and prognosis.

Currently, LR is the most widely used model for predicting kidney injury risk in patients undergoing nephrectomy, with limited application of ML algorithms [36,37]. Lee et al [38] used various ML algorithms to formulate a risk prediction model for AKI after nephrectomy, identifying that the LightGBM model outperforms others in terms of predictive accuracy. Compared to LR, LightGBM demonstrates enhanced speed, more efficient memory use, and superior parallel processing capabilities, which allow it to more effectively manage nonlinear relationships, large datasets, and high-dimensional data [39]. Our study has undertaken a thorough evaluation of the predictive abilities of several ML models, with LightGBM emerging as the most effective in forecasting high-risk AKD and CKD cases, alongside precisely pinpointing individual predicted risk factors. Early alerts assist in promptly notifying clinicians to undertake vigilant monitoring of patients at high risk. Addressing manageable predicted risk factors early, such as Hb, systolic blood pressure, and total bilirubin, presents a considerable opportunity to lower the occurrence of AKD and CKD, thereby enhancing patient outcomes.

Given our emphasis on interpretability, our methodology entails a thorough interpretation of the entire predictive algorithm, exploring the potential causal relationships between major features. First, we generate global-level diagrams that elucidate the contributions of each feature to the model’s output along with interactions among key features. Features denoting acute injury, such as surgical factors and AKI grade, exert a significant influence on AKD. Baseline eGFR and trajectories of renal function constitute pivotal features affecting CKD. Features such as advanced age or clear cell carcinoma may be associated with an elevated CKD risk. While these attributes are generally nonmodifiable, augmenting the frequency of follow-up visits for individuals with these characteristics can effectively facilitate the early detection of renal function deterioration. Second, this study delineates the decision-making process for each patient. The examples depicted in Figure 3 elucidate the predominant feature compositions among patients exhibiting diverse predicted probabilities of AKD or CKD. Using SHAP force plots and LIME plots amplifies the individualization and transparency of the decision-making process, thereby alleviating the black-box issue inherent in the model’s prediction process. Finally, DAGs were used to delve deeper into the potential causal relationships between features and outcomes. It was found that most of the top 10 features identified by SHAP values have the potential to directly or indirectly influence the occurrence of AKD or CKD.

For the sake of enhancing user convenience, we have developed web-based prediction tools for both AKD and CKD. Users can effortlessly input the values of their chosen features to calculate the probabilities of AKD and CKD following nephrectomy. Our research marks a pioneering effort in constructing web-based prediction tools for postnephrectomy AKD and CKD, which can assist clinicians in identifying high-risk individuals and risk factors. Given the clinical feasibility and straightforward accessibility of features derived from routine medical records, our models are eminently suitable for seamless integration into daily clinical practice.

The study exhibits several limitations. First, the web-based prediction tool is crafted to assist clinicians in discerning individuals with elevated risk of AKD and CKD rather than serving as a replacement for clinical diagnosis. Due to the retrospective nature of data collection, it is crucial to undertake additional validation using an independent population to ensure robust predictive validity across diverse usage scenarios. Second, the collection of urine output data is subjective, and a significant number of values are missing. Consequently, this study refrained from using urine output as a diagnostic criterion for AKI. Third, our study lacks time-variant monitored values among its features. Moving forward, we intend to collect longitudinally monitored data from patients undergoing nephrectomy to enable dynamic prediction of AKD and CKD before their clinical identification. Finally, DAGs visually represent the potential causal relationships between features and outcomes. This underscores the need to further explore and quantify the causal mechanisms in future work.

This study has developed prediction models that accurately estimate the risk of AKD and CKD following nephrectomy. These models provide interpretability from both global and instance-based perspectives. We recommend the use of the AKD criterion in clinical practice due to its superior accuracy in predicting prognosis, particularly the development of CKD.