Initially, we enrolled 542 adult patients with ESRD undergoing hemodialysis at the hemodialysis unit of a medical center between October 1, 2018, and December 31, 2021. Patients who had received hemodialysis for less than 3 months or who were transferred to other clinics during the study period were excluded. After these exclusions, 412 adult patients with ESRD undergoing chronic hemodialysis without transfer remained eligible for analysis. Among them, 242 patients had no occurrence of major adverse cardiovascular events (MACEs), while 170 experienced at least one MACE. The primary objective of this study was to determine the incidence of MACEs in this population. A flowchart of the study participants is presented in Figure 1. The detailed baseline demographic and clinical characteristics of the cohort are summarized in Table 1.

This study specifically targeted the unique clinical needs and complexities of patients with ESRD by collecting 84 variables. These variables were meticulously selected based on their significant impact on clinical outcomes in patients with ESRD. Demographic data, such as age and gender, were included, and dialysis vintage reflected the duration and history of each patient’s dialysis treatment. Additionally, we detailed the anatomical and functional characteristics of the aortic and mitral valves, which are crucial for understanding cardiovascular complications in patients with ESRD. Specifically, we included parameters such as types of arteriovenous access (AVA), mitral valve calcification (MV calc), aortic regurgitation (AR), aortic stenosis (AS), mitral regurgitation (MR), and mitral stenosis (MS).

In terms of cardiovascular health, this study placed particular emphasis on the grading and types of left ventricular hypertrophy (LVH) [17] and the ejection fraction (EF) of the heart, which are critical indicators of cardiovascular health in patients with ESRD. Given the high prevalence and impact of comorbidities such as diabetes mellitus, hypertension, dyslipidemia, coronary artery disease, heart failure, chronic obstructive pulmonary disease, liver cirrhosis, malignancy, arrhythmia, and a history of amputation among patients with ESRD, we included these comorbidities in our analysis.

To better meet the clinical needs of patients with ESRD, we expanded the range of biochemical laboratory data. This included comprehensive assessments of total protein, albumin, liver enzymes (aspartate aminotransferase and alanine aminotransferase), alkaline phosphatase [18], total bilirubin, lipid profiles, glucose levels, complete blood count, iron studies, aluminum levels, postdialysis weight, uric acid, and key electrolytes. To address the specific needs of patients with ESRD, we further measured calcium and phosphate metabolism indicators, such as calcium and phosphate levels, urea kinetics (Kt/V), parathyroid hormone levels, and the calcium-phosphate product, to more effectively manage mineral and bone disorders in these patients.

The medication history was also thoroughly documented, particularly focusing on drugs commonly used in the management of ESRD, such as phosphate binders, calcitriol, and other treatments relevant to the patients’ condition.

This study included 412 patients. The mean age was 69.19 years, with older patients more likely to experience MACE (70.94 years vs 67.96 years, P=.01). Females comprised 46.6% of the cohort, with a lower proportion in the MACE group, although this difference was not statistically significant (P=.08). Aortic valve calcification was more prevalent in the MACE group (72.6% vs 56.1%, P=.005), and AS as well as certain types of LVH were also associated with the occurrence of MACE. Diabetes mellitus (DM) and peripheral arterial occlusive disease (PAOD) were more common among patients who experienced MACE (DM: 58.2% vs 40.9%; P=.001; PAOD: 34.1% vs 21.9%; P=.008). In terms of medication use, a higher proportion of patients in the MACE group were on insulin (P<.001) and antiplatelet drugs (P<.001). These results provide an overview of the baseline characteristics of the patients and offer important references for improving the accuracy of predictive models.

The data preprocessing methodology in this study is organized into 2 primary components: variable encoding and data balancing. The overall analytical workflow and preprocessing framework are illustrated in Figure 2. This integrated approach is designed to enhance the quality and representativeness of clinical datasets, thereby improving the robustness and generalizability of subsequent predictive models.

Missing data are a common challenge in clinical datasets and can significantly compromise the validity of statistical inferences if not appropriately addressed. In this study, we used a nonparametric multiple imputation strategy using the missForest package in R (R Core Team) [19]. This method uses random forest models to iteratively predict missing values based on observed data, effectively capturing complex nonlinear associations and interactions between variables. Unlike simpler imputation techniques such as mean substitution or k-nearest neighbors, missForest has been shown to yield more accurate and less biased estimates in both continuous and categorical variables, particularly in mixed type medical data [15,20]. This approach offers a robust and flexible foundation for downstream ML analysis while preserving the integrity of the original dataset.

Compared to SMOTE, which generates new samples through linear interpolation between neighboring minority class points, ROSE applies a smoothed bootstrap technique. It estimates a kernel density around each minority instance and samples new points from this local distribution. This approach allows ROSE to preserve the original variance and nonlinear structure of the data, making it particularly effective for clinical datasets where preserving subtle distributional patterns is important.

Imputation models were fit on the training fold only and then applied to the corresponding validation fold within each split. No statistics from the validation fold were used to fit the imputation model. This fold-wise procedure was repeated across all folds to prevent information leakage.

In clinical datasets, categorical variables are abundant and require conversion into numerical formats to be effectively used in statistical and ML models. Three encoding methods were implemented to address this challenge, each offering a unique balance between preserving information and managing computational complexity. Although all 3 approaches aim to transform qualitative data into quantitative representations, they differ in their operational mechanisms and associated trade-offs.

One-hot encoding converts each categorical variable into a series of binary indicators, where each category is represented by an individual binary feature. This method maintains the inherent nonordinal nature of the original variable but can lead to a substantial increase in dimensionality, particularly when the variable in question has many unique categories. In contrast, target encoding substitutes each category with a statistical summary such as the mean, weighted mean, or smoothed mean of the target variable computed from the training dataset. This not only reduces dimensionality but also encapsulates the predictive relationship between the categorical feature and the outcome variable, although it necessitates careful handling to avoid target leakage [14]. A third approach, frequency encoding, assigns to each category a numerical value based on its relative frequency within the dataset. This method is highly efficient in reducing computational burden and memory usage, as it compresses categorical information into a single continuous variable without imposing any artificial order [21].

To combine the complementary strengths of these methods, a unified encoding strategy was adopted. Specifically, the one_hot function from the mltools package in R was applied to perform one-hot encoding, which expanded the original set of 83 variables to 113. This expansion increased the granularity of the dataset, facilitating a more detailed representation of the clinical phenomena under this research. The inclusion of target and frequency encoding provided an additional layer of comparison, enabling an evaluation of their relative performance under conditions of significant missingness and class imbalance. Prior work has demonstrated that, particularly in datasets with high proportions of missing data and imbalanced classes, preprocessing methods based on one-hot encoding can significantly enhance both accuracy and robustness in classification tasks [15].

All categorical encoders were fit on the training fold only and then applied to its validation fold. Target encoding used an out-of-fold smoothing scheme: for each fold, category means and smoothing weights were computed from the training fold and then mapped to the validation fold. No target information from the validation fold was used to compute encodings.

Clinical datasets often exhibit imbalanced class distributions, where the minority class, despite its clinical significance, is underrepresented. Such imbalance can lead to biased models that disproportionately favor the majority class. To counteract this, 2 complementary strategies, Random Over-Sampling (ROS) and the SMOTE, were incorporated, along with an evaluation of the ROSE method [22].

The ROSE method uses a bootstrap resampling framework augmented by kernel density estimation to generate synthetic samples for the minority class. This approach avoids the pitfalls associated with simply duplicating minority samples, offering a more nuanced correction of the class distribution. Menardi and Torelli [23] provide an extensive discussion of these resampling techniques, emphasizing their utility in balancing datasets for binary classification tasks. On the other hand, SMOTE, as introduced by Chawla et al [24], synthesizes new minority class instances by interpolating between existing samples. This technique enriches the minority class by generating additional, diverse examples, thereby improving the model’s ability to capture the characteristics of rare events. Empirical studies have shown that SMOTE can lead to marked improvements in performance metrics, such as the area under the ROC curve (AUC), though it may overgeneralize when faced with extreme imbalance.

Comparative evaluations in the literature further underscore the respective merits of these techniques. For instance, Kamalov et al [25] found that ROSE, despite its relative simplicity, remains a stable and computationally efficient solution, especially in multi-label contexts. Similarly, investigations by Gnip et al [26] and Nguyen et al [27] have confirmed that both ROSE and SMOTE are effective in mitigating the adverse effects of class imbalance, with the optimal choice being contingent upon the specific characteristics of the dataset and the available computational resources.

k-fold cross-validation is a commonly used resampling technique for evaluating the performance of ML models. Specifically, the dataset is divided into k subsets (folds), and in each iteration, one subset is used as the validation set while the remaining k-1 subsets are used as the training set. This process is repeated k times, with a different subset used for validation each time.

This method effectively prevents overfitting and provides a more robust evaluation of the model. It is particularly useful for estimating the generalization error in small datasets [28].

In this study, we used 5-fold cross-validation to obtain an accurate estimate of model performance through multiple splits and evaluations while reducing bias caused by data partitioning.

All preprocessing and modeling steps, including cleaning, standardization, encoding, resampling, model fitting, and hyperparameter tuning, were executed separately within each dataset. No features, encoders, parameters, or statistics were transferred across datasets. For each model, hyperparameters were tuned by grid search with 5-fold cross-validation on the training set. A single predefined hyperparameter search space and evaluation criterion, detailed in Table S1 in Multimedia Appendix 1, was used for both ESRD and BRFSS. Grid search and model selection were rerun independently in each dataset so that optimal hyperparameters were learned within ESRD and BRFSS separately and were never reused across datasets.

In Phase 1, we compared alternative encoding and class-imbalance handling strategies for predicting MACE. Traditional logistic regression and 6 ML models were evaluated, including decision trees [29], random forests [30,31], XGBoost [32], CatBoost [33], and LightGBM [34]. All models were trained and evaluated using 5-fold cross-validation [35] within each dataset, following the generic engineering framework described in the previous subsection, that is, grid search hyperparameter tuning with the shared search space in Table S1 in Multimedia Appendix 1 and dataset-specific model selection.

The workflow, including preprocessing, resampling, and hyperparameter tuning, was then rerun independently on the BRFSS 2015 dataset to assess cross-dataset portability of the pipeline rather than to externally validate a single ESRD-trained model. All training was executed on a workstation equipped with an Intel Core i9 10th-generation CPU (3.3 GHz) and 64 GB RAM, using the CPU only. Average runtimes for each pipeline are reported in Tables 2 and 3 to help readers gauge computational cost. In this phase, multiple performance metrics were used, including accuracy, sensitivity, specificity, precision, F₁-score, and AUC. The mean and SD of these metrics were calculated across folds to assess predictive accuracy, robustness, and consistency. The overall workflow is summarized in algorithm S1 in Multimedia Appendix 1.

Within each dataset, model performance was summarized from 5-fold cross-validation. For each metric (AUC, accuracy, and F₁-score), we calculated 95% CIs from the 5-fold–specific estimates using a 2-sided Student t interval. This procedure corresponds to applying the t.test function in R to the vector of fold-level values. Because these metrics are bounded between 0 and 1, upper limits were truncated at 1.0 when necessary, and models with zero variance across folds yield a point interval at the mean.

For pairwise AUC comparisons within the same dataset, we applied the nonparametric DeLong test to out-of-fold predictions pooled across the 5 validation folds, using the roc.test function from the pROC package. We report 2-sided P values; all tests are conducted within the dataset, and we do not compute pooled AUCs or conduct between-dataset hypothesis tests.

In the second phase, the AUC [36] was used to evaluate the models from the first phase. For the optimal data processing method and the corresponding best-performing ML model, a detailed scoring and ranking of variables was conducted based on their importance and contribution to the model’s predictive performance.

The variables were converted into percentile rankings, with the most important variable assigned a score of 100, while the least important variable was assigned a score of 0. Using the 5-fold cross-validation approach, the scores from 5 iterations were summed and ranked, with the highest possible score being 500. Based on this scoring system, the top 15 highest-ranked variables were selected for remodeling, allowing for a more focused and in-depth analysis of their impact on the outcomes.

This study was reviewed and approved by the Institutional Review Board of Shin-Kong Wu Ho-Su Memorial Hospital (IRB No. 20231101R; approval date: December 14, 2023). The requirement for informed consent was waived by the ethics committee due to the retrospective nature of the study and the use of deidentified data. All procedures were conducted in accordance with the ethical standards of the responsible institutional and national committees on human experimentation and with the principles of the Declaration of Helsinki. The privacy and confidentiality of all participants were strictly protected throughout the study, and no personally identifiable information was disclosed. No compensation was provided to participants.

In this phase, we systematically evaluated the predictive performance of multiple ML models, including logistic regression, decision trees, random forests, XGBoost, CatBoost, and LightGBM, using 5-fold cross-validation. The primary objective was to determine the most effective combination of encoding methods (one-hot Encoding, frequency encoding, and target encoding) and data imbalance handling techniques (ROSE and SMOTE) for predicting MACE.

Models were evaluated by accuracy, F₁-score, and AUC with 5-fold cross-validation, and Table 2 reports the fold mean, SD, and t-distribution–based 95% CIs for each combination. Among all pipelines, the OneHotE_ROSE–LightGBM model achieved the best overall performance, with mean accuracy of 0.932 (SD 0.112; 95% CI 0.759‐1.000), F₁-score of 0.918 (SD 0.137; 95% CI 0.754‐1.000), and AUC of 0.940 (SD 0.116; 95% CI 0.794‐1.000). Frequency and target encoding under ROSE also performed strongly (AUC 0.913 and 0.900, respectively), but at a slightly lower level than one-hot encoding in the same setting.

To formally assess differences between preprocessing strategies, we performed pairwise DeLong tests on out-of-fold AUCs, with results summarized in Table S2 in Multimedia Appendix 1. Most ROSE pipelines showed significantly higher AUC than their SMOTE counterparts (all P≤.002), and OneHotE_ROSE in particular was markedly superior to OneHotE_SMOTE. Within the ROSE group, OneHotE_ROSE, FreqE_ROSE, and TargetE_ROSE did not differ significantly from each other (P≥.67), indicating a top-performing cluster. Together with the averaged ROC curves in Figures 3-5, these results support selecting one-hot encoding with ROSE as the primary preprocessing strategy for Phase 2.

A direct comparison of encoding methods in Figure 4 further confirmed that one-hot encoding was the most effective approach, achieving the highest average AUC (0.78), outperforming both frequency encoding and target encoding (both at 0.67). Meanwhile, Figure 5 illustrates that ROSE significantly outperformed SMOTE, achieving an average AUC of 0.83, compared to 0.59 for SMOTE. This result reinforces the conclusion that ROSE generates more representative synthetic samples, preserves the original data distribution more effectively, and enhances model generalization.

Our findings strongly suggest that one-hot encoding combined with ROSE is the most effective preprocessing strategy for this predictive task. This combination preserves categorical feature integrity while mitigating class imbalance, leading to the most robust and accurate predictive models. In contrast, SMOTE not only struggled to enhance model performance but, in some cases, even contributed to its degradation. These insights guided the selection of the optimal model for further feature importance analysis in Phase 2.

Following the identification of the optimal preprocessing strategy in Phase 1, we conducted a detailed analysis of feature importance and remodeled the dataset using the most influential variables. Table 4 presents the top 15 most important variables ranked based on their cumulative scores from 5-fold cross-validation, with antiplatelet, chest X-ray for aortic arch calcification (CXR.AoAC).0, and insulin emerging as the most significant predictors. These variables were selected to refine the models, ensuring a more efficient and interpretable predictive framework.

The high-performing ML models from Phase 2 identified a set of correlated features predictive of MACE in patients with ESRD, which can be grouped into 3 main clusters, including structural predictors of MACE, systemic burdens with traditional risk factors, and other indicators of disease severity. The corresponding feature importance profile for the Phase 2 LightGBM model is visualized in Figure S1 in Multimedia Appendix 1. The first group includes aortic arch calcification statuses (CXR.AoAC.0 and CXR.AoAC.2), and concentric LVH (LVH.type.1). These features reflect the model’s ability to capture structural markers representative of the broader spectrum of chronic kidney disease–mineral and bone disorder pathology. Moderate AoAC and AV calcification signal cumulative metabolic and inflammatory insults from longstanding phosphate imbalance, high uremic toxin burden, and chronic inflammation [37-39], while the absence of calcification indicates a protective cardiovascular profile to MACE risk. Additionally, concentric LVH has been long recognized as an independent predictor of MACE and all-cause mortality in patients with ESRD [17,40], results from chronic pressure overload, often due to arterial stiffness secondary to vascular calcification. Collectively, these structural changes contribute to hemodynamic consequences such as heart failure and fatal arrhythmias, thereby resulting in higher MACE risk [41].

The second group of predictors comprises DM, dyslipidemia, and PAOD. DM and dyslipidemia are well-established contributors to atherosclerosis and are exacerbated in ESRD [42,43], forming a vicious cycle with both ESRD-specific and traditional risk factors. For example, hyperglycemia promotes the accumulation of advanced glycation end-products due to impaired clearance, leading to endothelial dysfunction, oxidative stress, and inflammation, thereby fueling the atherosclerosis process [44]. PAOD, a direct manifestation of exacerbated systemic atherosclerosis, also shares similar underlying pathophysiology. Despite not being an isolated predictor, PAOD’s strong association to higher MACE risk [45] makes it a valuable proxy for assessing a patient’s total atherosclerotic burden.

The third group includes antiplatelet, intradialytic hypotension (IDH), and valvular disease status. With a score of 496, antiplatelet use serves as a surrogate marker for clinician-identified ischemic risk and thus MACE vulnerability [46]. Furthermore, the number of IDH episodes ranked fourth, underscoring the hemodynamic stress of dialysis, with cumulative episodes contributing to myocardial stunning and ischemic damage, thus in turn linked to MACE [47]. Tracking IDH frequency could offer a more comprehensive view of cardiovascular stress rather than focusing on isolated episodes as well as the episode severity. In addition, the absence of valvular diseases such as AS or aortic regurgitation reflects a mitigated risk of LVH and atherosclerosis progression [48]. The Phase 2 model effectively learned patterns of clinical practice—such as medication use and hemodynamic instability—offering real-world applicability and insights that extend beyond static risk markers to isolated hypotensive events.

The models were then retrained with one-hot encoding combined with either ROSE or SMOTE, and their performance was evaluated by accuracy, F₁-score, and AUC with 5-fold cross-validation (Table 3). Among the ROSE-based models, LightGBM and XGBoost achieved the highest discrimination, with LightGBM reaching accuracy 0.906 (SD 0.115; 95% CI 0.763‐1.000), F₁-score 0.892 (SD 0.119; 95% CI 0.745‐1.000), and AUC 0.925 (SD 0.112; 95% CI 0.786‐1.000), and XGBoost reaching accuracy 0.913 (SD 0.122; 95% CI 0.762‐1.000), F₁-score 0.899 (SD 0.138; 95% CI 0.728‐1.000), and AUC 0.906 (SD 0.137; 95% CI 0.736‐1.000). Random forest under ROSE also performed well (AUC 0.915; 95% CI 0.750‐1.000), confirming that ensemble methods benefited most from the refined feature set.

To place these gains in context, we also examined the traditional logistic regression (LGR) baseline in the same table. LGR trained in only 0.08 s for both ROSE and SMOTE variants, confirming its computational efficiency. However, its discrimination lagged behind the ML models, with AUC values of 0.755 (ROSE) and 0.728 (SMOTE). This contrast highlights a clear trade-off: while LGR offers near-instantaneous runtime, the proposed ML pipeline delivers markedly stronger predictive accuracy, justifying its added complexity for clinical applications.

To validate the statistical significance of these differences, we conducted DeLong tests for all model pairs, with detailed P values reported in Table S3 in Multimedia Appendix 1. ROSE-based ensembles (LightGBM, XGBoost, and random forest) achieved significantly higher AUCs than all SMOTE-based models (all P≤.022), while differences among the ROSE ensembles themselves were small and often not statistically significant. In contrast, the logistic regression baseline under ROSE achieved AUC 0.755 (SD 0.050; 95% CI 0.694‐0.817), clearly lagging behind the gradient boosting models despite its very short training time. These findings confirm that combining ROSE with gradient-boosted ensembles yields the most stable and discriminative performance in the Phase 2 setting.

These findings highlight the robustness of ROSE in preserving critical data characteristics, while the selection of the top 15 features enhanced model efficiency without compromising predictive accuracy. The combination of ROSE with XGBoost and LightGBM yielded the most reliable performance, reinforcing the effectiveness of targeted feature selection and imbalance handling in MACE prediction.

To assess cross-dataset portability and reproducibility, we applied the integrated preprocessing and modeling framework to 2 datasets with distinct data-generating mechanisms: an in-house ESRD registry (n=412) and the population-based BRFSS 2015 survey. Cross-dataset comparisons of model performance, encoding strategies, and imbalance-handling methods are summarized in Figures 6-8. For each dataset, the full workflow was executed independently, including missForest imputation, one-hot, frequency, target encoding, ROSE or SMOTE resampling, model tuning, and evaluation, with no cross-dataset feature matching and no pooled AUC computations. The framework yielded improved discrimination and clinically interpretable feature-importance rankings in both settings.

Our study has several constraints that warrant consideration. The ESRD cohort includes 412 patients, which limits the capture of rare patterns; multicenter expansion is ongoing to increase diversity. Despite 5-fold cross-validation, nested grid search, and an independent BRFSS run to assess cross-dataset portability, overfitting cannot be fully excluded, so temporal and multi-site validations are planned. Although the workflow performed well for cardiovascular and metabolic endpoints, its applicability to other disease domains remains uncertain and will be explored. By design, the BRFSS analysis did not involve feature harmonization, pooled AUCs, or between-dataset hypothesis tests. Finally, while ROSE outperformed SMOTE in our experiments, optimal resampling and encoding choices are likely dataset dependent, and hybrid strategies and embedding-based representations merit further investigation.

In contrast to previous studies that tend to focus on isolated aspects of data preprocessing, our research offers a comprehensive, end-to-end framework that addresses the multifaceted challenges inherent in clinical data, namely high missingness, heterogeneity, and severe class imbalance. While earlier works typically investigated single techniques in a limited scope, our study integrates multiple advanced methods and validates the entire pipeline using cross-dataset comparisons. This holistic approach provides a stronger foundation for both improving predictive performance and facilitating clinical translation, marking a clear advancement over traditional, piecemeal methodologies.

This study introduces a robust, generalizable, and interpretable data preprocessing framework for predicting MACEs among patients with ESRD. Although the improvements in performance metrics are incremental, they directly address key challenges inherent in clinical data heterogeneity and quality. The proposed pipeline enhances both predictive reliability and model transparency, offering practical value for clinical decision support and operational planning in nephrology care.

While system-level integration was not the primary focus of this work, the pipeline was deliberately designed for seamless adoption in real-world settings. By using routinely available clinical variables and lightweight computational procedures, it can be incorporated into existing analytics environments with minimal architectural adjustment. This design supports efficient deployment across diverse institutions and datasets, as evidenced by the consistent performance observed in both the ESRD registry and the BRFSS dataset. Such scalability reinforces its translational potential for broader clinical and public health applications.

Future work should prioritize cross-context application across more heterogeneous populations and evaluate the integration of deep learning architectures to further enhance predictive accuracy and extend applicability to other high-risk patient groups and chronic disease domains.