Survival After Radical Cystectomy for Bladder Cancer: Development of a Fair Machine Learning Model

Introduction

The development and use of prediction models is increasing in many domains, including health care. These models can be used to guide clinical decision-making as they can potentially provide more accurate predictions of outcomes by adjusting for multiple patient and clinical features, compared to clinician assessment alone [1,2]. Machine learning (ML) methods are increasingly being used to develop prediction models as their use often yields models with superior predictive performance compared to models developed using traditional statistical methods [3]. However, there have been growing concerns about the potential for bias in ML models, as studies have demonstrated discrepancies in model performance among different population subgroups, which can lead to disparities in health care [4-6]. This concern for model bias, or an unfair model, has brought attention to the concept of algorithm fairness, where a fair model is described as one that works well for all populations, regardless of personal characteristics that are considered protected or sensitive, such as sex, race, age, sexual orientation, among others [7,8].

An unfair model is of particular concern in bladder cancer, where disparities in the likelihood of receiving treatment and timing of treatment have been identified in sex and racial subgroups [9]. While several predictive models have been developed to predict survival in patients with bladder cancer [10-12], the performance of these models among different population subgroups remains understudied. Predicting survival is an important component in counseling patients with bladder cancer given the relatively high mortality rates in this disease [13,14]. The use of a potentially unfair model may perpetuate and even exacerbate existing disparities for vulnerable patient populations.

In this study, we used the National Cancer Database (NCDB) to develop a ML model that predicts survival after radical cystectomy, the current standard treatment for muscle-invasive bladder cancer [15], and evaluated for potential model unfairness in sex and racial subgroups. We then applied and compared model unfairness mitigation techniques to improve algorithm fairness [16] and deployed the first web-based fair ML model predicting survival after radical cystectomy.

Methods

Data Source

We used data from the NCDB, which captures approximately 70% of all new invasive cancer diagnoses in the United States each year. The NCDB is a joint project of the Commission on Cancer of the American College of Surgeons and the American Cancer Society. Reporting hospitals are restricted to those accredited by the American College of Surgeons Commission on Cancer. Data reporting to the NCDB is highly standardized, and all data submitted undergo a thorough review for verification of data integrity [17].

Study Population

We identified patients with a histologic diagnosis of invasive urothelial carcinoma of the bladder between 2004 and 2016 and selected those who underwent radical cystectomy for clinically localized, muscle-invasive (cT2-4N0XM0X) disease. We excluded patients with missing dates of cystectomy; patients who received radiation, hormonal, immunotherapy, or other therapies prior to radical cystectomy; as well as patients who were treated with palliative intent.

For individuals included in our sample, we extracted patient and disease features. The patient features of interest were age at diagnosis, sex, race, ethnicity, facility type, insurance status, median income quartile, and comorbidity score; the disease factors were the year of diagnosis, pathological tumor stage, pathological nodal stage, and receipt of neoadjuvant or adjuvant chemotherapy.

We used the race and ethnicity features in the NCDB to categorize patients into the following National Institutes of Health (NIH) racial groups [18]: White, Black, Hispanic, or Asian. We a priori excluded American Indian or Alaska Native and Native Hawaiian or other Pacific Islanders as separate groups given their anticipated limited sample size [19,20]. The outcome of interest was overall survival at 5 years.

Model Development and Evaluation

We used stratified splitting to separate the sample into training (13,184/16,481, 80%) and test (3297/16,481, 20%) sets, ensuring balance on sex, NIH racial group, and the outcome of interest.

We trained and compared various machine learning classification models, including random forest, decision tree, extreme gradient boosting (XGBoost), and logistic regression, to predict 5-year overall survival. We used 5-fold cross-validation to select the hyperparameters that optimized the F₁-score. The baseline models were trained on the full training set, with optimized hyperparameters. We used a classification threshold of 0.5 for our baseline models.

Our primary model performance metric was the F₁-score, which is the harmonic mean of the positive predictive value, also known as precision, and sensitivity, also known as recall; the formulas have been provided previously [21]. The F₁-score can range from 0 to 1, with a value of 1 indicating perfect precision and recall. Secondary model performance metrics included true positive rate (TPR), false positive rate (FPR), and accuracy.

We evaluated model performance in the full test set and subgroups based on sex and race. The “best” naïve model was selected based on the highest F₁-score in the full test set.

Evaluation of Fairness

Several group fairness metrics have been described and fall under 3 broad principles: independence, separation, and sufficiency [8]. Independence is closely associated with demographic parity, which requires equivalent positive prediction rates across sensitive groups. Demographic parity is important when there are known historical biases in the dataset or objectivity of the target variable [8]. Separation is related to equalized odds and their relaxed variations; separation is suitable when the target variable is an objective ground truth and equality of error rates across sensitive groups is a priority. Equalized odds compare the TPR and FPR between different groups and are useful when true and false positives are considered to be of similar importance. Relaxed versions such as equality of opportunity and predictive equality can be used if prioritizing equal TPRs or FPRs, respectively. Separation-based criteria should not be used when the target variable may be prone to bias [8]. Finally, sufficiency is satisfied when for any predicted score, the probability of belonging to the positive or negative class is the same across all sensitive groups [8,22]. Unlike separation approaches, which may prioritize equalizing error rates across groups potentially at the expense of a specific group’s precision, sufficiency emphasizes fairness by ensuring similar prediction calibration for all groups without directly penalizing the overall model’s performance on any 1 group [22,23].

As some fairness criteria are not mutually compatible [24-26], for our clinical context, we used the equalized odds ratio (eOR) to estimate model fairness since we prioritized a model that minimized known disparities in bladder cancer by satisfying equality of error rates across sex and racial subgroups [8,27], there was no major concern for the retrospective data to contain measurement bias or historical bias, and true and false positive predictions of 5-year overall survival were of equal importance. The eOR ranges from 0 to 1, where a value of 1 indicates equivalent true positive and false positive rates across sensitive groups.

Mitigating Unfairness

Techniques to mitigate algorithm unfairness can be applied in the 3 different phases of model development: preprocessing, in-processing, and postprocessing [8,28]. Techniques in the preprocessing category transform input data before they are passed to the training algorithm. For example, correlation remover is a preprocessing technique that applies a linear transformation to the nonsensitive features in order to remove any correlation with sensitive features. In practice, however, preprocessing approaches can still result in classifiers that result in substantial algorithm unfairness [29]. In-processing techniques incorporate fairness constraints within the model training process. These techniques aim to steer the model toward producing fair predictions. One such example is the exponentiated gradient method, which implements a reduction approach. In reduction approaches, the model is treated as a black box optimizer. The algorithm iteratively reweights the training data points based on the current model’s predictions and a chosen fairness metric. The model is then retrained with these reweighted points, aiming to reduce the unfairness observed in the previous iteration [29,30]. In postprocessing techniques, algorithms transform the output of a trained model. A threshold optimizer is an example of a postprocessing technique that takes as input an existing machine learning classifier and the sensitive feature, uses its predictions as a scoring function, and identifies a separate threshold for each group defined by a sensitive feature in order to optimize the specified fairness constraints [22]. Some limitations to note of the postprocessing approach are that they are not guaranteed to find the most accurate fair classifier and they require test time access to the protected attribute, which may not be available [29,31].

In this study, we compared 3 different techniques to mitigate unfairness in the different phases of model development: correlation remover (preprocessing), exponentiated gradient (in-processing), and threshold optimizer (postprocessing). The final “fair” model was chosen based on the mitigated model with the highest eOR in the full test set.

Software and Analyses

All analyses were completed using Python (Python Software Foundation), and packages from scikit-learn and Fairlearn [16]. We estimated 95% CIs using 1000 bootstrap samples.

Ethical Considerations

The study was exempted by our institutional review board committee since the NCDB does not contain any identifiable patient information.

Results

Population Characteristics

We identified 374,881 patients with a histologic diagnosis of invasive urothelial carcinoma of the bladder between 2004 and 2016 and selected those that underwent radical cystectomy for clinically localized, muscle-invasive (cT2-4N0XM0X) disease. After applying exclusion criteria, our overall final sample consisted of 16,481 patients, and their characteristics are shown in Table 1. The full sample was 23.1% (n=3800) female and 77% (n=12,681) male; with regard to the NIH racial groups, 91.5% (n=15,080), 5% (n=832), 2.3% (n=373), and 1.2% (n=196), were “White,” “Black,” “Hispanic,” and “Asian,” respectively (Multimedia Appendix 1). The median age at diagnosis in the overall sample was 71.0 (IQR 63.0-77.0) years, the majority of patients had pathological tumor stage 2 disease (n=7329, 44.5%; Table 1), and most patients were found to have negative pathological (pN0) lymph nodes (n=14,049, 85.2%).

Patient characteristics stratified by sex and race are shown in Table 2. Black male patients were the youngest (median age 66.5, IQR 58.0-74.0 y), on average, and had the highest proportion of pathological tumor stage 4 disease (116/512, 23%), while pathological lymph node involvement was highest in White male and Hispanic male patients (n=1599 and n=38 respectively, both 3.5%).

Of the 16,481 patients, there were 12,290 (75%) deaths within 5 years. Stratified by sex and race, death rates within 5 years were the highest in Black male patients (407/512, 80%) and the lowest in Asian male patients (101/148, 68.2%; Table 2).

Naïve Model Performance and Evaluation of Fairness

The performance of the naïve models is shown in Table 3. The “best” naïve model based on F₁-score was XGBoost (F₁-score 0.860; 95% CI 0.8490.869). When evaluated for fairness, the “best” naïve model had an eOR ratio of 0.619 (Table 4).

In sex and racial subgroups, the “best” naïve model achieved the highest F₁-score in Black male patients (F₁-score 0.907; 95% CI 0.859-0.947) and the lowest in Asian female patients (F₁-score 0.824; 95% CI 0.571-0.947; Figure 1).

‎

Mitigating Unfairness

We compared the preprocessing, in-processing, and postprocessing algorithm unfairness mitigation techniques of correlation remover, exponentiated gradient, and threshold optimizer, respectively. We found that all techniques improved the eOR, with the largest improvements observed after applying correlation remover and threshold optimizer; application of these techniques resulted in an equivalent eOR of 0.750 (Table 4). The mitigated model using correlation remover had a higher F₁-score and was selected as our final “fair” model (Table 4).

The comparison of the “best” naïve and “fair” final models in sex and racial subgroups is shown in Figure 1. We found that compared to the “best” naïve model, the F₁-score of the “fair” final model did not change significantly for any of the subgroup populations. Secondary performance metrics in subgroups are shown in Multimedia Appendix 2.

Discussion

In this study, we used the NCDB to develop a ML model predicting survival after radical cystectomy for bladder cancer and found that the naïve model would be considered unfair due to inferior performance in certain sex and racial subgroups. We then compared and used algorithm unfairness mitigation techniques in the 3 different phases of model development and found that all techniques improved model fairness, with negligible impact on overall classifier accuracy. Our study highlights the importance of evaluating prediction models in health care for potential unfairness and demonstrates different techniques that can be applied to mitigate model unfairness. We also deploy the first web-based “fair” model predicting survival in patients undergoing radical cystectomy for muscle-invasive bladder cancer.

The potential consequences of algorithm unfairness have been highlighted in previous studies. In a study that evaluated software used by courts in the United States to decide whether to release an offender or to keep them in prison, they found higher FPRs for Black offenders, compared to White offenders [6]. The bias in this model is partly due to the skewed training data in which the 2 populations had unequal base rates of being charged with new crimes [32]. These discrepant base rates may be partially attributable to heavier policing in predominantly Black neighborhoods or bias in the decision to make an arrest [33]. Another study evaluated a model used by health insurers to select patients for complex care management programs and found that for the same level of model-predicted risk, Black patients have significantly higher illness burdens, compared to White patients [5]. This bias was attributed to the algorithm’s objective of accurately predicting costs, which biases against Black patients who have been shown to generate lower medical expenses, conditional on health. A review of clinical ML algorithms found that the majority of models exhibited some form of racial bias [28]. The application of these unfair models could amplify and perpetuate disparities for vulnerable populations. Fairness methods can be used to mitigate known biases; in the context of the prison software, for example, historical biases in the data could be addressed by developing a model to optimize demographic parity [8]. In the context of the biased complex care algorithm, fairness would be improved by having the model’s objective changed to accurately predict active chronic conditions, rather than costs [5].

In bladder cancer, several prediction models have been developed to predict survival [10-12], although none have been evaluated for potential unfairness in sex and racial subgroups. In this study, we found that a model predicting survival after radical cystectomy demonstrated unfairness whereby specific subgroups, such as Black male patients, had superior model performance compared to others, such as Asian female patients. If such an algorithm was used to support decision-making, quality-of-service harm may result in subgroups with less accurate predictions [31].

Identifying the source of model unfairness can be challenging, and several potential causes have been described [7,8,28,31]. Sampling bias occurs when the model performs worse in minority populations due to insufficient sampling during training [7]. This source of bias is of potential concern when developing prediction models in diseases such as bladder cancer, where the epidemiology is significantly skewed by sex and race; worldwide, bladder cancer is approximately 4 times more common in male patients than female patients [34] and in the United States, bladder cancer is approximately 2 times more common in White populations compared to Black, Hispanic, and Asian populations [35]. Given this skewed demographic, which was reflected in our study, a naïve ML model may not perform equally well in all subgroups of patients. However, despite Black male patients representing only 3.1% of our sample, we found that the naïve model performed best in this subgroup. Notably, Black male patients had the highest mortality rate at 5 years, and this imbalanced data may partly explain the superior model performance in this subgroup. This aligns with prior research where a ML model predicting survival in patients with prostate cancer undergoing radical prostatectomy showed superior performance in Black patients, which was also the minority group with the highest mortality rate in that study [4].

While the potential for model bias has been described in previous studies, research on applying techniques to mitigate algorithmic unfairness remains limited [28]. However, a recent scoping review found that all studies implementing methods to address racial bias successfully improved fairness metrics [28]. This underscores the value of using such techniques before model deployment to optimize fairness. Beyond metrics, ensuring that a model is perceived as fair is particularly important in health care as it can enhance trust among both patients and clinicians, potentially leading to greater acceptance and integration of the model’s predictions into clinical decision-making. Our study also highlights the importance of evaluating and comparing multiple techniques to achieve optimal fairness, as there was variability in the degree of improvement observed between each approach.

Our study is not without limitations. Despite the use of model unfairness techniques which improved the eORs of the naïve model, the final “fair” model did not achieve an eOR of 1, indicating the presence of residual unfairness; however, some might argue that an eOR of 1 is theoretical and could not be achieved in practice. This concept was illustrated in a study that demonstrated the restrictive conditions required for a deterministic classifier to achieve perfect equalized odds [36]. Therefore, while approaches can be used to reduce algorithmic unfairness, clinicians and patients should still be aware of residual potential discrepancies in model performance. Another limitation of optimizing eOR is that it violates other definitions of fairness and may compromise performance in certain subgroups [26]. In our study, we found that while the F₁-score of the mitigated model improved or was unchanged in some subgroups, performance decreased in certain subgroups, highlighting the potential trade-off between fairness and performance. This raises an ethically complex question regarding what is considered “fair,” and the answer varies depending on what conceptual framework of fairness is used. We specifically focused on the pillar of inclusion which seeks to ensure that the benefits and harms from prediction models are distributed in an equitable manner [37]. The eOR is also challenged as a metric in the context of imbalanced data, as is common in health care, as baseline differences can contribute to disparities in performance. However, despite the expected differences across sex and racial subgroups in our study, we prioritized the eOR to develop a model that satisfied equality of error rates across subgroups [8,27] and where true and false positive predictions of 5-year overall survival were of equal importance. Finally, our study uses NIH racial groups, which are composed of heterogeneous subgroups; we did not evaluate model performance in these subgroups given their expected limited sample size. Additionally, we chose to focus on sex and race given the existing disparities in bladder cancer among these subgroups [9], but other sensitive subgroups, such as those based on socioeconomic status or rurality, could be studied for potential bias in model performance. Despite these limitations, our study demonstrates the importance of evaluating model unfairness and, to the best of our knowledge, is the first to apply unfairness mitigation techniques to reduce disparities from potential algorithmic unfairness in bladder cancer.

In conclusion, we developed a ML model to predict survival after radical cystectomy for bladder cancer and found that a naïve model exhibited bias, as certain subgroups of patients, based on sex and race, had inferior performance compared to others. Using algorithm unfairness mitigation techniques to minimize potential disparities arising from biased models, we improved model fairness and deployed the first web-based fair ML model for predicting survival after radical cystectomy.