This study encompassed clinical data from 2454 patients with RC in the SEER database, wherein 1954 cases were randomly selected to form the training cohort, and the remaining 500 formed the internal validation cohort. Additionally, it included data from 500 patients with RC treated at the author’s hospital between January 1, 2017, and December 31, 2023, which served as the external validation cohort. The inclusion criteria were (1) patients with RC staged as I-IV according to the American Joint Committee on Cancer (AJCC) staging system; (2) those who underwent curative surgery; and (3) those with complete clinical and pathological information; and (4) radical surgery in stage IV patients. In this study, some patients with stage IV RC with oligometastasis (limited resectable metastases) or locally advanced tumors with resectable metastatic lesions underwent radical surgery after evaluation by a multidisciplinary team. The surgical decision was based on the multidisciplinary team’s comprehensive assessment of tumor burden, physical condition, expected survival, and surgical risks. The goal was to achieve R0 resection and combine it with postoperative adjuvant therapy to reduce recurrence risk. Data for these patients were complete and underwent strict screening to ensure they met the research criteria. The exclusion criteria included (1) a history of other malignant tumors, (2) inability to assess lymph node status, (3) incomplete clinical information, and (4) administration of neoadjuvant therapy. The LNM status for 2454 patients with RC in the SEER database was ascertained from pathological assessments of surgical specimens. For the 500 cases at the author’s hospital, LNM status, clinical T and N staging, and tumor length were determined by precisely annotating and measuring contrast-enhanced CT images with the radiological software ITK-SNAP software (version: 4.0.1; Professor Paul Yushkevich's team, University of Pennsylvania, USA), corroborated by pathological assessments from surgical specimens. For example, the definition of tumor length in the SEER database (training/internal validation cohorts) was measured from pathological specimens. In the external validation cohort, tumor length was annotated and measured on contrast-enhanced CT scans using ITK-SNAP software by radiologists, with cross-validation against postoperative pathology (mean error <0.3 cm).

The study collected the following clinical and pathological characteristics of the patients: sex, age, perineural invasion, carcinoembryonic antigen (CEA), tumor length, clinical T stage, N stage, tumor differentiation, chemotherapy and radiotherapy administration, liver metastasis status, and tumor histology. For the training and internal validation cohorts, data extraction was conducted using the SEER*STAT software, version 8.4.3 (Surveillance, Epidemiology, and End Results Program, National Cancer Institute, USA). In the external validation cohort, clinical data were independently collected and processed by 2 researchers, while CT images were annotated and measured with precision by 50 physicians, evenly distributed into 25 groups, each consisting of 2 physicians for cross-validation purposes. The inclusion criteria for metastatic lymph nodes were fulfilled if any of the following five conditions was met: (1) the short axis/long axis ratio was ≥0.8; (2) the short axis was ≥5 mm; (3) there was aggregation of three or more lymph nodes; (4) the lymph node had an irregular shape with a rough margin; (5) the lymph node signal was inhomogeneous, with high signal areas in the CT imaging for lymph nodes (including mesenteric and presacral lymph nodes). Figure 1 illustrates the acquisition process.

All collected data will be subjected to rigorous preprocessing in Python, encompassing data cleaning, outlier detection, and missing value imputation. Subsequent statistical analysis will utilize suitable methods including χ² tests, ANOVA, and both univariate and multivariate regression analyses to evaluate the correlations between clinical pathological features and LNM.

In this study, we standardized numerical features, including tumor length, to mitigate the effects of varying scales and enhance their compatibility with the input requirements of ML models. For categorical variables like clinical T staging, we applied label encoding to transform them into numerical data. All features were first normalized to eliminate scale effects (Multimedia Appendix 1). We then performed univariate feature screening via independent t tests or Mann-Whitney U tests (P≤.05), retaining features. Next, Pearson correlation analysis reduced redundancy by removing features with mean absolute correlation >0.9, yielding decorrelated features. Finally, features with nonzero coefficients were selected through LASSO regression (10-fold cross-validated) with regularization parameter α. Notably, this LASSO-based selection served as initial screening; tree-based models (Random Forest [RF]/XGBoost [XGB]/Light Gradient Boosting Machine [LGBM]) subsequently performed endogenous feature reweighting to capture nonlinear interactions during model training. This method incorporates an L1 regularization term that aids in pinpointing the most influential features while promoting model sparsity. Furthermore, we performed a correlation analysis among the features to verify the independence of the selected variables. Utilizing the principle of Permutation Importance, we evaluated the importance of each feature to ascertain its contribution to the model’s predictive performance.

This study developed 11 distinct ML models: Logistic Regression (LR), Naive Bayes (NB), Support Vector Machine, K-Nearest Neighbors (KNN), RF, Extremely Randomized Trees (ET), XGB, LGBM, Gradient Boosting (GB), Ada-Boost (AB), and Multilayer Perceptron (MLP), for predicting the risk of LNM in patients with RC (Multimedia Appendix 2). To counteract the imbalance in data distribution, we used random oversampling techniques and applied a penalty parameter λ to drive the coefficient estimates of nonsignificant features toward zero. The models’ performance was evaluated through 10-fold cross-validation. The performance of each ML classifier was assessed using the receiver operating characteristic curve, where a higher area under the receiver operating characteristic curve (AUC) indicates greater predictive accuracy. We evaluated the variable weights and permutation importance and used heatmaps to visualize the significance and correlations among the variables. To select the optimal model, we evaluated each model’s performance across multiple metrics: AUC (with 95% CI), accuracy, sensitivity, specificity, and F₁ score. Calibration curves and decision curve analysis (DCA) were further employed to assess calibration accuracy and clinical utility, particularly focusing on net benefits at critical thresholds.

This study was approved by the Ethics Committee of the First Affiliated Hospital of Anhui Medical University (Approval No.: Quick-PJ 2023-13-34). As a retrospective analysis using anonymized data, the requirement for informed consent was waived. Primary data collection in the SEER database obtained patient informed consent, and its Institutional Review Board explicitly authorized secondary analysis without additional approval. All direct identifiers (including names, ID numbers, medical record numbers) were removed from our hospital’s dataset, with only aggregated data utilized in the analyses to ensure privacy protection. No participant compensation was involved, and no identifiable images appear in the manuscript or supplementary materials.

In our study, the training cohort included 1954 patients with RC, the internal validation cohort included 500 patients with RC, and the external validation cohort included 500 patients with RC. Across the training cohort, internal validation cohort, and external validation cohort, stratification by LNM status (positive vs negative) revealed statistically significant differences in age, gender, total lymph node count, tumor length, neural invasion, clinical T stage, N stage, liver metastasis status, tumor histology, and tumor differentiation (all P<.001; see Table 1). For CEA, the P value was .001, indicating a statistically significant difference (Table 1). We used the LASSO regression method to identify a significant set of risk factors for predicting the risk of LNM in patients with RC.

Using the nonzero coefficients from the LASSO logistic regression model as a guide, we applied both the LASSO method and the multivariate logistic regression model to identify the risk factors associated with LNM in patients with RC, as depicted in Figure 2. The penalty parameters λ for the internal and external validation sets were found to be 0.0047 and 0.0036, respectively. Univariate and multivariate analyses yielded the forest plots for both the internal and external validation sets, as shown in Figure 3. Key predictors from univariate and multivariate analyses are detailed in Table 2. This analysis assists in pinpointing factors that potentially contribute to LNM, marking a pivotal step in comprehending disease progression and in formulating evidence-based treatment strategies. The multivariate logistic regression analysis of the external validation set identified number of peritumoral and total lymph nodes examined (P=.022), tumor length (P<.001), neural invasion (P<.001), clinical T stage (P<.001), N stage (P<.01), and tumor differentiation grade (P<.01) as independent risk factors for LNM.

Calibration curves (Figure 7) demonstrated the agreement between predicted probabilities and observed event frequencies. In the internal cohort (Figure 7A), the XGB curve adhered closest to the ideal diagonal (45° reference line) throughout, with minimal deviation from actual frequencies in the 0.3‐0.7 probability range; ET exhibited consistent positioning above the diagonal at >0.8 probabilities (systematic overestimation), while RF deviated slightly below the diagonal between 0.6 and 0.8 (mild underestimation). In the external cohort (Figure 7B), XGB maintained the smallest overall deviation (closest to the diagonal); GB showed systematic distribution below the diagonal at 0.4‐0.6 probabilities (underestimation), and LGBM positioned above the diagonal at <0.3 thresholds (overestimation).

We evaluated the net benefits of the models in both validation sets using DCA (Figure 8). In the internal validation set, DCA indicates that the XGB model has a net benefit similar to that of the XGB and LGBM models at lower threshold probabilities (approximately 0‐0.3). Furthermore, the XGB model maintains net benefit across a broader range of threshold probabilities (approximately 0‐0.9). Between 0.7 and 0.9, where the net benefits of other models decrease, the net benefit of the XGB model still sustains. The RF model shows net benefit within the range of 0‐0.8. The ET model has a net benefit from 0.1 to 0.75. The LGBM model demonstrates net benefit in the range of 0.1‐0.7. In the external validation set, DCA reveals that the XGB model’s net benefit is higher than that of the GB and LGBM models when the threshold probability is between 0.3 and 0.4. Furthermore, the XGB model maintains net benefit across a broader range of threshold probabilities (approximately 0‐0.4). The GB model shows net benefit within the range of 0.1‐0.3, while the LGBM model has net benefit from 0.1 to 0.35.

The main findings demonstrate that the XGB model achieved optimal performance in predicting RC LNM, with tumor differentiation grade, clinical T stage, tumor length, neural invasion, N stage, and total lymph node count identified as independent predictors. RC LNM significantly impacts patient prognosis, necessitating accurate prediction of LNM for the development of effective treatment plans. While traditional diagnostic methods offer some insights, their accuracy is constrained. This study employed ML algorithms to develop 11 prediction models for RC LNM. We extracted clinical data from the SEER database for the training and internal validation sets. Additionally, we acquired relevant clinical data from CT scans at the author’s hospital, utilizing precise pixel-level annotation and measurement for the external validation set. We filtered clinical data using the LASSO, univariate, and multivariate logistic regression analyses to develop and validate a predictive model for LNM. The selection of the optimal model was determined through comparative analysis of internal and external validation sets, leveraging the AUC, calibration curves, and DCA for precise assessment.

In this study, we successfully developed and validated 11 ML models for predicting LNM in RC. This study revealed three key findings. First, the six independent predictors of LNM include tumor differentiation, clinical T stage, N stage, tumor length, total number of lymph nodes, and neural invasion. Second, among the 11 constructed ML models for predicting LNM, 10 models had an AUC greater than 0.9 in the internal validation set and 7 models had an AUC greater than 0.80 in the external validation set. Lastly, following a comparison of the models’ performance and their validation set discrepancies, the XGB model emerged as the most suitable among the 11 models.

Utilizing LASSO logistic regression and multivariate analysis, we identified tumor differentiation, clinical T stage, N stage, tumor length, neural invasion, and total number of lymph nodes as the key clinical predictive factors. Low tumor differentiation, advanced clinical T stage, advanced N stage, increased tumor length, and neural invasion are significantly associated with LNM. Identifying these factors is essential to comprehend the tumor’s biological behavior and invasive potential. Variations in feature importance across models (Figure 5) arise from the following factors: First, clinical data encompass both continuous (eg, tumor length) and categorical variables (eg, pathological grade), where XGB’s exact splitting favors continuous features while LGBM’s histogram-based algorithm optimizes high-dimensional sparse feature processing. Second, regularization differences lead to distinct weight allocation—XGB’s combined L1/L2 regularization strictly controls overfitting, whereas LGBM’s leaf-wise growth prioritizes locally significant features. Third, although LASSO-based feature selection reduced dimensionality, residual feature correlations are differentially processed. These inherent algorithmic variations confirm that no single model can fully capture feature relationships, necessitating multimodel comparison for complementary insights. Poorly differentiated tumor cells show greater invasiveness and a higher likelihood of LNM. A study by Saraste et al [24] involving 1664 patients with RC from the Swedish Rectal Cancer Registry between 2007 and 2010 concluded through multivariate analysis that poorly differentiated tumors are a significant risk factor for LNM, aligning with the findings of this study. Advanced clinical T stage is a direct indicator of deeper tumor invasion. A study by von den Grün et al [25], which involved a binary logistic regression multivariate analysis of 776 patients with RC, revealed that advanced T stage is a significant prognostic factor for LNM in RC. Previous studies [25-27] have established that clinical T stage and N stage significantly influence the prognosis and treatment of RC, aligning with the results of the univariate and multivariate analyses presented in Table 2. Increased tumor length may afford tumor cells increased opportunities to interact with adjacent lymph nodes. In a study [28] that employed 7 clinical parameters as independent prognostic factors to develop a nomogram prediction model, 6484 patients with RC from the SEER database were analyzed using Cox proportional hazards regression. This analysis identified independent prognostic factors such as T stage and tumor length. However, unlike this study, the aforementioned research did not include an external validation cohort. The presence of neural invasion indicates a high level of tumor invasiveness and the potential for dissemination along neural pathways. Neural invasion is a significant factor in both univariate and multivariate analyses, as well as in evaluating the importance of model features. Studies by Ueno et al [29] and Song et al [30] have shown that neural invasion is a significant factor in LNM in RC. The total number of perirectal examined lymph nodes, as a key factor, plays a crucial role in accurate staging and prognostic assessment of RC. Guan et al [31] found that a higher number of perirectal examined lymph nodes in RC is linked to more accurate lymph node staging and improved survival. Analyzing data from stage I to III resected RC in multi-institutional Chinese and US SEER databases revealed that an increasing number of perirectal examined lymph nodes significantly raises the proportion of cases shifting from lymph node-negative to lymph node-positive. Furthermore, after adjusting for confounding factors, overall survival consistently improves.

Utilizing the selected clinical characteristics, we developed a predictive model for LNM in RC. We extracted a cohort of 2454 patients with RC from the SEER database and randomly selected 500 cases for the internal validation cohort. The remaining 1954 cases served as the training cohort, while an additional 500 cases from the author’s hospital constituted the external validation cohort. In the internal validation set, the RF and ET models achieved an AUC of 0.964. The LGBM and XGB models also performed remarkably well, with AUC values of 0.943 and 0.942, respectively. Among all models, the NB model had the lowest AUC of 0.859, while the remaining 10 models all exceeded 0.9. In the external validation set, the GB model demonstrated the best performance with an AUC of 0.838, followed by the XGB and LGBM models, which reached AUC values of 0.832 and 0.831, respectively. In the study by Guan et al [18], 6578 patients with RC were enrolled across several institutions, including the Cancer Hospital of the Chinese Academy of Medical Sciences, Peking Union Medical College, Changhai Hospital of Naval Medical University, and the Second Affiliated Hospital of Harbin Medical University. The XGB model was identified as the optimal in their study, achieving AUCs of 0.78 and 0.71 across two validation cohorts. In contrast, our study’s XGB model showed superior performance, achieving an AUC of 0.942 in the internal validation cohort and 0.832 in the external validation cohort. In the external validation cohort, LNM status, clinical T and N stages, and tumor length were ascertained through the annotation of CT imaging combined with clinical data. However, across both internal and external validation cohorts, most models in this study demonstrated relatively good performance across various validation sets, with the AUCs of the optimal models all exceeding 0.80, indicating a certain level of accuracy and reliability. These results demonstrate that our models exhibit high stability and generalizability across diverse datasets. Regardless of whether the patients with RC are in the United States or China, our models show accurate predictive ability for forecasting LNM in RC. Second, the LNM status and associated details in the 500 case records from the hospital were meticulously annotated by 50 physicians. These annotations were made by integrating clinical and pathological data using imaging software on enhanced CT scans. Following the initial annotation, two radiologists, possessing 8 and 20 years of experience, respectively, conducted a review and confirmation of the annotations. The multi-physician annotation strategy, which incorporates clinical pathology data and imaging software, boosts the reliability of the data and, in turn, elevates the predictive accuracy of the model. A similar approach was taken in Liu et al’s study [32], wherein integrating clinical data with ML for magnetic resonance image analysis elevated the AUC value of single-region radiomics from 0.702 to 0.827. In a study by Wan et al [33], an automated segmentation method using deep learning demonstrated potential for predicting LNM in RC. However, our manual annotation method ensures precise localization of small lymph nodes (diameter < 5 mm) and ambiguous lesions, which are crucial for RC staging but challenging for automated tools. This strategy is particularly valuable in resource-limited environments lacking artificial intelligence infrastructure. Although magnetic resonance imaging is widely regarded as the gold standard for local T-staging and lymph node status assessment in RC, in this study, we chose to analyze CT images due to the diversity of data sources and the prevalence of CT images. CT scanning is less costly and quicker. Particularly, in resource-limited medical settings, CT remains a routine examination tool that can effectively provide important information on lymph node status and tumor staging. This highlights the benefits of integrating both clinical data and imaging information into ML models. While the annotation of CT images and their incorporation are standard practices in traditional radiomics and the development of deep learning models, they are less common in models derived solely from clinical data. This study shows that employing imaging software to annotate CT scans and extract data for clinical predictive models not only yields high-quality data for model development but also presents an innovative approach to data acquisition and processing for future studies.

A key innovation of this study is the comprehensive utilization of diverse advanced ML algorithms, coupled with the validation of the model’s performance across data from various sources. Our models have shown strong generalization capabilities, as evidenced by their performance in both internal and external validation cohorts. Additionally, using LASSO logistic regression for feature selection enabled us to pinpoint key clinical predictive factors. This approach not only bolstered the model’s predictive accuracy but also improved its interpretability. We presented the importance of features in a ranked order and investigated the correlations among variables with a correlation heatmap. Finally, the optimal model was ascertained through a comprehensive, evidence-based methodology integrating multimetric performance evaluation (AUC, 95% CI, accuracy, sensitivity, specificity, and F₁-score), augmented by high-resolution assessment of calibration curves and DCA.

Despite strong performance in both the internal and external validation sets, our model in this study has some limitations. First, the dataset is constrained by its population representativeness. The SEER database comprises exclusively American cases, and the dataset from the author’s hospital consists solely of Chinese cases. This could impact the model’s ability to generalize across diverse populations. In Shulman et al.’s study [34], which analyzed 34,500 patients with RC, it was found that patients of different races exhibited varying lymph node statuses. Second, the model did not include variables like BMI, alpha-fetoprotein, cancer antigen 125, carbohydrate antigen 19-9 [18], which were previously explored in clinical information. This omission could impact the model’s predictive accuracy and comprehensiveness. Future studies could further refine predictive accuracy by incorporating serum biomarkers (eg, BMI, carbohydrate antigen 19-9) and emerging indicators such as circulating tumor DNA, combined with radiomic features. Third, FN classifications warrant clinical attention. The FN rate was 20.7% (28/135) in the internal cohort and 29.2% (89/305) in the external cohort. Such misclassifications may lead to undertreatment (eg, omission of adjuvant chemotherapy or lymph node dissection), increasing risks of recurrence and metastasis. Crucially, while our model was developed to surpass conventional CT assessments, reducing FNs remains a priority for future optimization through biomarker integration or advanced imaging techniques. The study by Yu et al [35] demonstrated that incorporating multicenter data and multimodal features significantly reduced FN rates, providing a feasible direction for refining our model. In this study, we utilized 11 ML methods. Future research could further enhance the model’s predictive performance by integrating diverse ML algorithms and leveraging their unique strengths.

In conclusion, this study successfully developed a ML-based risk prediction model for LNM in RC, validating its performance using both an internal and an external validation set. Through the analysis of extensive clinicopathological data, we identified tumor differentiation, clinical T stage, N stage, tumor length, neural invasion, and total number of lymph nodes as independent predictive factors. Among the 11 models evaluated, the XGB model demonstrated the best predictive performance. These models are anticipated to aid in clinical decision-making, offering vital insights for treatment selection and prognostic assessment of patients with RC. They hold significant clinical utility and scientific importance. Concurrently, the external validation set demonstrates that clinical data derived from CT imaging annotation and measurement are comparable to traditionally obtained clinical data and can effectively serve as a source of clinical data for ML applications.