The data used in this study were originally collected for a prior investigation on trends in NS organisms and antimicrobial sensitivity (Institutional Review Board of Jordan University [IRB] approval 10/2023/16360). For this secondary analysis, deidentified datasets were repurposed to predict NS using ML. All procedures adhered to international and national ethical standards, including the Declaration of Helsinki. Data were anonymized and accessible only to the research team. Because the study involved retrospective records without direct patient contact, a waiver of informed consent was approved by the Scientific Research Ethics Committee University of Jordan, School of Medicine (IRB 787/2023/76; renewal 4531/2025/67).

A retrospective cohort study was conducted at the 34-bed NICU of Jordan University Hospital, which admits approximately 1000 neonates annually. Between October 2018 and April 2024, 7005 neonates were admitted. Of these 7005 neonates, 3731 (53.3%) were excluded due to unavailability of blood culture data, leaving 3274 (46.7%) eligible for inclusion in the 3 predictive ML models for NS evaluation. Among the 3274 included cohort, 770 (23.5%) had culture-confirmed sepsis. Figure 1 shows the cohort flow diagram.

The eligibility criteria and definition of sepsis used to determine inclusion and classify the study population were as follows:

Data were extracted from the NICU EHR system and included the following variables:

A complete description of all predictor variables, including definitions, measurement units, timing, and transformations applied, is provided in Table S1 in Multimedia Appendix 1.

The dataset of 3274 neonates was randomly divided into training and testing sets using stratified sampling to preserve the original sepsis class distribution (23.5% sepsis in both subsets), with a training set comprising 2619 (80%) neonates and a testing set comprising 655 (20%) neonates.

XGBoost is an implementation of gradient-boosted decision trees that builds multiple trees sequentially, where each subsequent tree corrects the errors of the previous tree. The model incorporates L1 and L2 regularization to prevent overfitting. Refer to Multimedia Appendix 3 for the complete mathematical formulation, including the regularized objective function and regularization terms.

The neural network used in this study is an MLP classifier with an input layer, 2 hidden layers of 100 neurons each using ReLU activation, and an output layer for binary classification. The model was optimized using the Adam optimizer. Refer to Multimedia Appendix 3 for the mathematical formulation of neuron activation and loss functions. Figure 2 shows the neural network architecture.

The decision tree classifier splits data at each node based on criteria that maximize information gain or minimize Gini impurity. Tree depth was limited to 5, with minimum samples per split of 10 and minimum samples per leaf of 5 to prevent overfitting. Refer to Multimedia Appendix 3 for the mathematical formulations of Gini impurity and information gain.

For XGBoost, 95% CIs for accuracy, recall (sensitivity), and specificity were calculated using the Wilson score method, while area under the receiver operating characteristic curve (AUC) CIs were estimated using the DeLong method. For the neural network and decision tree models, all performance metric CIs (accuracy, sensitivity, specificity, precision, recall, F₁-score, and AUC) were estimated using bootstrap resampling with 1000 iterations.

The feature importance analysis was obtained from the XGBoost algorithm. Table 1 presents the contribution of each feature to predicting sepsis, ranked by their respective F scores. The F score indicates how many times a feature is used to perform a data split across all decision trees in the XGBoost model. Features with higher F scores are more frequently chosen for partitioning, exerting a greater impact on the model’s predictions, while features with lower F scores have less influence.

The most significant predictors of sepsis included CRP (F score=826.0), PLT (F score=641.0), hemoglobin (F score=561.0), and GA (F score=395.0). Moderately influential predictors were WBC count (F score=497.0) and neutrophil percentage (F score=416.0), with elevated neutrophils being strongly associated with bacterial invasion, consistent with the pathophysiology of sepsis. The APGAR score at 1 minute (F score=196.0) correlated with perinatal stress, such as hypoxia or trauma, highlighting birth complications as a potentially modifiable risk factor.

Exclusive breastfeeding at discharge (F score=78.0) appeared in the feature importance analysis derived from the trained XGBoost model; however, it is not temporally available at the time of prediction and therefore is not a clinically valid predictor for real-time sepsis detection.

In contrast, central line insertion and chest tube interventions had much lower F scores, suggesting a smaller role in model decisions. This analysis highlights which clinical parameters are most vital for predicting NS.

All 3 ML models were evaluated for their ability to predict NS. XGBoost showed the best accuracy and discriminatory power, followed by the decision tree and neural network models. Across models, the incorporation of class imbalance handling, feature selection, and overfitting mitigation contributed to improved sensitivity for minority sepsis cases while maintaining balanced performance for the majority class. The overall trends suggest that tree-based models, particularly gradient boosting, may be better suited for structured neonatal EHR data, while neural networks demonstrated moderate performance likely limited by dataset size and hyperparameter sensitivity.

The results are summarized in Table 2. Each model offered unique strengths and trade-offs regarding accuracy, recall, and precision, reflecting its suitability for this specific task.

XGBoost demonstrated superior performance among the 3 models. It achieved a mean cross-validation accuracy of 93.7% (SD 1.2%) and a test accuracy of 94% (95% CI 92%-96%; 616/655, 94% correct predictions), making it the most effective model for this task. For sepsis=0, precision was 0.98 (95% CI 0.96-0.99) and recall was 0.91 (95% CI 0.88-0.93), while for sepsis=1, precision was 0.91 (95% CI 0.88-0.93) and recall was 0.98 (95/97, 98% sepsis cases, 95% CI 0.96 to –0.99). The F₁-score for both classes was 0.94, reflecting a strong balance between precision and recall.

Figure 3 shows the receiver operating characteristic (ROC) curve for XGBoost, with an AUC of 0.98 (95% CI 0.97-0.99), indicating excellent discriminatory power. The curve rises steeply near the y-axis, demonstrating the model’s ability to correctly classify positive cases while maintaining a minimal false-positive rate. Its proximity to the upper-left corner emphasizes high sensitivity and specificity. These results underscore XGBoost’s robustness in handling structured and imbalanced datasets. Its incorporation of class weights and regularization enables a focus on minority classes without compromising overall accuracy, and minimal variation in cross-validation scores confirms stability and generalization capability.

The neural network model, despite its flexibility in modeling complex relationships, underperformed compared to XGBoost. It achieved a test accuracy of 75% (95% CI 71%-78%; 491/655, 75% correct predictions). For sepsis=0, precision was 0.76 (95% CI 0.73-0.79) and recall was 0.77 (95% CI 0.74-0.80). For sepsis=1, precision was 0.74 (95% CI 0.71-0.77) and recall was 0.73 (71/97, 73% sepsis cases, 95% CI 0.70 to –0.76). Figure 4 shows the ROC curve for the neural network, with an AUC of 0.81 (95% CI 0.78-0.84). The curve demonstrates a good, although not perfect, ability to discriminate between positive and negative cases. Its upward trajectory reflects the model’s capacity to achieve a high true-positive rate while maintaining a moderate false-positive rate, although deviations from the upper-left corner indicate some misclassification. The lower performance likely stems from the limited dataset size, the neural network’s sensitivity to hyperparameters, and the structured nature of the data, which tree-based models such as XGBoost handle more efficiently. Despite these limitations, the neural network demonstrated reasonable predictive capability and could potentially improve with hyperparameter tuning, feature scaling, or a larger dataset.

Decision trees are inherently interpretable, making them valuable in clinical applications where understanding model decisions is critical. Pruning techniques, including limiting tree depth and controlling the minimum number of samples per split and leaf, mitigated overfitting and ensured generalization.

The decision tree achieved a mean cross-validation accuracy of 79.1% and a test accuracy of 80% (95% CI 77%-83%; 524/655, 80% correct predictions). Precision and recall were relatively balanced across both classes. For sepsis=0, precision was 0.83 (95% CI 0.80-0.86) and recall was 0.77 (95% CI 0.74-0.80), while for sepsis=1, precision was 0.77 (95% CI 0.74-0.80) and recall was 0.84 (81/97, 84% sepsis cases, 95% CI 0.81 to –0.87). The F₁-score was 0.80 for both classes. Figure 5 shows the ROC curve, with an AUC of 0.86 (95% CI 0.83-0.89), reflecting good discriminative performance. The curve demonstrates the model’s ability to correctly identify positive cases, although a moderate false-positive rate is observed. Although its performance was lower than that of XGBoost, the decision tree provides an interpretable and clinically useful alternative for NS prediction.

This study highlights the transformative potential of ML for NS prediction, especially in resource-constrained settings. XGBoost emerged as the top-performing model, achieving consistent performance across all folds with a mean accuracy of 93.7% (SD 1.2%), a mean AUC of 0.98 (SD 0.01), and a mean sensitivity of 0.97 (SD 0.02). Minimal fold-wise variance confirmed strong generalizability, with better performance than neural networks (AUC 0.81, 95% CI 0.78-0.84) and decision trees (AUC 0.86, 95% CI 0.83-0.89).

These findings align with prior studies identifying gradient-boosting algorithms as particularly well-suited for structured medical data due to their robustness against class imbalance and capacity for regularization [19,22]. A recent systematic review also identified ensemble methods such as XGBoost among the top-performing models, despite heterogeneity in study designs and outcomes [26]. Our model (AUC 0.98) compares favorably with LMIC-based studies, including a regression nomogram from Ethiopia (AUC 0.81) [25] and an electronic medical record–based model from Uganda with moderate performance [24]. Together, these findings suggest that gradient boosting may outperform traditional regression approaches in LMIC settings, although external validation remains necessary.

Feature importance analysis further reinforced clinical relevance, with CRP, PLT, and GA identified as top predictors, consistent with established biomarkers for early sepsis detection [3,18].

Decision trees, while slightly less accurate (accuracy 80%, 95% CI 77%-83%; AUC 0.86, 95% CI 0.83-0.89), provided interpretable decision pathways that may support clinical decision-making and bridge the gap between algorithmic outputs and bedside practice [22].

Class imbalance, a pervasive issue in medical ML, was mitigated using SMOTE oversampling and class-weighted loss functions, improving sensitivity to sepsis cases (recall 0.98, 95% CI 0.96‐0.99) without compromising specificity [17,18]. Overfitting risks were addressed via L1/L2 regularization in XGBoost and neural networks, alongside constrained tree depth in decision trees [17,19]. Cross-validation ensured internal generalizability, although the single-center, retrospective design limits external validity. Collectively, these strategies may enhance model reliability and adhere to best practices for ML implementation in LMICs [15,20-26].

This study demonstrates high diagnostic accuracy, with XGBoost achieving an AUC of 0.98 (95% CI 0.97-0.99), surpassing conventional biomarker-based approaches and enabling timely identification of NS. The model effectively identifies actionable predictors, including CRP, PLT, and GA, which align with clinical workflows and support targeted monitoring. Additionally, the study effectively addressed class imbalance using SMOTE and class-weighted loss functions, improving detection of sepsis cases without compromising performance for the majority class. Collectively, these methodological innovations may enhance the robustness and clinical relevance of the predictive framework for LMIC neonatal populations.

The retrospective, single-center design may introduce selection bias and limits generalizability to other NICU settings or populations. The analysis was restricted to culture-positive cases, which may exclude culture-negative sepsis, limiting the ability to predict sepsis in its earliest stages. The timing of feature availability relative to sepsis onset is also uncertain, which may impact true early prediction. Although temporal constraints were applied to all procedural variables included in the predictive models, some laboratory and intervention-related variables may still partially reflect clinician-driven diagnostic or treatment decisions rather than purely biological manifestations of NS. Consequently, feature importance should be interpreted as reflecting predictive utility within the study cohort rather than causal relationships. Although cross-validation reduces overfitting, external validation in multicenter prospective cohorts is necessary to confirm model performance. Exclusive breastfeeding at discharge was included only in descriptive analyses and feature importance. It was not available at the time of prediction and is not a clinically valid predictor for real-time sepsis detection due to temporal misalignment. An additional limitation relates to missing data handling. Although mean and mode imputation provided a simple and consistent preprocessing strategy across all ML models, this approach assumes that missing values can be adequately represented by measures of central tendency. Because missingness in EHRs is often informative and not completely at random, the present results should be interpreted with this limitation in mind. Future studies should investigate advanced imputation techniques, such as multiple imputation, missing-indicator approaches, sensitivity analyses under different missingness assumptions, and missingness-aware learning methods, to evaluate their impact on predictive performance and model robustness. Finally, algorithmic transparency and data privacy considerations could influence clinical trust and adoption, highlighting the need for interpretability tools and secure integration into EHR systems.

Prospective multicenter validation is essential to ensure the robustness and generalizability of the models across diverse geographic and demographic neonatal cohorts. Recent comprehensive reviews have highlighted both the potential and the challenges of implementing AI for NS in resource-limited settings, emphasizing the need for context-specific solutions and robust external validation [27]. Integrating AMR data into ML frameworks could optimize antibiotic stewardship and support clinical decision-making in LMIC settings. Region-specific model tuning, using localized datasets and tailored algorithms, could enhance predictive accuracy and applicability for neonatal populations with unique epidemiological characteristics. Furthermore, the adoption of interpretability tools such as Shapley Additive Explanations (SHAP) or Local Interpretable Model-Agnostic Explanations (LIME) can improve clinician trust, facilitate integration into EHR systems, and support transparent, actionable insights for bedside decision-making. These interpretability considerations align with recommendations from recent scoping reviews, which emphasize that clinician trust and model transparency are critical barriers to real-world implementation of ML for NS [28].

This study represents the first ML-driven approach tailored to NS prediction in Jordan, addressing diagnostic gaps and highlighting the need for region-specific models in LMICs. Our findings demonstrate that ML models, particularly XGBoost, can integrate heterogeneous clinical, laboratory, and demographic data to provide actionable predictions, potentially enabling earlier interventions and improved neonatal outcomes. Although challenges such as class imbalance, data limitations, and ethical considerations remain, methodological innovations in regularization, oversampling, and interpretability may enhance model reliability and usability. Future work should focus on prospective multicenter validation, incorporation of AMR patterns, and equitable model deployment to support clinical decision-making and reduce neonatal mortality in resource-limited settings.