This retrospective study was conducted with ED data from a tertiary academic hospital in Seoul, Korea. The hospital has approximately 2000 beds. The outpatient department has an average of approximately 9000 patients per day, while the ED has approximately 220 patients per day. Since the opening of its comprehensive cancer center in 2003, the hospital has a large portion of oncology patients undergoing both surgical and medical procedures. This study was approved by the institutional review board of the study site (IRB File No: SMC 2018-08-125). This report follows the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.

Data were obtained from a clinical data warehouse containing medical data for research, which enables de-identification and retrieval of patient information from electronic medical records for research purposes. It uses global standard terminology and provides near realtime data through daily updates. In addition to basic patient demographics, it provides information on tests, medications, diagnoses, and operations.

Patients who visited the ED from March 2017 through February 2019 were included in the study. Then, only febrile (body temperature >38°C) [15] and adult patients (aged ≥18 years) were included. Patients were excluded if the main reason for their visit was determined as trauma.

We used a binary composite outcome for severity. Severity was considered as death or admission to the intensive care unit after transfer from the ED.

Only laboratory test data were used as predictors, and the list of laboratory tests was selected by experts. Predictors were selected based on the typical ED process in which all possible laboratory tests could be performed after the initial assessment by physicians [16]. Figure 1 shows the general process from the typical initial process to patient discharge from the ED. This study focused on the initial process and the ordering of laboratory tests and test results in particular. Two models were developed from the viewpoint of the initial process, using laboratory test order status and laboratory test results that became available later.

The laboratory test data provide the order status and result for each laboratory test, and all the variables were categorized. Order status indicates whether a patient has an order for a laboratory test, and the test result reflects whether it was normal, abnormal, or not reported. When the test was repeatedly performed, only the first test data were included. We developed two predictive models using these laboratory test data: order status only (OSO) for the first model and order status and results (OSR) for model 2 (Figure 2).

For a group of laboratory tests that are not frequently ordered but are conducted for only a few patients, the order status information causes severe data sparsity. Rather than using the order status information for each of those tests, new variables were introduced. First, rarely ordered tests (ROTs) were identified as tests that had an ordered rate <5%. The new variable ROT was defined as the number of ROTs ordered for each patient. Likewise, for tests that generate rarely detected abnormal results (RARs; <5% of the results are abnormal), a new variable RAR was defined as the number of RARs obtained among those tests for each patient.

Patients were randomly assigned to two cohorts for model development (70%) and validation (30%), which had similar distributions with respect to the outcome. We applied and compared various machine learning methods, including random forest (RF), support vector machine, logistic regression with least absolute shrinkage and selection operator, ridge, and elastic net (EN) regularization [17-21]. For optimization, the grid search was used for all algorithms, and the stochastic gradient descent method was used for penalized regression algorithms. The Gini index was selected to measure the split quality in RF. Linear and sigmoid kernels were considered for the support vector machine. Moreover, hyperparameters for each algorithm were tuned with the accuracy measure based on 10-fold cross-validations that were repeatedly conducted 5 times to reduce the partition bias in model development.

The predictive models were evaluated with the validation cohort using various performance measures, such as the area under the receiver operating characteristic curve (AUC), area under the precision recall curve (AUPRC), balanced accuracy (BA), sensitivity, specificity, F1 score, positive likelihood ratio (PLR), and negative likelihood ratio (NLR) [22]. For each measure, we provide 95% CIs, which were estimated by bootstrapping 2000 resamples, and selected the optimal thresholds with which BA was maximized. Moreover, net reclassification improvement [23] was used to measure the incremental value of adding test results to the order status in the prediction. Comparison of the performance between the models was conducted using the bootstrap-t method [24]. The resulting predictive models were further compared with the Modified Early Warning Score (MEWS), a reference algorithm currently used to predict the severity of a patient’s condition in clinical practice [25]. The clinical usefulness of the models was demonstrated in two ways. First, we conducted Spearman correlation analysis to evaluate whether clinically meaningful variables were selected consistently across the algorithms. Second, the Kaplan-Meier method and log-rank test were used to estimate and compare survival curves between high-risk and low-risk groups as predicted from the OSR model. We used 2-sided P<.05 for statistical significance.

Class imbalance existed in our outcome data. This can lead to the classifier having poor performance because it can create bias against a class and may not able to distinguish between noise and the individuals from the minority class [26,27]. We investigated its effects on prediction performance using various scenarios in which different techniques and class ratio were considered for imbalance reduction. Three oversampling and three undersampling approaches were considered. The oversampling methods included random, synthetic minority oversampling technique, and adaptive synthetic sampling. The undersampling methods included random, NearMiss-2, and edited nearest neighbors [28]. Furthermore, we increased the ratio between the minority and majority classes from 1:1 to 1:10 and tried to find the best performance. For each scenario, we used EN for model development, and the performance was assessed using 100 bootstrap resamples from the original dataset.

The preprocess was conducted using R version 3.4.4 [29], and the analytic process was performed with Python version 3.6.2 (Python Software Foundation, Wilmington, DE) using pandas, numpy, sklearn, and imblearn library. More details related to the model development are available in Multimedia Appendix 1.

A total of 154,402 patients visited the ED between March 1, 2017, and February 28, 2019. Based on the inclusion and exclusion criteria, 9491 patients remained in the final dataset used for modeling (Multimedia Appendix 2). The randomly divided model development and validation cohorts included 6645 and 2846 patients, respectively, with a composite adverse outcome frequency of 4.6% in the 2 datasets. The baseline characteristics reflect only the initial patient status. The mean age (SD) was 55.2 years (17.7 years); 4839 of the 9491 patients (51.0%) were female; and 432 of the 9491 patients (4.6%) experienced the composite adverse outcome. Patients in the development and validation cohorts were similarly distributed (Table 1).

The three most frequently observed laboratory tests were C reactive protein, chlorine, and sodium. Among a total of 286 laboratory tests after preprocessing, 201 ROTs (order rate <5%) and 231 tests with RARs (abnormal rate <5%) were identified. The OSO model had 85 order status variables as well as the ROT variable. Similarly, the OSR model had 55 result variables, the RAR variable, and the variables in the OSO model.

The OSO and OSR models were each developed based on 5 different algorithms. The RF-based models were selected as the final predictive OSO and OSR models because they had better performance overall in terms of the most evaluation measures, including specificity, precision, F1 score, NLR, and PLR (Multimedia Appendix 3). Note that the EN-based OSR model was comparable to the RF-based OSR model. Compared to the MEWS (AUC = 0.68), the final OSO and OSR models showed significant AUC improvement, at 12% and 20%, respectively (Table 2, Multimedia Appendix 4). Both models had better performance than the MEWS according to most of the other measures, including the AUPRC, maximum BA, and F1 score.

Compared with the OSO model, the OSR model showed significant improvement in the AUC, at 8%, and maximum BA, at 7%. Additionally, the OSR model was more informative than the OSO model in predicting low-risk and high-risk patients in terms of outcome (P<.001 for difference in both PLR and NLR). A significant additional increment in reclassification was also observed between the OSO and OSR models (net reclassification improvement=0.15). Despite the lack of information from laboratory test results, the OSO model showed considerable performance (AUC = 0.80). Therefore, the order pattern itself can be important information for prediction, and it is better to use the OSO model in the early stages than to wait until test results are obtained. However, because it utilizes laboratory test results obtained later in time, the OSR model has higher accuracy and better performance than the OSO model. According to the Kaplan-Meier survival curves (log-rank test, P<.001) comparing the predicted outcome groups, the OSR model can classify patients well (Figure 3). The complementary use of these two models can be beneficial both before and after all laboratory test results are available in clinical situations such as those in the ED.

Important variables selected from the RF-based and EN-based models were moderately correlated in terms of their value importance and odds ratios, respectively (r_s=0.603 and 0.626 for the OSO and OSR models, respectively; Multimedia Appendix 5). Among the top 10 variables selected in each of the RF-based and EN-based models, 80% and 60%, respectively, were shared by the OSO and OSR models. Therefore, the important variables were very similar between the RF and EN models, potentially suggesting our models are robust, regardless of the algorithm used. The order statuses of cardiac troponin I, creatine kinase, and creatine kinase-MB were the top 3 variables in terms of importance in both the RF-based and EN-based OSO models. The order status of creatine kinase remained in the top 10 important variables in the OSR models. The lactic acid test result was the most important variable in both the RF-based and EN-based OSR models.

The data had severe outcome imbalance: 95.4% (9059) for the majority class and 4.6% (432) for the minority class. However, the sensitivity analysis to calibrate the imbalance with various reduction scenarios did not reveal any considerable improvement in the prediction performance. Therefore, our models are not affected much by the imbalance problem (Multimedia Appendix 6).

In this study, we developed a time adaptive model to predict adverse outcomes for patients in the ED. These patients are likely to have insufficient and unconfirmed clinical information, especially in the early stages of the ED process. The OSO model, which only utilizes test order status, supports our hypothesis that it is feasible to predict patient prognosis based only on the fact that a laboratory test has been ordered and without the test results. Patient demographics or vital signs were also not required for the prediction.

Febrile patients have a considerable number of laboratory tests to consider. The ED receives patients with different illnesses and febrile patients with various diseases in particular. Fever is also the most common sign of potential sepsis [30,31], requiring more laboratory tests. Febrile patients were selected as the target population to investigate the sparsity and large number of tests typically required initially and to reflect as many tests as possible.

The OSO model mimics the ED physician’s clinical reasoning process in practical settings, while prediction models developed previously are limited by using only confirmed results [11-14]. For time-sensitive conditions, multiple tests are performed simultaneously, resulting in a combination of confirmed and unconfirmed results, which necessitates models that can be applied in real practice. This study is the first step to overcome these limitations. Furthermore, it is possible to predict the initial outcome of patients with a severe condition using the OSO model and then update the prediction using the OSR model when additional information becomes available in a time adaptive manner.

In modern medicine, a multidisciplinary approach is a cornerstone of better quality [32,33]. In the ED, multiple providers work as a team to simultaneously treat various conditions [7]. Analyzing laboratory test orders and results could benefit the whole treatment team by interpreting the initial impression and intention of physicians who give the orders for tests. It is possible that each physician may not be aware of the intention of others while simultaneously treating the same patient [7,34,35]. In this context, the prediction tool can help share information, lead dynamic communication, and consequently prevent medical errors.

This study could be expanded further by including vital signs, procedures, and medications for better prediction. In addition, the application could be broadened to include diagnosis as well as adverse outcomes, especially for diseases where the patient’s response over time after a particular treatment is important. Additionally, it can be extended to anticipate clinical decisions, which may be integrated as clinical decision support. The time variable is the most essential component for these predictions, and this model has successfully shown its feasibility.

This study has some limitations. First, the models were developed and internally validated using data from a single large hospital. Although cross-validation was performed with repetition, optimization, and several candidates of hyperparameters, along with survival analysis to increase their clinical impact, further studies are required for external and prospective validation.

Second, the primary parameters such as laboratory results, which may vary across individuals and clinical fields, were from febrile patients. Therefore, it could be difficult to apply these to other populations, although we attempted to include as many tests as possible. However, we believe the important variables that were selected related to laboratory tests from the 2 models are clinically relevant for the outcome variables, so there is potential to extend the models to other target populations in future studies.

Third, the OSO and OSR models were not developed with a continuous time sequence. Instead of creating a continuous model, we tried to build representative models to reflect test order status and results. Further research will be required to create a continuous model for practical use, which can be applied to various time thresholds.

Last, the imbalance of data could have affected the performance of models developed using raw data. Although various methods to deal with the issues related with imbalanced data were applied to develop and validate the model, only a few of algorithms among the methods for calibrating the class imbalance were used. It is possible that the use of other algorithms would have changed the results even though the results in this study were not significantly improved after addressing the issue of imbalanced data. Therefore, various additional algorithms should be used to address the imbalanced data in future studies.

Adverse outcomes during the early stages for febrile patients could be predicted using a time adaptive model and machine learning approach based on the highly sparse data from test order status and laboratory test results.