This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
Delirium frequently occurs among patients admitted to the intensive care unit (ICU). There is limited evidence to support interventions to treat or resolve delirium in patients who have already developed delirium. Therefore, the early recognition and prevention of delirium are important in the management of critically ill patients.
This study aims to develop and validate a delirium prediction model within 24 hours of admission to the ICU using electronic health record data. The algorithm was named the Prediction of ICU Delirium (PRIDE).
This is a retrospective cohort study performed at a tertiary referral hospital with 120 ICU beds. We only included patients who were 18 years or older at the time of admission and who stayed in the medical or surgical ICU. Patients were excluded if they lacked a Confusion Assessment Method for the ICU record from the day of ICU admission or if they had a positive Confusion Assessment Method for the ICU record at the time of ICU admission. The algorithm to predict delirium was developed using patient data from the first 2 years of the study period and validated using patient data from the last 6 months. Random forest (RF), Extreme Gradient Boosting (XGBoost), deep neural network (DNN), and logistic regression (LR) were used. The algorithms were externally validated using MIMIC-III data, and the algorithm with the largest area under the receiver operating characteristics (AUROC) curve in the external data set was named the PRIDE algorithm.
A total of 37,543 cases were collected. After patient exclusion, 12,409 remained as our study population, of which 3816 (30.8%) patients experienced delirium incidents during the study period. Based on the exclusion criteria, out of the 96,016 ICU admission cases in the MIMIC-III data set, 2061 cases were included, and 272 (13.2%) delirium incidents occurred. The average AUROCs and 95% CIs for internal validation were 0.916 (95% CI 0.916-0.916) for RF, 0.919 (95% CI 0.919-0.919) for XGBoost, 0.881 (95% CI 0.878-0.884) for DNN, and 0.875 (95% CI 0.875-0.875) for LR. Regarding the external validation, the best AUROC were 0.721 (95% CI 0.72-0.721) for RF, 0.697 (95% CI 0.695-0.699) for XGBoost, 0.655 (95% CI 0.654-0.657) for DNN, and 0.631 (95% CI 0.631-0.631) for LR. The Brier score of the RF model is 0.168, indicating that it is well-calibrated.
A machine learning approach based on electronic health record data can be used to predict delirium within 24 hours of ICU admission. RF, XGBoost, DNN, and LR models were used, and they effectively predicted delirium. However, with the potential to advise ICU physicians and prevent ICU delirium, prospective studies are required to verify the algorithm’s performance.
Delirium, defined as acute brain dysfunction characterized by disturbances of awareness, attention, and cognition with a fluctuating course linked with an underlying medical condition, frequently occurs among patients admitted to intensive care units (ICUs) [
There is currently limited evidence to support interventions to treat or resolve delirium in patients who have already developed delirium [
Therefore, we developed a machine learning–based model for the early prediction of delirium among medical and surgical ICU patients using electronic health record (EHR) data. This prediction model uses data obtained within 4 hours of ICU admission to predict delirium within 24 hours after ICU admission.
We conducted a retrospective study of all critically ill patients admitted to the ICUs of the Samsung Medical Center (a 1989-bed university-affiliated, tertiary referral hospital in Seoul, South Korea) from July 1, 2016, to August 31, 2019. We only included patients who were 18 years or older at the time of admission and who stayed in the medical or surgical ICU. Patients were excluded if they lacked a Confusion Assessment Method for the ICU (CAM-ICU) record from the day of ICU admission or if they had a positive CAM-ICU record at the time of ICU admission. The flow diagram in
Flow diagram of the participant selection process. CAM-ICU: confusion assessment method for the intensive care unit; ICU: intensive care unit.
This study used data from the Clinical Data Warehouse Darwin-C database of the SMC and the Medical Information Mart for Intensive Care III (MIMIC-III) database (v1.4). The SMC data set was used for the derivation and validation cohort, and the MIMIC-III data set was used for the external validation cohort. The MIMIC-III database is a clinical database consisting of data from more than 38,000 ICU patients (medical, surgical, trauma-surgical, coronary, and cardiac-surgery data) admitted to Beth Israel Deaconess Medical center (Boston, MA) from June 2001 to October 2012 [
To screen for delirium, all ICU patients were assessed with the CAM-ICU [
We used clinical characteristics, ICU admission category (medical or surgical), primary cause of admission (respiratory, cardiovascular, gastrointestinal, neurology, perioperative, nephrology, metabolic, or trauma), primary diagnosis, vital signs, prescription medications, and laboratory test results as the predictor variables. All variables were extracted from the EHR data set.
We first extracted all relevant variables for the prediction model from other studies. Next, 2 clinical experts (CRC and REK) reviewed the relevant variables and selected the crucial ones based on previous clinical studies and clinical relevance. We then further restricted the variables depending on whether they could be automatically extracted from EHRs and had low missing rates. Finally, for the external validation in the MIMIC-III data set, we selected variables found in both SMC and MIMIC-III. The MIMIC-III data set shows the final variables used as input for model development. The list of variables used is shown in
Age, sex, and invasive mechanical ventilation
Medical intensive care unit (ICU) or surgical ICU
Respiratory, cardiovascular, gastrointestinal, neurology, perioperative, nephrology, metabolic, and trauma
Systolic blood pressure, diastolic blood pressure, heart rate, respiratory rate, peripheral capillary oxygen saturation, and Glasgow Coma Scale (eye, verbal, and motor)
Charlson Comorbidity Index
Complete blood count:
white blood count, hemoglobin, hematocrit, platelet count, and erythrocyte sedimentation rate
Coagulation:
prothrombin time (INR) and activated partial thromboplastin time
Chemistry:
Total protein, albumin, total bilirubin, aspartate aminotransferase, alanine aminotransferase, glucose fasting, blood urea nitrogen, creatinine, phosphorus, sodium, potassium, magnesium, calcium (ionized), C-reactive protein quantitative, and lactic acid
Arterial Blood Gas Analysis:
pH, PaCO2, PaO2, HCO3, and O2 Saturation
Antibiotics, anticholinergic and antipsychotics, benzodiazepines, miscellaneous antidepressants, anxiolytics, sedatives and hypnotics, vasopressors, opiate agonists, opiate antagonists, cholinergic agents, and steroids
With regard to the general data processing, we first processed invalid values by eliminating them. Invalid values include extreme outliers in numerical values (for example, numerical values for vitals are eliminated using certain rules (ie, heart rate values should be between 0 and 300). Second, we processed numerical values by normalizing and scaling them. We performed standard normalization and min-max scaling such that the final numerical values were between 0 and 1. Finally, we processed the missing values. Missing values in the numerical data were filled with mean values, whereas missing values in categorical data were left blank such that the dummy variables were all equal to 0.
For certain variables with temporal information, such as vital values and laboratory test results, we determined statistical values such as the mean, standard deviation, min, max, and the closest values to the ICU admission to ensure multiple rows of numerical values can be summarized into one. Subsequently, to reduce the number of features necessary to train the model, we picked only one of the statistical values according to the feature importance of the random forest (RF). For example, there were initially multiple values for diastolic blood pressure (DBP) with respect to time. We calculated statistical values such as the mean, standard deviation, min, max, and the latest DBP values. Finally, we only selected the mean DBP because it was the most important among the statistical values of the DBP according to the feature importance of the RF.
We split the data set into a development data set and a data set. For the development data set, we used the data obtained between July 1, 2016, and December 31, 2018. For the validation set, we used the data obtained between January 1, 2019, and August 31, 2019. Of the 37,543 admitted cases, 12,409 cases were selected in this study. These were divided into the development set (n=9589, 77.3%) and the internal validation set (n=2820, 22.7%). Among the 9589 cases in the development data set, there were 3060 (31.9%) cases of delirium, and among the 2820 cases in the validation data set, there were 756 (26.8%) cases of delirium. We did not apply specific methodology (eg, undersampling) to resolve the outcome imbalance problem because it was not extreme.
We employed RF, extreme gradient boosting (XGBoost), deep neural network (DNN), and logistic regression (LR) as the candidate prediction models.
We also used an automated machine learning called the Tree-based Pipeline Optimization Tool for model selection and parameter searching [
For DNN, we used 512, 256, and 128 neurons for hidden layers, ReLU function for activation function in hidden layers, sigmoid function for activation function in the output layer, and binary cross-entropy function as the loss function. For XGBoost, we used a tree booster with 100 estimators, the learning rate as 0.1, and the subsample ratio as 0.75.
After development and internal validation, we performed the external validation of our delirium prediction model using the MIMIC-III database. The validation set was extracted from the MIMIC-III database, which included patients with at least two CAM-ICU records obtained within at least 24 hours.
The model with the highest area under the receiver operating characteristics (AUROC) curve in the external validation was named the PRIDE (Prediction of ICU Delirium) algorithm.
Continuous variables are presented in terms of means and SD, and categorical variables are presented in terms of their frequencies and percentages. The performances of the different models were compared using the AUROC, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) at the threshold. In the internal validation, model performance was evaluated through the average and 95% CI of the AUROCs. Additionally, we used a calibration curve and the Brier score to test the reliability of our model. To determine the clinically relevant threshold, we used a decision curve.
We employed the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) statement to report the results of our prediction model. Data processing, statistical analysis, and the development and validation of the machine learning algorithms were performed using R version 3.6.2 [
The source code has been made available on Github [
During the study period, a total of 37,543 cases were collected. Patients who were 18 years or older at the time of ICU admission were included. Cases with less than two CAM-ICU records after admission to the ICU and those with a positive CAM-ICU upon ICU admission were excluded. After patient exclusion, 12,409 remained as our study population. The case group consisted of 3816 (30.8%) patients who experienced delirium incidents during the study period. With regard to the MIMIC-III (external validation) data set, patients younger than 18 years of age, those with less than two CAM-ICU records recorded within 24 hours, and those with positive CAM-ICU records upon ICU admission were excluded. Based on the exclusion criteria, out of the 96,016 ICU admission cases, 2061 cases were included, and 272 (13.2%) delirium incidents occurred.
Baseline characteristics of the training and test sets of the SMC and MIMIC-III data sets are shown in
Baseline characteristics of data sets.
Characteristics | Development | Internal validation | External validation (MIMIC-IIIa) | ||||||||||
|
Case (n=3060) | Control (n=6529) | Case (n=756) | Control (n=2064) | Case (n=272) | Control (n=1789) | |||||||
Age (years), mean (SD) | 65.4 (14.4) | 61.1 (13.3) | 65.3 (14.7) | 59.7 (14.0) | 82.5 (59.7) | 73.7 (52.3) | |||||||
Sex (male), n (%) | 1994 (65.2) | 4227 (64.7) | 466 (61.6) | 1190 (57.7) | 137 (50.4) | 925 (51.7) | |||||||
|
|||||||||||||
|
Medical | 1431 (46.8) | 1639 (25.1) | 293 (38.8) | 344 (16.7) | 151 (55.5) | 1,141(63.8) | ||||||
|
Surgical | 1629 (53.2) | 4890 (74.9) | 1,720 (83.3) | 463 (61.2) | 121 (44.5) | 648(36.2) | ||||||
|
|||||||||||||
|
Respiratory | 692 (22.6) | 374 (5.7) | 166 (22.0) | 71 (3.4) | 34 (12.5) | 247 (13.8) | ||||||
|
Cardiovascular | 480 (15.7) | 1166(17.9) | 105 (13.9) | 283 (13.7) | 73 (26.8) | 561 (31.4) | ||||||
|
Gastrointestinal | 145 (4.7) | 139 (2.1) | 28 (3.7) | 25 (1.2) | 55 (20.2) | 379 (21.2) | ||||||
|
Neurology | 102 (3.3) | 115 (1.8) | 46 (6.1) | 174 (8.4) | 57 (21.0) | 270 (15.1) | ||||||
|
Peri-operation | 1168 (38.2) | 4278 (65.5) | 320 (42.3) | 1448 (70.2) | 14 (5.1) | 81 (4.5) | ||||||
|
Nephrology | 83 (2.7) | 78 (1.2) | 17 (2.2) | 17 (0.8) | 4 (1.5) | 54 (3.0) | ||||||
|
Metabolic | 15 (0.5) | 6 (0.1) | 1 (0.1) | 0 (0.0) | 5 (1.8) | 85 (4.8) | ||||||
|
Hematology | 15 (0.5) | 22 (0.3) | 4 (0.5) | 6 (0.3) | 5 (1.8) | 33 (1.8) | ||||||
|
Trauma | 10 (0.3) | 10 (0.2) | 2 (0.3) | 0 (0.0) | 25 (9.2) | 79 (4.4) | ||||||
|
Others | 350 (11.4) | 341 (5.2) | 67 (8.9) | 40 (1.9) | — | — | ||||||
Initial SOFAc, mean (SD) | 7.1 (3.6) | 3.2 (2.7) | 7.0 (3.7) | 2.8 (2.5) | 5.4 (3.3) | 3.1 (2.4) | |||||||
Vasopressord, n (%) | 1,145 (37.4) | 793 (12.1) | 292 (38.6) | 175 (8.5) | 38 (14.0) | 71 (4.0) | |||||||
Invasive mechanical ventilator, n (%) | 1,900 (62.1) | 1,165 (17.8) | 456 (60.3) | 344 (16.7) | 19 (7.0) | 73 (4.1) | |||||||
CCIe, mean (SD) | 0.9 (2.2) | 0.3 (1.3) | 1.1 (2.5) | 0.4 (1.4) | 3.2 (1.8) | 2.8 (1.8) | |||||||
|
|||||||||||||
|
Heart disease | 149 (4.9) | 123 (1.9) | 17 (2.2) | 20 (1.0) | 33 (12.1) | 232 (13.0) | ||||||
|
Stroke | 98 (3.2) | 33 (0.5) | 29 (3.8) | 24 (1.2) | 16 (5.9) | 57 (3.2) | ||||||
|
Malignancy | 434 (14.2) | 319 (4.9) | 80 (10.6) | 60 (2.9) | 5 (1.8) | 12 (0.7) | ||||||
|
Renal failure | 71 (2.3) | 99 (1.5) | 16 (2.1) | 26 (1.3) | 17 (12.1) | 106 (12.7) | ||||||
|
Liver disease | 146 (4.8) | 92 (1.4) | 23 (3.0) | 15 (0.7) | 38 (14.0) | 168 (9.4) | ||||||
|
Dementia | 55 (1.8) | 9 (0.1) | 17 (2.2) | 6 (0.3) | — | — | ||||||
|
|||||||||||||
|
Systolic BPf | 125.8 (24.7) | 127.4 (21.7) | 128.6 (26.4) | 130.8 (22.2) | 118.5 (25.0) | 119.5 (22.7) | ||||||
|
Diastolic BP | 71.6 (15.5) | 74.3 (14.1) | 72.3 (17.9) | 73.3 (14.7) | 63.3 (15.9) | 63.9 (15.4) | ||||||
|
Heart rate | 85.4 (20.7) | 80.7 (16.6) | 84.5 (20.5) | 80.6 (16.1) | 89.9 (20.3) | 85.8 (19.4) | ||||||
|
Respiratory rate | 19.3 (3.9) | 18.3 (2.5) | 18.8 (3.7) | 17.7 (2.5) | 20.5 (6.5) | 19.2 (5.4) | ||||||
|
SpO2g | 96.1 (5.5) | 96.9 (4.3) | 96.5 (4.0) | 97.4 (2.1) | 95.9 (3.6) | 96.2 (3.1) | ||||||
|
Body temperature (°C) | 36.7 (0.8) | 36.6 (0.5) | 36.7 (0.8) | 36.6 (0.4) | 36.8 (0.7) | 36.8 (0.6) | ||||||
|
|||||||||||||
|
pH | 7.4 (0.1) | 7.4 (0.1) | 7.4 (0.1) | 7.4 (0.1) | 7.4 (0.1) | 7.4 (0.1) | ||||||
|
PaCO2 | 35.5 (14.1) | 36.2 (7.5) | 35.3 (11.3) | 36.4 (6.2) | 42.0 (12.4) | 41.1 (12.1) | ||||||
|
PaO2 | 119.8 (83.4) | 184.4 (110.8) | 122.0 (82.5) | 190.4 (104.8) | 131.9 (106.0) | 125.0 (67.9) | ||||||
|
HCO3 | 22.0 (5.5) | 23.1 (3.6) | 22.0 (5.3) | 23.5 (2.9) | 24.1 (5.4) | 24.2 (4.7) |
aMIMIC-III: Medical Information Mart for Intensive Care III.
bICU: intensive care unit.
cSOFA: sequential organ failure assessment.
dVasopressor: epinephrine, norepinephrine, dobutamine, dopamine, vasopressin.
eCCI: Charlson Comorbidity Index.
fBP: blood pressure.
gSpO2: peripheral capillary oxygen saturation.
hABGA: arterial blood gas analysis.
The average AUROCs and 95% CIs for internal validation were 0.919 (95% CI 0.919-0.919) for XGBoost, 0.916 (95% CI 0.916-0.916) for RF, 0.881 (95% CI 0.878-0.884) for DNN, and 0.875 (95% CI 0.875-0.875) for LR. For each model, we selected the highest value of specificity among sensitivities over 0.9 as the cut-off point for the threshold. The best model for the internal validation was XGBoost, with an AUROC of 0.919 (95% CI 0.919-0.919). Its sensitivity, specificity, PPV, and NPV were 0.904 (95% CI 0.904-0.905), 0.731 (95% CI 0.729-0.732), 0.565 (95% CI 0.563-0.566), and 0.952 (95% CI 0.952-0.952), respectively.
For the external validation, the average AUROCs and 95% CI were 0.721 (95% CI 0.72-0.721) for RF, 0.697 (95% CI 0.695-0.699) for XGBoost, 0.655 (95% CI 0.654-0.657) for DNN, and 0.631 (95% CI 0.631-0.631) for LR. For the external validation on the MIMIC-III database, the model with the best AUROC was the RF model, with an AUROC of 0.721 and a sensitivity, specificity, PPV, and NPV of 0.91 (95% CI 0.909-0.912), 0.27 (95% CI 0.266-0.273), 0.159 (95% CI 0.159-0.16), and 0.952 (95% CI 0.951-0.953), respectively. A comparison of the performances of all of the models is shown in
For the external validation with MIMIC-III, we only selected variables that could be found both in SMC and MIMIC-III. As a result, only 59 variables were selected. The variables were categorized into general information, flowsheet, laboratory test results, and prescription of medication. The most important variable was the use of invasive mechanical ventilation in the general information. The importance of each final variable used in model development is shown in
Predictive performance of each model.
Model and data set | AUROCa, mean (95% CI) | Sensitivity, mean (95% CI) | Specificity, mean (95% CI) | Positive predictive value, mean (95% CI) | Negative predictive value, mean (95% CI) | |
|
||||||
|
Internal data set | 0.916 (0.916-0.916) | 0.904 (0.904-0.905) | 0.746 (0.744-0.747) | 0.579 (0.578-0.580) | 0.953 (0.952-0.953) |
|
External data set | 0.721 (0.720-0.721) | 0.910 (0.909-0.912) | 0.270 (0.266-0.273) | 0.159 (0.159-0.160) | 0.952 (0.951-0.953) |
|
||||||
|
Internal data set | 0.919 (0.919-0.919) | 0.904 (0.904-0.905) | 0.731 (0.729-0.732) | 0.565 (0.563-0.566) | 0.952 (0.952-0.952) |
|
External data set | 0.697 (0.695-0.699) | 0.908 (0.906-0.909) | 0.250 (0.245-0.255) | 0.156 (0.155-0.156) | 0.946 (0.945-0.947) |
|
||||||
|
Internal data set | 0.881 (0.878-0.884) | 0.906 (0.905-0.907) | 0.622 (0.608-0.635) | 0.485 (0.477-0.492) | 0.944 (0.943-0.945) |
|
External data set | 0.655 (0.654-0.657) | 0.907 (0.905-0.908) | 0.197 (0.192-0.201) | 0.147 (0.146-0.147) | 0.932 (0.931-0.933) |
|
||||||
|
Internal data set | 0.875 (0.875-0.875) | 0.901 (0.901-0.901) | 0.605 (0.605-0.605) | 0.469 (0.469-0.469) | 0.940 (0.940-0.940) |
|
External data set | 0.631 (0.631-0.631) | 0.904 (0.904-0.904) | 0.155 (0.155-0.155) | 0.140 (0.140-0.140) | 0.914 (0.914-0.914) |
aAUROC: area under the receiver operating characteristic curve.
bXGBoost: extreme gradient boosting.
Receiver operating characteristic curves for all the prediction of intensive care unit delirium models. AUROC: area under the receiver operating characteristic curve; XGBoost: extreme gradient boosting.
Variable importance of the prediction of intensive care unit delirium model. ALT: alanine aminotransferase; aPTT: activated partial thromboplastin time; AST: aspartate aminotransferase; BUN: blood urea nitrogen; CRP: C-reactive protein; GCS: Glasgow Coma Scale; ICU: intensive care unit; INR: international normalized ratio.
For further model evaluation, calibration and decision curve analyses were performed. The Brier score for the XGBoost model with regard to predicting delirium was 0.094 for the internal validation data set, indicating that our model is reliable. The best model for external validation is RF, with the Brier score of 0.168. A Brier score of 0 indicates a perfect calibration, and the closest the value is to 0, the better model calibration. The calibration plot is shown in
We have demonstrated that the proposed delirium prediction model, which employs a machine learning algorithm with EHR data, can predict the development of delirium in medical and surgical ICU patients. In addition to our internal validation, we externally validated our findings using the MIMIC-III patient database. With the PRIDE model, we showed that delirium prediction models could be automated exclusively using risk factors derived from EHR data. The three main results of our study are as follows: (1) the model predicted delirium within the first 24 hours of ICU admission by only using data collected within the first 4 hours after ICU admission, (2) all variables were extracted from EMR data obtained from both medical and surgical intensive care patients, and (3) the model showed acceptable performance with regard to the external validation data set.
Among the various departments in a hospital, the incidence of delirium is the highest in the ICU, and it is well-documented that delirium occurs in 25% of critically ill adults in ICUs within the first 24 hours after admission [
Owing to the prevalence of delirium in patients admitted to ICUs, the routine use of preventive measures for delirium is recommended. However, previous studies have shown that clinicians’ predictions of the development of delirium are less accurate than those of ICU delirium prediction models [
Previous studies have used several risk factors for delirium in ICUs, including age, severity score, cause of admission, usage of sedative agents, and laboratory results. In contrast with previous studies, the PRIDE model includes several additional variables such as vital signs (heart rate and blood pressure) and medication information that is excluded from EHRs. These differences allow our model to predict delirium incidents within 4 hours of ICU admission only using EHR data. Further, the PRIDE model did not include a severity score because this can only be obtained after 24 hours of ICU admission; in addition, since this information is separate from EMR data, using a severity score would require further efforts by the clinician. A few reports have also presented EMR-based machine learning models to predict delirium [
The strength of our study is the EMR-driven model that was both internally and externally validated, using SMC and MIMIC-III data, respectively. Although our result showed lower accuracy with external data than internal data, this result can be improved if the missing rate of key features decreases. In the case of CAM-ICU, 96% was missing in MIMIC-III. In addition, a decrease in accuracy with an external database was not uncommon in literature [
In clinical settings, missing values occur for various reasons. To handle missing data, we used mean values in the numerical data. We left the missing values in the categorical data blank such that the dummy variables were all equal to 0 method. Recently, deep learning–based advanced techniques, such as long short-term memory and recurrent neural network, were also introduced to impute missing data, and by employing these methods, they could improve model performances [
There are potential limitations to our study that should be acknowledged. First, our study was retrospectively performed and validated. Prospective interventional studies are needed to verify the performance of the model and to reconfirm its clinical usefulness. Second, a selection bias might exist because we selected variables available in all cohorts, and this study was conducted in a retrospective manner. Furthermore, we excluded patients without CAM-ICU data (47% of the total number of ICU-admitted patients). In this regard, it should be noted that the purpose of this study was to develop a readily available model; therefore, we only selected the variables that could be used commonly in all cohorts. Finally, although the CAM-ICU tool is regarded as highly sensitive and specific to the detection of ICU delirium, it has critical limitations. As it only has binary labels, we cannot access the degree of delirium exacerbation. Furthermore, it is recorded in a “point-in-time” manner; thus, there may be some patients whose CAM-ICU tests were missed because they were completed outside the study’s time frame [
We have developed and validated the delirium prediction model, which can predict the occurrence of delirium within 24 hours of ICU admission, using clinical data obtained in the first 4 hours after ICU admission. The PRIDE algorithm has acceptable AUCROC and sensitivity; thus, it has the potential to help advise ICU physicians and prevent ICU delirium.
Institutional review board approval form.
The missing rate in the variable list.
Calibration curve of the best model.
Decision curve of the best model. PRIDE: prediction of intensive care unit delirium model.
area under the receiver operating characteristic curve
confusion assessment method for the intensive care unit
diastolic blood pressure
deep neural network
electronic health record
intensive care unit
institutional review board
logistic regression
Medical Information Mart for Intensive Care III
negative predictive value
Prediction of ICU Delirium
positive predictive value
random forest
Samsung Medical Center
extreme gradient boosting
This research was supported by a Korea Health Technology R&D Project grant through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health and Welfare, Republic of Korea (grant HI19C0275). REK, WCC, and CRC contributed to the conception and study design. SH, JY, JH, WCC, and CRC performed the data analysis and interpretation. SH, REK, and CRC drafted the manuscript for intellectual content. SH, REK, WCC, JY, JH, and CRC revised the manuscript. All authors have read and approved the final manuscript.
SH and REK are co-first authors of this study. CRC (chiryang.chung@gmail.com) and WCC (docchaster@gmail.com) are co-corresponding authors for this article.
None declared.