This retrospective study was conducted at the Presbyterian Medical Center in the Republic of Korea, where a rapid response system (RRS) had not yet been established. Instead, the condition of each patient was monitored by an assigned registered nurse for potential clinical deterioration. In March 2023, the hospital introduced an AI solution named VitalCare (VC) for continuous patient monitoring in the general ward (GW) and ICU to predict patient deterioration. The study population consisted of patients aged 19 years and older who were admitted to the GW or ICU over a 17-month period from December 2022 to May 2024. A “silent period” was designated from December 2022 to February 2023 to establish control settings, during which health care providers did not use VC. Excluding the first 1-month adaptation period following the introduction of the system, data from April 2023 to May 2024 were set as the “entire alert period.” We classified and analyzed the intervention period into two categories. To control for seasonal bias, the alert period was defined as the winter months between December 2023 and February 2024 to match the control period, which was also confined to the winter months. Winter was associated with a higher incidence of AEs and mortality. Therefore, this study was designed to focus on comparative analyses specific to winter. To evaluate the overall trend in Code Blue occurrences, a comparative analysis was conducted using 5 years of event data. This analysis compared the 4-year average before the implementation of VC with data from the first year after implementation.

The Institutional Review Board of the Presbyterian Medical Center approved this study on December 12, 2023, and waived the requirement for informed consent given the specific nature of the retrospective study (2023-12-051). This study was conducted in accordance with the principles of the 1975 Declaration of Helsinki. To protect the privacy of participants, all data were anonymized during extraction from the hospital’s EMR system and used solely for analysis purposes. As this was a retrospective study, no compensation was provided to participants.

The VC algorithm is a deep-learning-based model that uses a bidirectional long short-term memory architecture as a binary classifier to predict the occurrence of major AEs (positive or negative) in GW inpatients. It comprises VC-Major Adverse Event Score (MAES) and VC-Mortality Score (MORS), both of which have been approved by the Korean Ministry of Food and Drug Safety. The VC-MAES is designed to comprehensively predict 3 key events, namely, UIT, death, or cardiopulmonary resuscitation, within the GW. In contrast, the VC-MORS is designed to provide a score that could predict death in patients who are admitted to the ICU. ICU transfers within 3 hours of surgery completion or 12 hours of procedure initiation were considered as planned ICU transfers and were excluded from the outcome analyses. The VC-MAES was trained using data from 334,185 hospitalizations of 209,825 adult patients between 2013 and 2017 at Yonsei Severance Hospital, which is a 2454-bed tertiary academic medical center in Seoul, Republic of Korea. The VC-MORS was trained using data from 21,186 hospitalizations of 19,570 adult patients who were admitted to the ICU between 2013 and 2017 at the same hospital. The dataset encompasses more than 35 surgical and medical specialties. The details of the model architecture were the same as those described in a previous study, with slight modifications [10].

The model primarily uses 5 vital signs—systolic and diastolic blood pressure, heart rate, respiratory rate, and body temperature—along with age to generate risk scores ranging from 0 to 100. Higher scores indicate an increased risk of target events occurring within the next 6 hours. When available, the model also incorporates optional variables, including oxygen saturation (SpO₂), mental status (Glasgow Coma Scale), total bilirubin, lactate, creatinine, platelet count, pH, sodium, potassium, hematocrit, white blood cell count, bicarbonate, and C-reactive protein. These inputs are updated whenever new data are recorded in the EMR system to ensure that the score reflects the patient’s most recent condition. Given that these optional parameters are infrequently measured, missing values are imputed using the last observation carried forward method, where the most recent available value is used. If no prior values are available, normal reference values are assigned for score computation [11].

We integrated the hospital network’s EMR system with the pretrained VC algorithm, which was deployed to another institution without any retraining or recalibration, thereby enabling validation using real-world data from hospitalized patients. The system was configured to allow VC access across all GWs and ICUs. Health care providers reviewed single parameters, such as vital signs, laboratory data, and VC prediction scores, for all hospitalized patients via the VC web platform. The VC web platform presents a trend graph of score changes over time, alarm history, and single-parameter history (Figure 1). It was displayed continuously, allowing the health care providers to access it at any time, and clinicians used the information provided by the VC as a reference during clinical decision-making.

We analyzed clinical variables such as the in-hospital Code Blue incidence rate, AE (death and UIT cases) incidence rate, proportion of long-term hospitalization during each period, and frequency of early interventions to evaluate the clinical effectiveness following the implementation of VC.

We investigated the annual number of Code Blue events from 2019 to 2023 and calculated the incidence rate relative to the total number of hospitalized patients, excluding those with documented “Do Not Resuscitate” status. The silent period was represented by the average values from 2019 to 2022. The pLOS was defined as the 75th percentile of the total length of stay in the study population [12,13]. Based on this criterion, patients who were admitted to the GW for 9 days or more and those who were admitted to the ICU for 4 days or more were classified as having a prolonged hospital stay. The proportion of patients meeting these criteria was subsequently calculated to evaluate the prevalence of prolonged hospital stay within the cohort.

Given the varying total number of hospital admissions across different periods, the incidence rate of AEs was calculated per 1000 admissions. The number of interventions or evaluations performed within 2 hours of the first alarm activation was calculated to analyze the frequency of early interventions. For this analysis, intervention and evaluation items were compared using prescription codes, such as new oxygenation or change orders, laboratory tests, and any new code orders.

The overall accuracy of the scores was determined by calculating the area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) based on whether a target event occurred within 6 hours of the predicted scores. The bootstrap method was used to compare the AUROC and AUPRC with the National Early Warning Score (NEWS), Modified Early Warning Score (MEWS), Sequential Organ Failure Assessment (SOFA), and Acute Physiology and Chronic Health Evaluation II (APACHE II). Given the extensive use of NEWS, MEWS, and SOFA in studies worldwide to predict cardiac arrest or clinical deterioration, these scores were also used as comparators in this study [1,2,14,15]. NEWS and MEWS were calculated using current values of systolic blood pressure, heart rate, respiratory rate, body temperature, mental status, SpO₂, and the fraction of inspired oxygen (FiO₂) recorded in the EMR within 1 hour before the prediction point. The SOFA score was calculated using the worst recorded values in the EMR within 24 hours following ICU admission. As the APACHE II score is specifically used to predict the mortality risk of acutely ill patients within 24 hours of ICU admission [16], the AUROC and AUPRC were calculated based on whether death occurred during the ICU stay.

Demographic characteristics during the silent and alert periods were compared using chi-square tests for categorical variables. For continuous variables, normality was assessed using the Shapiro-Wilk test and histogram inspection, which noted that large sample sizes may cause the test to reject normality due to high sensitivity. Normally distributed variables were compared using independent 2-tailed t tests, and nonnormally distributed variables were compared using the Mann-Whitney U test (2-tailed).

To evaluate predictive performance, 95% CIs for the AUROC were estimated using the bootstrap method with 1000 resamples. DeLong’s test was used for statistical comparison of AUROC values, while a bootstrap-based test with 1000 resamples was applied for AUPRC.

Sensitivity, specificity, positive predictive value, and negative predictive value were calculated to evaluate the performance of the scores, using predefined thresholds corresponding to moderate and high alarm levels. Proportions were compared using the 2-sample proportion test, applying either 1- or 2-tailed approaches as appropriate. One-tailed tests were used to compare AE rates when a directional effect was prespecified, whereas 2-tailed tests were applied when no prior direction was assumed.

All statistical analyses were performed using Python (version 3.12.9; Python Software Foundation), with statistical significance defined as P<.05, applied uniformly across 1- and 2-tailed tests. The analyses used the MLstatkit and scikit-learn libraries.

This study initially included 32,750 admissions from 24,187 patients. After applying the exclusion criteria, 30 admissions involving patients aged younger than 19 years of age were excluded, resulting in 32,720 admissions being included in the study dataset, which was used for performance evaluation. Considering the adaptation period, 1935 admissions were excluded from the study. The final dataset consisted of 5375 admissions during the silent period, 25,410 admissions during the alert period, and 5780 admissions during the seasonally matched alert period. There were 3576 ICU admissions, of which 2 were younger than 19 years of age and were excluded, leaving 3574 admissions for the VC-MORS performance evaluation. After excluding 214 admissions during the adaptation period, 3360 admissions remained, including 647 admissions during the silent period, 2713 admissions during the entire alert period, and 597 admissions during the seasonally matched alert period (Figure 2).

We compared baseline characteristics of patients admitted during the silent and alert periods, including vital signs, laboratory values, and the Charlson comorbidity index, which was calculated according to previously published methods [17,18]. While statistically significant differences were observed in pulse rate, respiratory rate, SpO₂, bilirubin, sodium, NEWS, MEWS, Charlson Comorbidity Index, and other baseline characteristics were similar (Table 1). These findings indicate that patient baseline characteristics were largely similar between the 2 periods.

The incidence of Code Blue was expected to decrease in 2023 compared to the 4-year average between 2019 and 2022. Since the implementation of the VC system in 2023, the hospital’s Code Blue incidence rate (per 1000) has decreased by 24.97%, from 10.57 to 7.93 (P=.004). As shown in Table 2, even year-by-year comparisons during the pre-VC period show that the incidence was lower after the introduction of VC.

Moreover, we compared the rates of UIT and in-hospital mortality across three intervals: the silent (pre-VC) period, the season-matched period, and the entire study period. The baseline UIT event rate during the silent period was 69.40 events per 1000 admissions, which significantly decreased to 60.73 (P=.03) in the season-matched period and 62.37 (P=.03) in the entire study period. For in-hospital mortality, the baseline rate was 19.72 deaths per 1000 admissions, decreasing to 16.78 (P=.12) during the season-matched period and further to 15.87 (P=.02) during the entire period (Table 3).

The pLOS rate in the GW decreased significantly following VC implementation. No significant difference in the pLOS rate was observed before and after the implementation of the VC system in the ICU (Table 4).

As shown in Tables 5 and 6, significantly more early interventions such as oxygenation, laboratory tests, and new code orders occurred following the activation of both moderate (VC-MAES=30) and high (VC-MAES=50) alarms. However, in the ICU, significant earlier interventions were observed only following a moderate VC-MORS alarm (VC-MORS=20), particularly with the ordering of laboratory tests, such as that for lactate.

VC-MAES and VC-MORS outperformed conventional EWS in both AUROC and AUPRC. VC-MAES achieved an AUROC of 0.865 (95% CI 0.857-0.873) and an AUPRC of 0.088 (95% CI 0.080‐0.098), which were significantly higher than those of NEWS (AUROC=0.804; AUPRC = 0.018) and MEWS (AUROC=0.772; AUPRC=0.020).

Similarly, VC-MORS showed strong predictive performance for 6-hour mortality, with an AUROC of 0.937 (95% CI 0.931‐0.944) and an AUPRC of 0.098 (95% CI 0.087-0.109), which outperformed SOFA (AUROC=0.754; AUPRC=0.011). APACHE II, calculated once within 24 hours of ICU admission, showed an AUROC of 0.814 and an AUPRC of 0.259 for predicting ICU mortality (Table 7).

We retrospectively collected the EMR data of hospitalized patients and compared the pre- and postintervention clinical outcomes to evaluate the effectiveness of VC. The results demonstrated that VC use was associated with high predictive performance and a significant reduction in Code Blue events. Specifically, VC-MAES exhibited a reduced pLOS and a lower number of AEs compared with the silent period, along with an increased frequency of early interventions.

Globally, the importance of RRS as a means of preventing AEs has been increasingly emphasized. However, rescue failures may occur even in environments where RRSs are implemented owing to challenges in the afferent limb [19]. The afferent limb refers to the “recognition” phase, encompassing processes such as monitoring, identification, and triggering a response [20]. VC supports automated monitoring, aids in the detection of early high-risk patients, and offers potential solutions for delaying and preventing Code Blue events [1]. In this regard, it can be used systematically and efficiently in environments where an RRS is already established. Conversely, it can maximize the potential impact by addressing unmet needs, even in hospitals where the RRS is not organized or is understaffed.

In practical settings, the initial assessment and recognition of a patient’s condition are typically performed by nursing staff. Subsequently, the attending physician is notified of the patient’s condition to facilitate decision-making regarding diagnosis and treatment. Vital signs and laboratory tests are also intermittently monitored, making the automatic interpretation and screening of time-series data a critical clinical requirement. As AEs often occur abruptly, such sequential processes can serve as barriers to timely intervention. VC can address this challenge by providing predictive alerts before the occurrence of AEs. This study confirmed a short-term decrease in UIT and death cases, which was found to have led to a significant long-term reduction in Code Blue events.

VC provides the opportunity to assess a patient’s condition (eg, laboratory tests, blood culture, and imaging) proactively or report it to the attending physician early, enabling the preparation of the necessary resources in advance. Accordingly, our findings suggest that VC implementation leads to notable clinical improvements, including increased early intervention and reduced AEs. Prior studies have indicated that several patients who ultimately experience CA remain in the GW for extended periods before being transferred to the ICU (or are never transferred at all), which negatively affects their survival rates. Consequently, these studies asserted that AI-based EWSs have the potential to reduce in-hospital mortality [15,21-23].

We conducted a subgroup analysis of the frequency of early interventions and observed a more significant increase in the GW following VC use (Table 5). It is well established that early intervention improves patient outcomes [24-26]. The occurrence of any intervention or evaluation within 2 hours of the initial risk prediction of VC can be interpreted as an indication that the clinician agreed with this information and suspected patient deterioration. VC can detect AEs more quickly than traditional scoring systems, thereby offering the benefit of securing opportunities for early recognition and treatment, and potentially preventing AEs in the GW, where closed observation may not be as rigorous. Conversely, as patients in the ICU were already under real-time close observation by medical staff, there was no change in the intervention rate within 2 hours of the first elevation. Similarly, a trend was observed in which the difference in intervention rates diminished as the patient severity increased. These results were predictable and reasonable. Based on these findings, we propose that the use of VC-MORS in the ICU is more beneficial for real-time patient monitoring, enabling the early prediction of critical events with high predictive power, rather than offering new information that could lead to additional prescriptions.

In conclusion, VC serves as a clinical decision-supporting system that provides supplementary information for health care providers rather than functioning as an independent diagnostic tool. Ultimately, the clinical judgment of health care professionals remains the final determinant in decision-making. The successful integration and adaptation of AI-EWS into clinical practice suggest the potential for clinical decision-supporting systems to be used more extensively. Various research will be needed to achieve this. We believe that the synergy between the enhancement of health care professionals’ ability to use predictive information and the advancement of AI-EWS could yield significant outcomes for patients and clinical settings.

This study demonstrates the effectiveness of AI-based EWS in a community hospital without an RRS; however, its applicability to institutions in resource-limited settings requires further consideration. The adoption and integration of health AI can vary significantly across countries, depending on geopolitical and socioeconomic development [27,28]. Consequently, several challenges may arise in rural hospitals or institutions with limited digital infrastructure [29]. These include the absence of real-time EMR systems, poor clinical data quality, limited clinician training in AI tools, and variability in clinical workflows and institutional acceptance. While simplified models may offer a feasible alternative, future research should explore implementation strategies for lightweight AI-EWS systems and scalable integration approaches in community-based and resource-limited settings.

This study has several limitations. First, the control period was relatively short owing to constraints in data extraction following the computerized system upgrade period at our institution. A comparative analysis was conducted within a given period using the same computerized system. Second, the effect of VC on patient prognosis following discharge was not investigated. Third, although this was a real-time EWS validation study, it did not evaluate interaction variables between AI-based EWS and the clinician, such as alarm fatigue. Given that real-time EWS operates as a continuous monitoring system, some degree of alert fatigue may perhaps be unavoidable. While adjusting thresholds—such as lowering sensitivity—may mitigate excessive alerts, it may also increase the risk of missing critical events. Future research will focus on assessing the effectiveness of clinician-AI interactions through the AI-based EWS and their influence on discharge outcomes and postdischarge prognosis.

A well-designed and validated AI-based EWS, such as VC, can effectively reduce major in-hospital AEs by enabling early detection and intervention. By supporting clinical decision-making without replacing professional judgment, this study demonstrates the potential of AI-EWS to enhance patient outcomes and optimize resource utilization in hospital settings.