This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
Patient falls are a common cause of harm in acute-care hospitals worldwide. They are a difficult, complex, and common problem requiring a great deal of nurses’ time, attention, and effort in practice. The recent rapid expansion of health care predictive analytic applications and the growing availability of electronic health record (EHR) data have resulted in the development of machine learning models that predict adverse events. However, the clinical impact of these models in terms of patient outcomes and clinicians’ responses is undetermined.
The purpose of this study was to determine the impact of an electronic analytic tool for predicting fall risk on patient outcomes and nurses’ responses.
A controlled interrupted time series (ITS) experiment was conducted in 12 medical-surgical nursing units at a public hospital between May 2017 and April 2019. In six of the units, the patients’ fall risk was assessed using the St. Thomas’ Risk Assessment Tool in Falling Elderly Inpatients (STRATIFY) system (control units), while in the other six, a predictive model for inpatient fall risks was implemented using routinely obtained data from the hospital’s EHR system (intervention units). The primary outcome was the rate of patient falls; secondary outcomes included the rate of falls with injury and analysis of process metrics (nursing interventions that are designed to mitigate the risk of fall).
During the study period, there were 42,476 admissions, of which 707 were for falls and 134 for fall injuries. Allowing for differences in the patients’ characteristics and baseline process metrics, the number of patients with falls differed between the control (n=382) and intervention (n=325) units. The mean fall rate increased from 1.95 to 2.11 in control units and decreased from 1.92 to 1.79 in intervention units. A separate ITS analysis revealed that the immediate reduction was 29.73% in the intervention group (
This early-stage clinical evaluation revealed that implementation of an analytic tool for predicting fall risk may to contribute to an awareness of fall risk, leading to positive changes in nurses’ interventions over time.
Clinical Research Information Service (CRIS), Republic of Korea KCT0005286; https://cris.nih.go.kr/cris/search/detailSearch.do/16984
Inpatient falls are preventable adverse events that are the top 10 sentinel events in hospitals. Up to 1 million fall events occur annually in the United States, and the average cost of each event has been estimated at $7900–$17,099 (2019 USD) [
Despite the availability of a considerable body of literature on fall prevention and reduction, falls remain a difficult, complex, and common problem that consume a great deal of time, attention, and mitigation efforts among nurses in practice [
The increased adoption of electronic health record (EHR) systems over the past decade has stimulated the development of predictive fall risk models using machine learning techniques, which are reported to exhibit better predictive performance than the existing fall risk assessment tools alone [
In this study, we used the prediction model that was developed in our previous study [
This research team previously reported on the development of a fall risk prediction model [
The model was tested in two study cohorts obtained from two hospitals with different EHR systems and nursing vocabularies. The model concepts were mapped to local data elements of each EHR system, and two implementation models were developed for a proof-of-concept approach, followed by cross-site validation. The EHR data included in the model were demographics, administrative information, medications, Korean patient classification based on nursing needs, the fall risk assessment tool, and nursing fall risk prevention processes, including assessments and interventions. The two implementation models exhibited error rates of 11.7% and 4.87%, with
The validation site model was implemented at a 900-bed public hospital in the metropolitan area of Seoul (Republic of Korea) that used STRATIFY to assess fall risks for all inpatients. The project, named Intelligent Nursing @ Safety Improvement Guide of Health information Technology (IN@SIGHT), was designed as a platform to support analytic tools as part of the infrastructure of a hospital EHR system, starting with a fall prediction analytic tool. The fall prediction analytic tool was integrated into the locally developed EHR system that had been in use for more than 10 years. The tool was deployed in 6 targeted nursing units (intervention group) on April 5, 2017, and all 204 nurses at those units automatically received the prediction results on a daily basis. This implementation process involved the chief of the Nursing Department, unit managers, unit champions, personnel of the Department of Medical Informatics, and the Patient Safety Committee. For 3 months before system deployment, three sessions of education on the IN@SIGHT system were provided to the intervention group, followed by peer-to-peer education provided by unit champions.
The Nursing Department decided to replace the existing STRATIFY with the analytic tool during this quasi-experimental study. The original model was customized by replacing the six data elements of STRATIFY with proxy data elements in the EHRs. The adjusted model, consisting of 40 nodes and 68 links, had an error rate of 9.3%, a spherical payoff of 0.92, and a
A study framework was developed based on a nursing role effectiveness model (
Conceptual framework of the study.
In accordance with the aim of this study, the impact of an electronic analytic tool for fall risk prediction on patient outcomes and nurses’ responses was explored by addressing the following specific research questions:
Did the predictive analytic tool influence the quality of nursing care as assessed using outcome indicators?
Did the predictive analytic tool affect nursing fall prevention activities provided to patients?
How did the effects change over time?
This nonrandomized controlled trial used an interrupted time series (ITS) design. To control for bias due to time-varying confounders, such as other quality improvement (QI) initiatives occurring in parallel with the intervention and other events, the 12 medical-surgical units were selected and allocated to 1 of 2 groups using pairs of units matched according to the known fall rates and unit characteristics for individual units (
This study was approved by the hospital’s ethical review board (IRB no. NHIMC 2016-08-005). A waiver of informed consent was granted by the IRB due to the QI nature of the intervention, thus enabling the inclusion of all patients and nurses in the participating units. This study followed the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) reporting guidelines [
Nonequivalent-group design of the study.
Nurses in the intervention units received 24-hour fall risk prediction results for each patient every morning. These results could be overridden based on the nurses’ clinical judgment, such as when patients were receiving treatments, procedures, operations, or fall related high-risk drugs or whether they suffered a fall, seizure, or syncope. The fall risk predictions were created by the analytic tool using the data collected within the past 24 hours. For missing data, a priori values from the day before were assigned first, and then a replacement was used: a mean value for continuous variables and a modal value for categorical variables. Nurses in the intervention units used the STRATIFY risk assessment tool only on the day of admission. When an at-risk patient was selected by nurses in the EHR system, they received an alert once each shift informing them that the patient was at risk and were guided to a care plan screen that listed pertinent interventions ordered by priority according to the patient’s risk factors. Nurses in the control units used only STRATIFY to assess fall risk according to their individual clinical judgment. They were able to manually open the same care plan window through menu navigation but received no alerts for at-risk patients.
The primary outcome was the overall rate of patient falls per 1000 patient-days during the study period, as defined by the National Database of Nursing Quality Indicator (NDNQI) outcome metrics of the American Nurses Association [
A patient fall is a sudden, unintentional descent, with or without injury to the patient, that results in the patient coming to rest on the floor, on or against some other surface (e.g., a counter), on another person, or on an object (e.g., a trash can). NDNQI counts only falls that occur on an eligible inpatient unit that reports falls. When a patient rolls off a low bed onto a mat or is found on a surface where you would not expect to find a patient, this is considered a fall. If a patient who is attempting to stand or sit falls back onto a bed, chair, or commode, this is only counted as a fall if the patient is injured. All unassisted and assisted falls... are to be reported, including falls attributable to physiological factors such as fainting (known as physiological falls).
The secondary outcomes were the overall rate of falls with injury, and process metrics. The rate of falls with injury was also measured using the aforementioned NDNQI definition. Process metrics were defined according to the Institute for Healthcare Improvement definition as “process indicators that measure compliance with key components of evidence-based prevention” [
Monthly rates of patient falls were collected from 16 months before the experiment started (the preintervention period) from the hospital’s quality assurance department to provide a baseline reference for comparisons. However, monthly rates of falls with injury before the experiment were not comparable due to differences in the criteria used to calculate them; only severe injuries were used as a sentinel event at the hospital. For process metrics, 1 month of data from before the experiment were collected as a baseline. During the study, data on patient demographics and medications, nursing activities, STRATIFY data, and administrative information were collected from the EHR system, and fall data were collected from the hospital’s quality assurance department. To monitor and minimize the underreporting rate noted previously [
The study hypothesis was that the fall rate would be reduced by 15% during the 24-month implementation of the prediction program. We conservatively estimated the required sample size based on previous research [
The participant characteristics were compared using chi-square tests for categorical variables and
This study involved 42,476 admissions of 40,345 unique patients in 12 units, corresponding to 362,805 patient-days in nursing units across both the control and intervention groups. In total, 2131 patients (5.02% of all admissions) were admitted to both an intervention and a control unit at different times. The patient characteristics differed significantly between the two groups (
Characteristics of patients in the intervention and control groups.
Variable | Intervention (n=24,336) | Control (n=18,140) |
|
||||||
|
|||||||||
|
Respiratory or digestive disease | 6150 (25.21) | 3472 (19.14) | <0.001 |
|
||||
|
Cancer | 5990 (24.61) | 2382 (13.13) | <0.001 |
|
||||
|
Symptom or injury | 2784 (11.44) | 2561 (14.12) | <0.001 |
|
||||
|
Cardiovascular disease | 995 (4.09) | 3096 (17.07) | <0.001 |
|
||||
|
Benign tumor | 860 (3.53) | 211 (1.16) | <0.001 |
|
||||
|
Infectious disease | 514 (2.11) | 388 (2.14) | <0.001 |
|
||||
|
Neurologic disease | 182 (0.75) | 597 (3.29) | <0.001 |
|
||||
|
Othera | 6861 (28.19) | 5433 (29.95) | <0.001 |
|
||||
|
|||||||||
|
Age (years), mean (95% CI) | 61.45 (61.23-61.67) | 65.30 (65.05-65.54) | <0.001 |
|
||||
|
Length of stay (days), mean (95% CI) | 7.96 (7.91-8.00) | 9.25 (9.13-9.37) | <0.001 |
|
||||
|
Sex (female), n (%) | 12,512 (51.41) | 9053 (49.91) | 0.002 |
|
||||
|
History of fall at admission, n (%) | 2873 (11.88) | 4138 (23.58) | <0.001 |
|
||||
|
Secondary diagnoses, n (%) | 10,641 (43.73) | 9361 (51.60) | <0.001 |
|
||||
|
History of surgical procedures, n (%) | 2483 (10.20) | 8575 (47.27) | <0.001 |
|
aIncluding genitourinary, musculoskeletal, eye, ear, and skin diseases.
There were 325 fall events in the intervention group and 382 in the control group. The mean monthly rate of falls decreased from 1.92 to 1.79 in the intervention group and increased from 1.95 to 2.11 in the control group
Due to the significant differences in patient characteristics between the control and intervention groups, we conducted separate before versus after comparisons between a period of time postintervention and the same period of time preintervention. In the intervention group, there was a significant reduction in the rate of falls of 29.73% (0.57 falls per 1000 patient-days) immediately postintervention (SE 0.14,
Results of interrupted time series analysis of rates of patient falls.
Group | Preintervention period trend | Change immediately after introduction of intervention | Postintervention period trend | |
Intervention | −0.07 (−0.22 to 0.08) | −0.30 (−0.58 to –0.14)a | 0.01 (<−0.01 to 0.02) | |
Control | 0.08 (−0.07 to 0.22) | −0.17 (−0.42 to 0.09) | 0.01 (<−0.01 to 0.02) |
|
a
Data are rate ratio (95% CI) values.
During the intervention period, the mean monthly injury rate per 1000 patient-days was 0.42 in the intervention group and 0.31 in the control group. The comparative time series analysis revealed a nonsignificant increase in the rate ratio of 0.18 (
Regarding process metrics, fall risk assessment was not conducted in almost three-quarters of patient-days in the control group, while in the intervention group, fall risk assessment was conducted on 100% of patient-days (
Temporal changes in process metrics in the control and intervention groups.
Item | Baseline (1 month) | First 6 months of intervention | Second 6 months of intervention | Third 6 months of intervention | Fourth 6 months of intervention | ||||||
|
|||||||||||
|
Patient-days | 8254 vs 4207a | 45,133 vs 31,675 | 46,403 vs 39,733 | 44,418 vs 44,741 | 42,553 vs 43,161 | |||||
|
Days on which no risk assessment performed, % | 72.5 vs 73.4b | 0 vs 72.6 | 0 vs 77.1 | 0 vs 71.7 | 0 vs 79.8 | |||||
|
At-risk days, % | 43.0 vs 42.1b | 24.5 vs 43.5c | 31.4 vs 38.6c | 32.7 vs 42.9c | 34.6 vs 41.5c | |||||
|
|||||||||||
|
Use of a fall risk tool | 99.3 vs 98.6 b | 100.0 vs 99.2c | 100.0 vs 70.8c | 100.0 vs 95.3c | 100.0 vs 98.8c | |||||
|
Injury risk factors (ABCsd) | 0 vs 0b | 100.0 vs 0 | 100.0 vs 0 | 100.0 vs 0 | 100.0 vs 0 | |||||
|
|||||||||||
|
Universal precautionse | 86.1 vs 100.0c | 69.7 vs 78.9c | 88.8 vs 99.9c | 37.8 vs 99.9c | 91.2 vs 99.9c | |||||
|
Education interventionse | 86.1 vs 100.0c | 69.7 vs 78.9c | 88.8 vs 99.9c | 33.1 vs 98.1c | 79.6 vs 97.8c | |||||
|
Risk-targeted interventions | <0.01 vs <0.01 | <0.01 vs <0.01 | <0.01 vs <0.01 | 12.5 vs 13.3b | 29.5 vs 18.1c | |||||
|
Communication interventionse | 61.7 vs 79.4c | 87.6 vs 99.9c | 76.0 vs 81.1c | 30.2 vs 38.7c | 66.2 vs 66.7b | |||||
|
Environmental interventionse | 61.7 vs 79.4c | 87.6 vs 99.9c | 76.0 vs 81.1c | 39.5 vs 54.9c | 76.7 vs 76.0b |
aAll data shown as intervention group versus control group.
bNot significant.
c
dABCs: age, bone health, anticoagulants, and current surgery (function that was performed automatically in the intervention group).
eData collection not categorized in detail from baseline to the second observation point.
For the care components of nursing assessments, nurses in the intervention group performed various observation types, such as mental status, cognitive function, communication ability, and incontinence, including mobility, at each observation point (
Changes in nursing assessments (A) and interventions (B) according to care components. ob.: observation point; †includes assessments of cognitive function, communication ability, gait status, incontinence, sleep pattern, and use of constraints; ‡includes interventions of toileting aids and for impaired mental and cognitive function, impaired sensory function, and sleep disturbance.
Implementation of an electronic analytic tool designed to predict fall risk was associated with reduced fall rates among inpatients at a public hospital in South Korea. However, comparison with the control group should be considered with caution due to notable differences in patient characteristics between the two groups. There was no significant difference in the rate of falls with injury between the control and intervention groups. Use of the electronic analytic tool was feasible, and it was accepted by nurses and improved the completion of risk assessments. Moreover, the process metrics for multifactorial and risk-targeted interventions for at-risk days were lower in the intervention group but increased over time. These findings suggest that although the effectiveness of an electronic analytic tool may be limited, it has potential as an aid to help nurses make informed clinical decisions.
The main challenges in this study were threefold: (1) random assignment of patients to the study groups was not possible; (2) it was not possible to control for co-interventions or external events at the hospital that may have affected the outcome, including QI activities; and (3) nurses’ understanding of the analytic tool developed by a machine learning approach was not assessed. These issues were managed by selecting only medical-surgical units and assigning patients according to the particular characteristics of each unit. A controlled ITS design was adopted to control for time-varying confounders. Finally, the development and validation process of the predictive model and the mechanism of chaining joint probabilities of a Bayesian network were introduced via user education sessions. However, during the study, the research team confronted additional issues that made interpretation of the results challenging. Discussion on these issues is valuable for future research into risk prediction and alerting in real-world settings.
The fall rates of 1.79 and 2.11 in the intervention and control groups, respectively, in this study were lower than previously reported rates of 2.08-4.18 for an intervention study involving a cluster randomized controlled trial (RCT) in four urban US hospitals [
According to international guidelines for preventing falls [
This study had limitations. The control group patients had more comorbidities that rendered them more vulnerable to falls than the intervention group. They were on average 4 years older, had a hospital stay that was 1.3 days longer, and had a greater history of falls. These variables are known important covariates [
The temporal changes in process metrics and nursing activities can provide important clues as to the overall impact of this trial. In a previous study [
The design of this study had several limitations that impacted the interpretation of its findings. First, due to the unexpected differences in baseline characteristics between the intervention and control groups, robust conclusions could not be drawn regarding comparison of the primary outcome between them. Future studies should implement matching techniques, such as propensity score matching [
Inpatient fall prevention is a difficult and complex issue, for which there is little high-quality evidence [
This was an early-stage clinical evaluation of a nursing predictive analytic application designed to forecast patient fall events in real time and at the point of care to improve outcomes and reduce costs. The effectiveness of the electronic analytic tool was supported only by the before-after comparison, not by the intervention-control comparison. Nurses were amenable to using the tool in practice, and over the course of the study, there were meaningful changes in process metrics, leading to more multifactorial and risk-targeted interventions to prevent patient falls.
electronic health record
Intelligent Nursing @ Safety Improvement Guide of Health information Technology
interrupted time series
non-adoption, abandonment, scale-up, spread, and sustainability
National Database of Nursing Quality Indicator
quality improvement
St. Thomas’ Risk Assessment Tool in Falling Elderly Inpatients
Transparent Reporting of Evaluations with Nonrandomized Designs
We thank Chihang Kim and Jisun Cho of the Nursing Department and Yunjeong Choi of the medical information department at Ilsan Hospital for helping us conduct this study administratively and technically. We also appreciate the clinical staff of the Nursing Department and graduate students involved in data collection, review, and analysis. This study was supported by grants from the Korea Healthcare Technology R&D Project, Ministry for Health and Welfare (no. HI17C0807), the National Research Foundation of Korea (no. NRF-2019R1A2C2007583), and the Ministry of Trade, Industry and Energy of Korea (no. 20004861).
None declared.