Alert Override Patterns With a Medication Clinical Decision Support System in an Academic Emergency Department: Retrospective Descriptive Study

Introduction

An emergency department (ED) is a challenging environment in which multiple interventions are delivered within a short period [1]. The severity of the patients’ conditions demands that providers often order medications and tests simultaneously, which could contribute to a higher rate of medical errors [2-4]. Physicians working in an ED must often make decisions in the context of uncertainty due to the pace of the environment and resource limitations [5]. Specifically, the concept of physicians working in an ED is not limited to emergency medicine specialists, but rather covers various medical department physicians who treat patients in the geographical area of the ED.

Computerized provider order entry (CPOE) combined with a clinical decision support system (CDSS) was introduced to reduce preventable adverse drug events [6]. This system was expected to improve physicians’ prescribing patterns by supporting their decision-making process in a variety of ways. However, previous studies have revealed that physicians’ override rates on CDSS alerts are high [7-10], raising concerns about the effectiveness of CDSSs in many implementations [11-13].

Many factors, including physician and patient characteristics, environmental factors, and factors associated with the system itself, affect physicians’ alert override patterns in multifactorial ways with probable interactions among them [14-16]. Additionally, many previous studies regarding physicians’ alert override patterns have focused on specific diseases or alert types for well-defined targets as well as particular settings [10,17,18]. Thus, it is not clear how these results will generalize to patients at large or in settings such as the ED. Moreover, alert-related fatigue and physician burnout are very frequent among ED physicians, and also appear to be associated with worse performance of a CDSS [19-21].

Based on this background, the aim of this study was to describe and assess alert override patterns with a medication CDSS in a large academic ED.

Methods

Study Setting

This study was conducted at an ED with an annual visit volume of 80,000 patients. The hospital is an academic institute with 2000 inpatient beds. The institution has utilized a home-grown electronic health record (EHR) system since 2003, which was replaced by a next-generation EHR system named Data Analytics and Research Window for Integrated Knowledge (DARWIN) in 2016. DARWIN is an all-in-one home-grown EHR that includes CPOE, nursing, pharmacy, billing, research support, and a patient portal (Figure 1) [22]. The institution’s ethics committee approved this study (Institutional Review Board File No. 2019-05-038).

‎View this figure

Minimally Interruptive CDSS

When developing DARWIN’s CDSS, a minimally interruptive medication CDSS was introduced. This CDSS is mainly designed for physicians and utilizes only medication-specific information so that, for instance, there is no interference with laboratory data. This database is supplied from Medi-Span (Wolters Kluwer Health, Philadelphia, PA, USA) and is updated monthly (Figure 2).

The user interface was designed to minimize interruption in physician prescription workflow. First, the rules engine operates simultaneously with the physician’s entry of each order component such as a drug name, dose, and route. Second, its feedback appears as an in-line message so that physicians are not interrupted during order processing (Figure 3). The CDSS operates with the following areas of medication: age, allergy, disease, duplication, gender, lactation, pregnancy, route, drug-drug interaction, and dosage.

‎View this figure

‎View this figure

Study Subjects

The inclusion criteria for this study were that patients had to have visited the ED between July 1, 2016 and December 31, 2017. Patients were excluded if they visited the ED but left without being examined by physicians or if they visited the ED without a medical purpose. The eligibility and selection process is presented in Figure 4. As we aimed to extensively investigate alert override patterns in an ED, the term “physician in ED” includes physicians from various medical departments, including the ED, pediatrics, internal medicine, and plastic surgery, among others.

‎View this figure

Data Extraction and Preparation

Clinical data were extracted from the clinical data warehouse of the study site. We collected the following data: patient information (deidentified patient identifier, date of birth, gender, chief complaint, visit time, type of disposition, length of ED stay, severity level, and International Classification of Diseases-10 code), alert information (medication code based on the generic product identifier, alert firing time), order information (medication code, order time), and physician information (physician identifier, department, and career status). The severity score was measured by the Korean Triage and Acuity Scale (KTAS), which has been widely used by triage nurses in Korean EDs [23]. The KTAS was developed based on the Canadian Triage Scale to assess ED visiting patients’ acuity and severity. Patients with a score corresponding to level 1 have the highest acuity and severity, whereas level 5 indicates the lowest acuity and severity.

Override Determining Algorithm

An override determining algorithm was developed for assessing the outcome measures. The algorithm was based on a rule-based, alert type–specific logic, newly generated for this study for validation (Table 1).

For the chart review, we selected a sample of 20 alerts among each type of alert that was performed for both overridden and nonoverridden orders. In the first round of the review process, two clinicians independently reviewed the sample alerts and then evaluated the interrater reliability using the Cohen κ statistic. In the second round, both clinicians worked together to resolve any case of disagreement. The two clinicians who reviewed the log data consisted of a doctor and nurse who have worked at the ED of the study site for over 4 years. Accuracy of the override determining algorithm was assessed using the reviewed data as the gold standard.

Data Analysis

We conducted a descriptive analysis of patients, physicians, alert characteristics, and the alert firing and override rates. We used a logistic regression model for assessing the risk of the alert override using scalable medical variables such as physician factors (physicians’ specialties and designation), patient factors (severity scores and chief complaints), and alert factors (types of alerts and medication categories of alerts). The statistical significance level was set at P<.05. The variable with the smallest difference between the overall mean override rate and overrode rate of each variable (eg, resident) within each group (eg, physicians’ designation) was selected as the reference variable of the logistic regression. We employed R (version 3.6.0) software for the analysis.

Results

Interrater Reliability of the Override Determining Algorithm

In the first round of the review process conducted by two independent clinicians, Cohen κ was 0.82 (95% CI 0.74-0.90). All discrepancies were resolved in the second round of the review process. The accuracy of the override determining algorithm was 0.95.

Basic Characteristics

During the study period, 611 physicians took care of 71,546 patients with 101,186 visits. General characteristics of the physicians and patients are described in Table 2 and Table 3, respectively. The physicians prescribed 748,339 medication orders and 102,887 (13.75%) alerts were fired.

Override Patterns

Of the total 102,887 alerts, 65,616 (63.77%) alerts were overridden. We then analyzed the effects of physician-related factors, patient-related factors, and alert-related factors that could affect the physicians’ alert override behavior (Table 4).

In terms of physician-related factors, the resident group had a higher override rate than both the fellow and faculty groups. The top three physicians’ specialty departments that generated the most alerts in the ED were the emergency medicine, pediatrics, and general internal medicine departments. Physicians working in the ED were found to override over 64% of the total alerts received. In terms of patient factors, the alert override rate tended to decrease with the increase in severity. Additionally, the alert override rate also showed wide variation according to the patients’ chief complaints. The override rate tended to be higher in patients with trauma such as laceration and injuries than in patients with other chief complaints.

Two-thirds of the total alerts were dose alerts, and 65.3% of these were overridden. ED physicians overrode approximately half of the gender- and allergy-type alerts. Regarding the medication group, alerts for gastrointestinal agents, central nervous system drugs, respiratory agents, and endocrine and metabolic drugs showed the lowest override rates. Among the variables used in multivariate logistic regression models, the following variables emerged as statistically significant risk factors: miscellaneous department group in physician’s specialty; patients with lower and lowest severity; the presence of dizziness, head trauma, headache, hematochezia, melena, hematuria, facial injuries, lower extremity injuries, upper extremity injuries, laceration, minor complaints, and upper extremity pain; drug-drug interaction, lactation, and the route; and medication categories of analgesics and anesthetics, anti-infective agents, and topical products (Table 4).

Discussion

Principal Results

In this study, we examined the alert override patterns in an ED with a CDSS that was designed in a minimally interruptive way. Approximately two-thirds of the alerts were overridden, which is a lower rate than reported in many previous related studies. We assessed many covariates, including both physician-related and patient-related factors, as well as alert-related factors. These results could be used for optimizing and maximizing the effectiveness of the CDSS.

Physicians’ Specialty, Patients’ Severity, and Alert Override Rate

Many studies examining alert override rates have not examined the physicians’ designations as an input variable or did not find it to have a significant effect [14,24]. Despite lack of evidence, it is generally considered that experienced physicians do not need alerts because “they already know” [25], and even if they receive them, they may be more likely to override them. In contrast to this expectation, we found that the alert override rate was the highest among residents, followed by fellows and senior faculty members (Table 4). This finding establishes that even an experienced physician still requires assistances from a CDSS.

We also demonstrated that physicians override more alerts in patients with complaints of lower severity. The override rate for patients with the highest severity level was 57.1% and that for patients with the lowest severity level was 63.5%. The override rate decreased significantly as the patients’ severity increased (Table 4). To date, relatively little attention has been paid to patients’ severity as a covariate affecting physicians’ alert overriding behavior. Further investigation is needed to confirm this finding.

Comparison With Prior Work

Low Override Rate

One of the major findings in this study is the relatively low override rate. The low override rate was observed consistently across the three factors (physician, patient, and alert), implying that a systematic influence exists aside from those discussed above. One potential explanation could be related to the phenomenon that has been termed the “cloud of context.” Coiera et al [26] proposed that “variations in the workflow, patient population and morbidity, resources, pre-existing infrastructure, and the education and experience of both clinical staff and patients” function as a “cloud of context” that affect how physicians respond to a CDSS. Given that most of the existing CDSS studies were conducted in inpatient and outpatient settings in the United States, these contextual factors may have influenced the lower override rate observed in this study. Previous studies have reported that override rates increased from 72.8% to 93% [7-10,24]. In comparison, a Korean study reported a rate of 71.7% [14], which is relatively lower than that reported for the United States. However, there are no other ED-based studies that we are aware of for direct comparison; thus, the conclusion warrants further investigation.

Another possible explanation is the system design. It is well known that utilizing a human factors design reduces the error rate and that an interactive design can reduce alert fatigue [15,27]. We leveraged several strategies to ensure that the system is integrated into the clinical workflow by designing a noninterruptive system. For example, the concept of timely alerts was implemented more seamlessly by designing a system that can generate an alert whenever a new order component is entered. Although our research scope did not include direct measurements and analysis of usability, we believe that integrating the system into the clinical workflow contributes to the override rate for ED physicians. We also believe that the underlying knowledge used was relatively robust.

Importance of the ED Environment in Alert Override–Related Research

In this study, we assessed the alert overriding patterns of ED physicians in routine care in the ED. An ED is an optimal environment to examine physicians’ alert overriding behaviors from a broad perspective because patients have a wider range of severity than those in other departments, and many receive interdisciplinary care in this environment. This viewpoint is important because the CDSS does not influence physicians concerning a single type of alert but rather acts as a bundle of user interfaces.

To the best of our knowledge, this is the first study to analyze physicians’ CDSS usage patterns in an ED comprehensively. In a recent review, less than one-tenth of the studies were found to target physicians’ behavior in the ED [15]. Some research studies only targeted specific alert types such as drug-drug interaction or opioid alerts, or only a specific group such as pediatric patients [10,17,18]. In this study, the analysis was performed on the whole system rather than focusing on particular types of alerts, physicians, or patients. The effect of a CDSS may be critical in this complex environment, and the results of this study may play an essential role in establishing CDSSs in EDs or in cases in which a multidisciplinary approach is needed.

Limitations

First, this study was performed in a single ED with a homegrown EHR. The ED part of an academic referral center receives patients with conditions of high severity, and the majority of physicians are trainees of its residency program. Furthermore, the EHR, DARWIN, has only been implemented in a few hospitals in Korea to date, which should be considered while generalizing our results.

Second, the alert override was the only outcome measured. The outcome of the alert override or the reason for override was not recorded in the database. Thus, it was not possible to determine the clinical appropriateness of the alerts or overrides, or the consequences for patients. Therefore, there is a need for a follow-up interview study or an institution-level investigation of adverse drug events.

Third, the effect of the minimally interrupted CDSS was not demonstrated in this ED. For clarity, a comparative study is required. Such a comparison would require a control group with more interruptive alerts on similar targets. The nature of the CDSS makes it difficult to perform a randomized controlled trial.

Finally, we did not scale all potential factors related to alert overrides in previous studies, such as alert fatigue, alert severity, and workload. In particular, to estimate the size of the effects of alert fatigue, there is a need for further observational studies using devices such as eye trackers that can quantitatively measure whether the physician paid attention to the CPOE alert.

Conclusions

In this retrospective study, we described alert override patterns with a medication CDSS in an academic ED. We found a relatively low rate of overrides and also assessed the influence of multiple contributing factors on these rates. This study could aid CDSS implementers by providing knowledge regarding physicians’ alert overriding behaviors as well as empirical evidence that contradicts conventional notions.