Published on in Vol 8, No 11 (2020): November

Preprints (earlier versions) of this paper are available at, first published .
Alert Override Patterns With a Medication Clinical Decision Support System in an Academic Emergency Department: Retrospective Descriptive Study

Alert Override Patterns With a Medication Clinical Decision Support System in an Academic Emergency Department: Retrospective Descriptive Study

Alert Override Patterns With a Medication Clinical Decision Support System in an Academic Emergency Department: Retrospective Descriptive Study

Original Paper

1Institution of Healthcare Resource, School of Nursing, Sahmyook University, Seoul, Republic of Korea

2Samsung Advanced Institute for Health Sciences & Technology (SAIHST), Department of Digital Health, Sungkyunkwan University, Seoul, Republic of Korea

3Department of Gastroenterology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea

4Health Information and Strategy Center, Samsung Medical Center, Seoul, Republic of Korea

5Center for Health Promotion, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea

6Division of General Internal Meidicine and Primary Care, Brigham and Women's Hospital, Boston, MA, United States

7Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea

Corresponding Author:

Won Chul Cha, MD

Department of Emergency Medicine

Samsung Medical Center

Sungkyunkwan University School of Medicine

81 Irwon-ro


Seoul, 06351

Republic of Korea

Phone: 82 10 5386 6597


Background: Physicians’ alert overriding behavior is considered to be the most important factor leading to failure of computerized provider order entry (CPOE) combined with a clinical decision support system (CDSS) in achieving its potential adverse drug events prevention effect. Previous studies on this subject have focused on specific diseases or alert types for well-defined targets and particular settings. The emergency department is an optimal environment to examine physicians’ alert overriding behaviors from a broad perspective because patients have a wider range of severity, and many receive interdisciplinary care in this environment. However, less than one-tenth of related studies have targeted this physician behavior in an emergency department setting.

Objective: The aim of this study was to describe alert override patterns with a commercial medication CDSS in an academic emergency department.

Methods: This study was conducted at a tertiary urban academic hospital in the emergency department with an annual census of 80,000 visits. We analyzed data on the patients who visited the emergency department for 18 months and the medical staff who treated them, including the prescription and CPOE alert log. We also performed descriptive analysis and logistic regression for assessing the risk factors for alert overrides.

Results: During the study period, 611 physicians cared for 71,546 patients with 101,186 visits. The emergency department physicians encountered 13.75 alerts during every 100 orders entered. Of the total 102,887 alerts, almost two-thirds (65,616, 63.77%) were overridden. Univariate and multivariate logistic regression analyses identified 21 statistically significant risk factors for emergency department physicians’ alert override behavior.

Conclusions: In this retrospective study, we described the alert override patterns with a medication CDSS in an academic emergency department. We found relatively low overrides and assessed their contributing factors, including physicians’ designation and specialty, patients’ severity and chief complaints, and alert and medication type.

JMIR Med Inform 2020;8(11):e23351



An emergency department (ED) is a challenging environment in which multiple interventions are delivered within a short period [1]. The severity of the patients’ conditions demands that providers often order medications and tests simultaneously, which could contribute to a higher rate of medical errors [2-4]. Physicians working in an ED must often make decisions in the context of uncertainty due to the pace of the environment and resource limitations [5]. Specifically, the concept of physicians working in an ED is not limited to emergency medicine specialists, but rather covers various medical department physicians who treat patients in the geographical area of the ED.

Computerized provider order entry (CPOE) combined with a clinical decision support system (CDSS) was introduced to reduce preventable adverse drug events [6]. This system was expected to improve physicians’ prescribing patterns by supporting their decision-making process in a variety of ways. However, previous studies have revealed that physicians’ override rates on CDSS alerts are high [7-10], raising concerns about the effectiveness of CDSSs in many implementations [11-13].

Many factors, including physician and patient characteristics, environmental factors, and factors associated with the system itself, affect physicians’ alert override patterns in multifactorial ways with probable interactions among them [14-16]. Additionally, many previous studies regarding physicians’ alert override patterns have focused on specific diseases or alert types for well-defined targets as well as particular settings [10,17,18]. Thus, it is not clear how these results will generalize to patients at large or in settings such as the ED. Moreover, alert-related fatigue and physician burnout are very frequent among ED physicians, and also appear to be associated with worse performance of a CDSS [19-21].

Based on this background, the aim of this study was to describe and assess alert override patterns with a medication CDSS in a large academic ED.

Study Setting

This study was conducted at an ED with an annual visit volume of 80,000 patients. The hospital is an academic institute with 2000 inpatient beds. The institution has utilized a home-grown electronic health record (EHR) system since 2003, which was replaced by a next-generation EHR system named Data Analytics and Research Window for Integrated Knowledge (DARWIN) in 2016. DARWIN is an all-in-one home-grown EHR that includes CPOE, nursing, pharmacy, billing, research support, and a patient portal (Figure 1) [22]. The institution’s ethics committee approved this study (Institutional Review Board File No. 2019-05-038).

Figure 1. Overall schematic description of the hospital information system architecture at the Samsung Medical Center. DARWIN: Data Analytics and Research Window for Integrated Knowledge; CPOE: computerized physician order entry; MIS: management information system; MDM: master data management; CRM: customer relationship management. Reproduced with permission from Jung et al [22].
View this figure

Minimally Interruptive CDSS

When developing DARWIN’s CDSS, a minimally interruptive medication CDSS was introduced. This CDSS is mainly designed for physicians and utilizes only medication-specific information so that, for instance, there is no interference with laboratory data. This database is supplied from Medi-Span (Wolters Kluwer Health, Philadelphia, PA, USA) and is updated monthly (Figure 2).

The user interface was designed to minimize interruption in physician prescription workflow. First, the rules engine operates simultaneously with the physician’s entry of each order component such as a drug name, dose, and route. Second, its feedback appears as an in-line message so that physicians are not interrupted during order processing (Figure 3). The CDSS operates with the following areas of medication: age, allergy, disease, duplication, gender, lactation, pregnancy, route, drug-drug interaction, and dosage.

Figure 2. System architecture of the computerized provider order entry (CPOE). DB: database.
View this figure
Figure 3. Screenshot of the computerized provider order entry system and features of its interface (zoomed out).
View this figure

Study Subjects

The inclusion criteria for this study were that patients had to have visited the ED between July 1, 2016 and December 31, 2017. Patients were excluded if they visited the ED but left without being examined by physicians or if they visited the ED without a medical purpose. The eligibility and selection process is presented in Figure 4. As we aimed to extensively investigate alert override patterns in an ED, the term “physician in ED” includes physicians from various medical departments, including the ED, pediatrics, internal medicine, and plastic surgery, among others.

Figure 4. Flow diagram of the eligibility and selection process for study inclusion. ED: emergency department; ID: identification.
View this figure

Data Extraction and Preparation

Clinical data were extracted from the clinical data warehouse of the study site. We collected the following data: patient information (deidentified patient identifier, date of birth, gender, chief complaint, visit time, type of disposition, length of ED stay, severity level, and International Classification of Diseases-10 code), alert information (medication code based on the generic product identifier, alert firing time), order information (medication code, order time), and physician information (physician identifier, department, and career status). The severity score was measured by the Korean Triage and Acuity Scale (KTAS), which has been widely used by triage nurses in Korean EDs [23]. The KTAS was developed based on the Canadian Triage Scale to assess ED visiting patients’ acuity and severity. Patients with a score corresponding to level 1 have the highest acuity and severity, whereas level 5 indicates the lowest acuity and severity.

Override Determining Algorithm

An override determining algorithm was developed for assessing the outcome measures. The algorithm was based on a rule-based, alert type–specific logic, newly generated for this study for validation (Table 1).

Table 1. Description of the alert overrides determining logic.
Type of alertOverride determining logic
Age, allergy, disease, duplication, gender, lactation, pregnancy, routeIf a physician completed the order without alert adjustment
DoseIf a physician did not adjust the dose-related order components such as prescription day or daily dosage
Drug-drug interactionIf a physician ordered both medications indicated in a drug-drug interaction alert

For the chart review, we selected a sample of 20 alerts among each type of alert that was performed for both overridden and nonoverridden orders. In the first round of the review process, two clinicians independently reviewed the sample alerts and then evaluated the interrater reliability using the Cohen κ statistic. In the second round, both clinicians worked together to resolve any case of disagreement. The two clinicians who reviewed the log data consisted of a doctor and nurse who have worked at the ED of the study site for over 4 years. Accuracy of the override determining algorithm was assessed using the reviewed data as the gold standard.

Data Analysis

We conducted a descriptive analysis of patients, physicians, alert characteristics, and the alert firing and override rates. We used a logistic regression model for assessing the risk of the alert override using scalable medical variables such as physician factors (physicians’ specialties and designation), patient factors (severity scores and chief complaints), and alert factors (types of alerts and medication categories of alerts). The statistical significance level was set at P<.05. The variable with the smallest difference between the overall mean override rate and overrode rate of each variable (eg, resident) within each group (eg, physicians’ designation) was selected as the reference variable of the logistic regression. We employed R (version 3.6.0) software for the analysis.

Interrater Reliability of the Override Determining Algorithm

In the first round of the review process conducted by two independent clinicians, Cohen κ was 0.82 (95% CI 0.74-0.90). All discrepancies were resolved in the second round of the review process. The accuracy of the override determining algorithm was 0.95.

Basic Characteristics

During the study period, 611 physicians took care of 71,546 patients with 101,186 visits. General characteristics of the physicians and patients are described in Table 2 and Table 3, respectively. The physicians prescribed 748,339 medication orders and 102,887 (13.75%) alerts were fired.

Table 2. Physician characteristics (N=611).
Characteristicn (%)

Resident357 (58.3)

Fellow154 (25.2)

Faculty100 (16.4)

Emergency Medicine41 (6.7)

General Internal Medicine60 (9.8)

Gastroenterology39 (6.4)

Cardiology17 (2.8)

Pulmonary Medicine15 (2.5)

Nephrology13 (2.1)

Hematology & Oncology10 (1.6)

Endocrinology & Metabolism8 (1.3)

Infectious Disease7 (1.2)

Allergic Medicine2 (0.3)

Rheumatology2 (0.3)

General Surgery58 (9.5)

Gynecology & Obstetrics39 (6.4)

Thoracic surgery26 (4.3)

Orthopedic Surgery19 (3.1)

Neurosurgery15 (2.5)

Urology15 (2.5)

Plastic Surgery11 (1.8)

Pediatrics53 (8.7)

Family Medicine24 (3.9)

Ophthalmology24 (3.9)

Otolaryngology24 (3.9)

Neurology20 (3.3)

Radiology16 (2.6)

Psychiatry14 (2.3)

Anesthesiology & Pain Medicine10 (1.6)

Dermatology9 (1.5)

Critical Care Medicine7 (1.2)

Rehabilitation7 (1.2)

Dentistry5 (0.8)

Radiation Oncology1 (0.2)
Table 3. Patient characteristics (N=101,186 visits).
Age (years), mean (SD)44.50 (25.68)
Male, n (%)51,221 (50.62)
Severity score, n (%)

1 (Highest severity)1122 (1.11)

26331 (6.25)

338,456 (38.01)

446,696 (46.15)

5 (Lowest severity)8581 (8.48)
Chief complaint, n (%)

Fever15,080 (14.90)

Abdominal Pain14,285 (14.12)

Dyspnea6920 (6.84)

Minor Complaint6536 (6.46)

Dizziness4920 (4.86)

Headache3643 (3.60)

Laceration2366 (2.34)

Skin Rash2323 (2.30)

Head Trauma2259 (2.23)

Pain (Lower Extremity)2011 (1.99)

Chest Pain (Suspected Cardiogenic Pain)1859 (1.84)

Injury (Upper Extremity)1820 (1.80)

Injury (Lower Extremity)1734 (1.71)

Pain (Upper Extremity)1639 (1.62)

Limb Weakness1488 (1.47)

Inter-Hospital Transfer1434 (1.42)

Altered Mentality1393 (1.38)

Back Pain1317 (1.3)

Coughing and Stuffy Nose1197 (1.18)

Palpitation and Irregular Heart Rate1196 (1.18)

Hematochezia/Melena1169 (1.16)

Seizure1154 (1.14)

Nausea/Vomiting1106 (1.09)

General Weakness1037 (1.02)

Injury (Facial)994 (0.98)

Chest Pain (Noncardiogenic)860 (0.85)

Hematuria854 (0.84)

Other18,592 (18.37)

Override Patterns

Of the total 102,887 alerts, 65,616 (63.77%) alerts were overridden. We then analyzed the effects of physician-related factors, patient-related factors, and alert-related factors that could affect the physicians’ alert override behavior (Table 4).

Table 4. The risk of alert overrides according to various factors.
FactorFrequency of alert (n)Alert override rate, n (%)Univariate logistic regression odds ratio (95% CI)Multivariate logistic regression odds ratio (95% CI)
Physician-related factors

Physicians’ Designation

Resident93,02259,678 (64.15)1 [Reference]1 [Reference]

Fellow81744993 (61.08)0.88 (0.84-0.92)0.9 (0.86-0.94)

Faculty1691945 (55.88)0.71 (0.64-0.78)0.73 (0.66-0.81)

Physicians’ Specialty

Emergency Department50,81232,542 (64.04)1 [Reference]1 [Reference]

Internal Medicine17,47610,623 (60.79)0.87 (0.84-0.9)1.03 (0.99-1.07)

Surgical Department87375501 (62.96)0.95 (0.91-1)0.90 (0.85-0.94)

Other Department25,86216,950 (65.54)1.07 (1.03-1.1)1.10 (1.06-1.14)
Patient-related factors

Patients’ severity score

1 (Highest Severity)1597912 (57.11)0.80 (0.73-0.89)0.82 (0.74-0.91)

289855421 (60.33)0.92 (0.88-0.96)0.89 (0.85-0.94)

345,75928,536 (62.36)1 [Reference]1 [Reference]

441,17127,059 (65.72)1.16 (1.13-1.19)1.07 (1.04-1.10)

5 (Lowest Severity)53753688 (68.61)1.32 (1.24-1.40)1.23 (1.15-1.32)

Patients’ chief complaints

Fever26,33416,769 (63.68)1 [Reference]1 [Reference]

Abdominal Pain11,3006525 (57.74)0.78 (0.75-0.82)0.95 (0.91-1.00)

Altered Mentality32202080 (64.6)1.04 (0.96-1.12)1.49 (1.37-1.62)

Back Pain16841079 (64.07)1.02 (0.92-1.13)1.04 (0.94-1.16)

Chest Pain (Non-Cardiogenic)680388 (57.06)0.76 (0.65-0.88)0.84 (0.72-0.98)

Chest Pain (Suspected Cardiogenic Pain)19461216 (62.49)0.95 (0.86-1.05)1.22 (1.09-1.37)

Coughing and Stuffy Nose1195769 (64.35)1.03 (0.91-1.16)1.04 (0.92-1.18)

Dizziness31282098 (67.07)1.16 (1.07-1.26)1.65 (1.52-1.79)

Dyspnea84324894 (58.04)0.79 (0.75-0.83)0.93 (0.88-0.98)

General Weakness1419909 (64.06)1.02 (0.91-1.14)1.28 (1.15-1.44)

Head Trauma1327985 (74.23)1.64 (1.45-1.86)1.64 (1.45-1.87)

Headache41652723 (65.38)1.08 (1.01-1.15)1.08 (1.01-1.16)

Hematochezia/Melena32362209 (68.26)1.23 (1.13-1.33)2 (1.84-2.18)

Hematuria561379 (67.56)1.19 (1.00-1.42)1.25 (1.05-1.50)

Injury (Facial)556446 (80.22)2.31 (1.88-2.87)2.18 (1.76-2.70)

Injury (Lower Extremity)1335905 (67.79)1.2 (1.07-1.35)1.29 (1.14-1.46)

Injury (Upper Extremity)1103817 (74.07)1.63 (1.42-1.87)1.56 (1.36-1.80)

Inter-hospital Transfer1354855 (63.15)0.98 (0.87-1.10)1.10 (0.97-1.24)

Laceration18341585 (86.42)3.63 (3.18-4.17)3.43 (2.98-3.95)

Limb Weakness1173689 (58.74)0.81 (0.72-0.91)1.01 (0.89-1.14)

Minor Complaint50823408 (67.06)1.16 (1.09-1.24)1.18 (1.10-1.27)

Nausea/Vomiting921455 (49.4)0.56 (0.49-0.64)0.7 (0.61-0.80)

Pain (Lower Extremity)18001111 (61.72)0.92 (0.83-1.02)0.97 (0.88-1.08)

Pain (Upper Extremity)1043731 (70.09)1.34 (1.17-1.53)1.3 (1.13-1.49)

Palpitation and Irregular Heart Rate693404 (58.3)0.8 (0.68-0.93)0.98 (0.84-1.15)

Seizure19931179 (59.16)0.83 (0.75-0.91)0.93 (0.84-1.02)

Skin Rash17761141 (64.25)1.02 (0.93-1.13)1.18 (1.06-1.32)

Others13,5978867 (65.21)1.07 (1.02-1.12)1.29 (1.23-1.35)
Alert-related factors

Type of alert

Duplication1414911 (64.43)1 [Reference]1 [Reference]

Age17,94911,035 (61.48)0.88 (0.79-0.99)0.8 (0.71-0.90)

Allergy1583822 (51.93)0.6 (0.51-0.69)0.54 (0.46-0.62)

Disease40412479 (61.35)0.88 (0.77-0.99)0.93 (0.82-1.06)

Dose67,21243,873 (65.28)1.04 (0.93-1.16)0.99 (0.88-1.11)

Drug-Drug Interaction1399926 (66.19)1.08 (0.93-1.26)1.07 (0.91-1.25)

Gender381183 (48.03)0.51 (0.41-0.64)0.43 (0.33-0.56)

Lactation1813 (72.22)1.44 (0.54-4.50)1.33 (0.50-4.18)

Pregnancy86515199 (60.10)0.83 (0.74-0.93)0.72 (0.64-0.81)

Route239175 (73.22)1.51 (1.12-2.06)1.19 (0.88-1.64)

Medication categories (based on generic product ID)

Neuromuscular Drugs1511961 (63.60)1 [Reference]1 [Reference]

Analgesics and Anesthetics36,68524,808 (67.62)1.20 (1.07-1.33)1.23 (1.10-1.38)

Anti-Infective Agents18,30812,419 (67.83)1.21 (1.08-1.35)1.08 (0.96-1.21)

Antineoplastic326198 (60.74)0.89 (0.69-1.13)0.94 (0.73-1.22)

Biologicals192138 (71.88)1.46 (1.06-2.05)1.09 (0.78-1.53)

Cardiovascular Agents58663637 (62)0.93 (0.83-1.05)0.95 (0.84-1.07)

Central Nervous System Drugs54343066 (56.42)0.74 (0.66-0.83)0.67 (0.59-0.76)

Endocrine and Metabolic Drugs33742009 (59.54)0.84 (0.74-0.95)0.84 (0.74-0.96)

Gastrointestinal Agents15,5898619 (55.29)0.71 (0.63-0.79)0.6 (0.53-0.67)

Genitourinary Agents343211 (61.52)0.91 (0.72-1.17)1.42 (1.07-1.90)

Hematological Agents49383210 (65.01)1.06 (0.94-1.20)1.04 (0.91-1.18)

Miscellaneous Products1168711 (60.87)0.89 (0.76-1.04)0.92 (0.78-1.09)

Nutritional Product18431214 (65.87)1.1 (0.96-1.27)1.09 (0.94-1.26)

Respiratory Agents67783994 (58.93)0.82 (0.73-0.92)0.85 (0.75-0.96)

Topical Products532421 (79.14)2.17 (1.72-2.75)1.62 (1.27-2.07)

In terms of physician-related factors, the resident group had a higher override rate than both the fellow and faculty groups. The top three physicians’ specialty departments that generated the most alerts in the ED were the emergency medicine, pediatrics, and general internal medicine departments. Physicians working in the ED were found to override over 64% of the total alerts received. In terms of patient factors, the alert override rate tended to decrease with the increase in severity. Additionally, the alert override rate also showed wide variation according to the patients’ chief complaints. The override rate tended to be higher in patients with trauma such as laceration and injuries than in patients with other chief complaints.

Two-thirds of the total alerts were dose alerts, and 65.3% of these were overridden. ED physicians overrode approximately half of the gender- and allergy-type alerts. Regarding the medication group, alerts for gastrointestinal agents, central nervous system drugs, respiratory agents, and endocrine and metabolic drugs showed the lowest override rates. Among the variables used in multivariate logistic regression models, the following variables emerged as statistically significant risk factors: miscellaneous department group in physician’s specialty; patients with lower and lowest severity; the presence of dizziness, head trauma, headache, hematochezia, melena, hematuria, facial injuries, lower extremity injuries, upper extremity injuries, laceration, minor complaints, and upper extremity pain; drug-drug interaction, lactation, and the route; and medication categories of analgesics and anesthetics, anti-infective agents, and topical products (Table 4).

Principal Results

In this study, we examined the alert override patterns in an ED with a CDSS that was designed in a minimally interruptive way. Approximately two-thirds of the alerts were overridden, which is a lower rate than reported in many previous related studies. We assessed many covariates, including both physician-related and patient-related factors, as well as alert-related factors. These results could be used for optimizing and maximizing the effectiveness of the CDSS.

Physicians’ Specialty, Patients’ Severity, and Alert Override Rate

Many studies examining alert override rates have not examined the physicians’ designations as an input variable or did not find it to have a significant effect [14,24]. Despite lack of evidence, it is generally considered that experienced physicians do not need alerts because “they already know” [25], and even if they receive them, they may be more likely to override them. In contrast to this expectation, we found that the alert override rate was the highest among residents, followed by fellows and senior faculty members (Table 4). This finding establishes that even an experienced physician still requires assistances from a CDSS.

We also demonstrated that physicians override more alerts in patients with complaints of lower severity. The override rate for patients with the highest severity level was 57.1% and that for patients with the lowest severity level was 63.5%. The override rate decreased significantly as the patients’ severity increased (Table 4). To date, relatively little attention has been paid to patients’ severity as a covariate affecting physicians’ alert overriding behavior. Further investigation is needed to confirm this finding.

Comparison With Prior Work

Low Override Rate

One of the major findings in this study is the relatively low override rate. The low override rate was observed consistently across the three factors (physician, patient, and alert), implying that a systematic influence exists aside from those discussed above. One potential explanation could be related to the phenomenon that has been termed the “cloud of context.” Coiera et al [26] proposed that “variations in the workflow, patient population and morbidity, resources, pre-existing infrastructure, and the education and experience of both clinical staff and patients” function as a “cloud of context” that affect how physicians respond to a CDSS. Given that most of the existing CDSS studies were conducted in inpatient and outpatient settings in the United States, these contextual factors may have influenced the lower override rate observed in this study. Previous studies have reported that override rates increased from 72.8% to 93% [7-10,24]. In comparison, a Korean study reported a rate of 71.7% [14], which is relatively lower than that reported for the United States. However, there are no other ED-based studies that we are aware of for direct comparison; thus, the conclusion warrants further investigation.

Another possible explanation is the system design. It is well known that utilizing a human factors design reduces the error rate and that an interactive design can reduce alert fatigue [15,27]. We leveraged several strategies to ensure that the system is integrated into the clinical workflow by designing a noninterruptive system. For example, the concept of timely alerts was implemented more seamlessly by designing a system that can generate an alert whenever a new order component is entered. Although our research scope did not include direct measurements and analysis of usability, we believe that integrating the system into the clinical workflow contributes to the override rate for ED physicians. We also believe that the underlying knowledge used was relatively robust.

Importance of the ED Environment in Alert Override–Related Research

In this study, we assessed the alert overriding patterns of ED physicians in routine care in the ED. An ED is an optimal environment to examine physicians’ alert overriding behaviors from a broad perspective because patients have a wider range of severity than those in other departments, and many receive interdisciplinary care in this environment. This viewpoint is important because the CDSS does not influence physicians concerning a single type of alert but rather acts as a bundle of user interfaces.

To the best of our knowledge, this is the first study to analyze physicians’ CDSS usage patterns in an ED comprehensively. In a recent review, less than one-tenth of the studies were found to target physicians’ behavior in the ED [15]. Some research studies only targeted specific alert types such as drug-drug interaction or opioid alerts, or only a specific group such as pediatric patients [10,17,18]. In this study, the analysis was performed on the whole system rather than focusing on particular types of alerts, physicians, or patients. The effect of a CDSS may be critical in this complex environment, and the results of this study may play an essential role in establishing CDSSs in EDs or in cases in which a multidisciplinary approach is needed.


First, this study was performed in a single ED with a homegrown EHR. The ED part of an academic referral center receives patients with conditions of high severity, and the majority of physicians are trainees of its residency program. Furthermore, the EHR, DARWIN, has only been implemented in a few hospitals in Korea to date, which should be considered while generalizing our results.

Second, the alert override was the only outcome measured. The outcome of the alert override or the reason for override was not recorded in the database. Thus, it was not possible to determine the clinical appropriateness of the alerts or overrides, or the consequences for patients. Therefore, there is a need for a follow-up interview study or an institution-level investigation of adverse drug events.

Third, the effect of the minimally interrupted CDSS was not demonstrated in this ED. For clarity, a comparative study is required. Such a comparison would require a control group with more interruptive alerts on similar targets. The nature of the CDSS makes it difficult to perform a randomized controlled trial.

Finally, we did not scale all potential factors related to alert overrides in previous studies, such as alert fatigue, alert severity, and workload. In particular, to estimate the size of the effects of alert fatigue, there is a need for further observational studies using devices such as eye trackers that can quantitatively measure whether the physician paid attention to the CPOE alert.


In this retrospective study, we described alert override patterns with a medication CDSS in an academic ED. We found a relatively low rate of overrides and also assessed the influence of multiple contributing factors on these rates. This study could aid CDSS implementers by providing knowledge regarding physicians’ alert overriding behaviors as well as empirical evidence that contradicts conventional notions.


This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI19C0275). We would like to thank Editage ( for English language editing.

Authors' Contributions

JY collected, coded, and analyzed the data, and wrote the manuscript. JL reviewed sample alerts and inspected the manuscript. PR developed the CDSS and reviewed the manuscript. DC, MK, and JC implemented the CDSS and reviewed the manuscript. DB analyzed and inspected the manuscript. WC developed and implemented the CDSS, designed the study, oversaw data analysis, and cowrote the manuscript.

Conflicts of Interest

None declared.

  1. Bernstein S, Aronsky D, Duseja R, Epstein S, Handel D, Hwang U, Society for Academic Emergency Medicine‚ Emergency Department Crowding Task Force. The effect of emergency department crowding on clinically oriented outcomes. Acad Emerg Med 2009 Jan;16(1):1-10. [CrossRef] [Medline]
  2. Hafner JW, Belknap SM, Squillante MD, Bucheit KA. Adverse drug events in emergency department patients. Ann Emerg Med 2002 Mar;39(3):258-267. [CrossRef] [Medline]
  3. Hohl CM, Badke K, Zhao A, Wickham ME, Woo SA, Sivilotti MLA, et al. Prospective Validation of Clinical Criteria to Identify Emergency Department Patients at High Risk for Adverse Drug Events. Acad Emerg Med 2018 Sep;25(9):1015-1026. [CrossRef] [Medline]
  4. Karpov A, Parcero C, Mok CP, Panditha C, Yu E, Dempster L, et al. Performance of trigger tools in identifying adverse drug events in emergency department patients: a validation study. Br J Clin Pharmacol 2016 Oct;82(4):1048-1057. [CrossRef] [Medline]
  5. Gerrity MS, DeVellis RF, Light DW. Uncertainty and Professional Work: Perceptions of Physicians in Clinical Practice. Am J Sociol 1992 Jan;97(4):1022-1051. [CrossRef]
  6. Shojania KG, Duncan BW, McDonald KM, Wachter RM, Markowitz AJ. Making health care safer: a critical analysis of patient safety practices. Evid Rep Technol Assess (Summ) 2001(43):i-x, 1-668. [Medline]
  7. Wong A, Rehr C, Seger DL, Amato MG, Beeler PE, Slight SP, et al. Evaluation of Harm Associated with High Dose-Range Clinical Decision Support Overrides in the Intensive Care Unit. Drug Saf 2019 Apr;42(4):573-579. [CrossRef] [Medline]
  8. Slight SP, Beeler PE, Seger DL, Amato MG, Her QL, Swerdloff M, et al. A cross-sectional observational study of high override rates of drug allergy alerts in inpatient and outpatient settings, and opportunities for improvement. BMJ Qual Saf 2017 Mar;26(3):217-225 [FREE Full text] [CrossRef] [Medline]
  9. Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018 May 01;25(5):476-481. [CrossRef] [Medline]
  10. Isaac T, Weissman JS, Davis RB, Massagli M, Cyrulik A, Sands DZ, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009 Feb 09;169(3):305-311. [CrossRef] [Medline]
  11. Shekelle P, Wachter R, Pronovost P, Schoelles K, McDonald K, Dy S, et al. Making health care safer II: an updated critical analysis of the evidence for patient safety practices. Evid Rep Technol Assess (Full Rep) 2013 Mar(211):1-945. [Medline]
  12. Prgomet M, Li L, Niazkhani Z, Georgiou A, Westbrook JI. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis. J Am Med Inform Assoc 2017 Mar 01;24(2):413-422. [CrossRef] [Medline]
  13. Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014 Sep;23(9):773-780. [CrossRef] [Medline]
  14. Cho I, Lee Y, Lee J, Bates DW. Wide variation and patterns of physicians' responses to drug-drug interaction alerts. Int J Qual Health Care 2019 Mar 01;31(2):89-95. [CrossRef] [Medline]
  15. Hussain MI, Reynolds TL, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc 2019 Oct 01;26(10):1141-1149 [FREE Full text] [CrossRef] [Medline]
  16. Tolley C, Forde N, Coffey K, Sittig D, Ash J, Husband A, et al. Factors contributing to medication errors made when using computerized order entry in pediatrics: a systematic review. J Am Med Inform Assoc 2018 May 01;25(5):575-584. [CrossRef] [Medline]
  17. Genco EK, Forster JE, Flaten H, Goss F, Heard KJ, Hoppe J, et al. Clinically Inconsequential Alerts: The Characteristics of Opioid Drug Alerts and Their Utility in Preventing Adverse Drug Events in the Emergency Department. Ann Emerg Med 2016 Feb;67(2):240-248.e3 [FREE Full text] [CrossRef] [Medline]
  18. Sethuraman U, Kannikeswaran N, Murray KP, Zidan MA, Chamberlain JM. Prescription errors before and after introduction of electronic medication alert system in a pediatric emergency department. Acad Emerg Med 2015 Jun;22(6):714-719. [CrossRef] [Medline]
  19. Shanafelt TD, Boone S, Tan L, Dyrbye LN, Sotile W, Satele D, et al. Burnout and satisfaction with work-life balance among US physicians relative to the general US population. Arch Intern Med 2012 Oct 08;172(18):1377-1385. [CrossRef] [Medline]
  20. Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J, et al. Relationship Between Clerical Burden and Characteristics of the Electronic Environment With Physician Burnout and Professional Satisfaction. Mayo Clin Proc 2016 Jul;91(7):836-848. [CrossRef] [Medline]
  21. Arora M, Asha S, Chinnappa J, Diwan AD. Review article: burnout in emergency medicine physicians. Emerg Med Australas 2013 Dec;25(6):491-495. [CrossRef] [Medline]
  22. Jung KY, Kim T, Jung J, Lee J, Choi JS, Mira K, et al. The Effectiveness of Near-Field Communication Integrated with a Mobile Electronic Medical Record System: Emergency Department Simulation Study. JMIR Mhealth Uhealth 2018 Sep 21;6(9):e11187 [FREE Full text] [CrossRef] [Medline]
  23. Choi H, Ok JS, An SY. Evaluation of Validity of the Korean Triage and Acuity Scale. J Korean Acad Nurs 2019 Feb;49(1):26-35. [CrossRef] [Medline]
  24. Cho I, Slight SP, Nanji KC, Seger DL, Maniam N, Fiskio JM, et al. The effect of provider characteristics on the responses to medication-related decision support alerts. Int J Med Inform 2015 Sep;84(9):630-639. [CrossRef] [Medline]
  25. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-147 [FREE Full text] [CrossRef] [Medline]
  26. Coiera E, Ammenwerth E, Georgiou A, Magrabi F. Does health informatics have a replication crisis? J Am Med Inform Assoc 2018 Aug 01;25(8):963-968 [FREE Full text] [CrossRef] [Medline]
  27. Brown CL, Mulcaster HL, Triffitt KL, Sittig DF, Ash JS, Reygate K, et al. A systematic review of the types and causes of prescribing errors generated from using computerized provider order entry systems in primary and secondary care. J Am Med Inform Assoc 2017 Mar 01;24(2):432-440. [CrossRef] [Medline]

CDSS: computerized decision support system
CPOE: computerized provider order entry
DARWIN: Data Analytics and Research Window for Integrated Knowledge
ED: emergency department
EHR: electronic health record
KTAS: Korean Triage and Acuity Scale

Edited by G Eysenbach; submitted 09.08.20; peer-reviewed by A Azzam, P Beeler, S Sarbadhikari; comments to author 31.08.20; revised version received 12.09.20; accepted 21.10.20; published 04.11.20


©Junsang Yoo, Jeonghoon Lee, Poong-Lyul Rhee, Dong Kyung Chang, Mira Kang, Jong Soo Choi, David W Bates, Won Chul Cha. Originally published in JMIR Medical Informatics (, 04.11.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.