This study was conducted at an ED with an annual visit volume of 80,000 patients. The hospital is an academic institute with 2000 inpatient beds. The institution has utilized a home-grown electronic health record (EHR) system since 2003, which was replaced by a next-generation EHR system named Data Analytics and Research Window for Integrated Knowledge (DARWIN) in 2016. DARWIN is an all-in-one home-grown EHR that includes CPOE, nursing, pharmacy, billing, research support, and a patient portal (Figure 1) [22]. The institution’s ethics committee approved this study (Institutional Review Board File No. 2019-05-038).

When developing DARWIN’s CDSS, a minimally interruptive medication CDSS was introduced. This CDSS is mainly designed for physicians and utilizes only medication-specific information so that, for instance, there is no interference with laboratory data. This database is supplied from Medi-Span (Wolters Kluwer Health, Philadelphia, PA, USA) and is updated monthly (Figure 2).

The user interface was designed to minimize interruption in physician prescription workflow. First, the rules engine operates simultaneously with the physician’s entry of each order component such as a drug name, dose, and route. Second, its feedback appears as an in-line message so that physicians are not interrupted during order processing (Figure 3). The CDSS operates with the following areas of medication: age, allergy, disease, duplication, gender, lactation, pregnancy, route, drug-drug interaction, and dosage.

The inclusion criteria for this study were that patients had to have visited the ED between July 1, 2016 and December 31, 2017. Patients were excluded if they visited the ED but left without being examined by physicians or if they visited the ED without a medical purpose. The eligibility and selection process is presented in Figure 4. As we aimed to extensively investigate alert override patterns in an ED, the term “physician in ED” includes physicians from various medical departments, including the ED, pediatrics, internal medicine, and plastic surgery, among others.

Clinical data were extracted from the clinical data warehouse of the study site. We collected the following data: patient information (deidentified patient identifier, date of birth, gender, chief complaint, visit time, type of disposition, length of ED stay, severity level, and International Classification of Diseases-10 code), alert information (medication code based on the generic product identifier, alert firing time), order information (medication code, order time), and physician information (physician identifier, department, and career status). The severity score was measured by the Korean Triage and Acuity Scale (KTAS), which has been widely used by triage nurses in Korean EDs [23]. The KTAS was developed based on the Canadian Triage Scale to assess ED visiting patients’ acuity and severity. Patients with a score corresponding to level 1 have the highest acuity and severity, whereas level 5 indicates the lowest acuity and severity.

An override determining algorithm was developed for assessing the outcome measures. The algorithm was based on a rule-based, alert type–specific logic, newly generated for this study for validation (Table 1).

For the chart review, we selected a sample of 20 alerts among each type of alert that was performed for both overridden and nonoverridden orders. In the first round of the review process, two clinicians independently reviewed the sample alerts and then evaluated the interrater reliability using the Cohen κ statistic. In the second round, both clinicians worked together to resolve any case of disagreement. The two clinicians who reviewed the log data consisted of a doctor and nurse who have worked at the ED of the study site for over 4 years. Accuracy of the override determining algorithm was assessed using the reviewed data as the gold standard.

We conducted a descriptive analysis of patients, physicians, alert characteristics, and the alert firing and override rates. We used a logistic regression model for assessing the risk of the alert override using scalable medical variables such as physician factors (physicians’ specialties and designation), patient factors (severity scores and chief complaints), and alert factors (types of alerts and medication categories of alerts). The statistical significance level was set at P<.05. The variable with the smallest difference between the overall mean override rate and overrode rate of each variable (eg, resident) within each group (eg, physicians’ designation) was selected as the reference variable of the logistic regression. We employed R (version 3.6.0) software for the analysis.

In the first round of the review process conducted by two independent clinicians, Cohen κ was 0.82 (95% CI 0.74-0.90). All discrepancies were resolved in the second round of the review process. The accuracy of the override determining algorithm was 0.95.

During the study period, 611 physicians took care of 71,546 patients with 101,186 visits. General characteristics of the physicians and patients are described in Table 2 and Table 3, respectively. The physicians prescribed 748,339 medication orders and 102,887 (13.75%) alerts were fired.

Of the total 102,887 alerts, 65,616 (63.77%) alerts were overridden. We then analyzed the effects of physician-related factors, patient-related factors, and alert-related factors that could affect the physicians’ alert override behavior (Table 4).

In terms of physician-related factors, the resident group had a higher override rate than both the fellow and faculty groups. The top three physicians’ specialty departments that generated the most alerts in the ED were the emergency medicine, pediatrics, and general internal medicine departments. Physicians working in the ED were found to override over 64% of the total alerts received. In terms of patient factors, the alert override rate tended to decrease with the increase in severity. Additionally, the alert override rate also showed wide variation according to the patients’ chief complaints. The override rate tended to be higher in patients with trauma such as laceration and injuries than in patients with other chief complaints.

Two-thirds of the total alerts were dose alerts, and 65.3% of these were overridden. ED physicians overrode approximately half of the gender- and allergy-type alerts. Regarding the medication group, alerts for gastrointestinal agents, central nervous system drugs, respiratory agents, and endocrine and metabolic drugs showed the lowest override rates. Among the variables used in multivariate logistic regression models, the following variables emerged as statistically significant risk factors: miscellaneous department group in physician’s specialty; patients with lower and lowest severity; the presence of dizziness, head trauma, headache, hematochezia, melena, hematuria, facial injuries, lower extremity injuries, upper extremity injuries, laceration, minor complaints, and upper extremity pain; drug-drug interaction, lactation, and the route; and medication categories of analgesics and anesthetics, anti-infective agents, and topical products (Table 4).

In this study, we examined the alert override patterns in an ED with a CDSS that was designed in a minimally interruptive way. Approximately two-thirds of the alerts were overridden, which is a lower rate than reported in many previous related studies. We assessed many covariates, including both physician-related and patient-related factors, as well as alert-related factors. These results could be used for optimizing and maximizing the effectiveness of the CDSS.

Many studies examining alert override rates have not examined the physicians’ designations as an input variable or did not find it to have a significant effect [14,24]. Despite lack of evidence, it is generally considered that experienced physicians do not need alerts because “they already know” [25], and even if they receive them, they may be more likely to override them. In contrast to this expectation, we found that the alert override rate was the highest among residents, followed by fellows and senior faculty members (Table 4). This finding establishes that even an experienced physician still requires assistances from a CDSS.

We also demonstrated that physicians override more alerts in patients with complaints of lower severity. The override rate for patients with the highest severity level was 57.1% and that for patients with the lowest severity level was 63.5%. The override rate decreased significantly as the patients’ severity increased (Table 4). To date, relatively little attention has been paid to patients’ severity as a covariate affecting physicians’ alert overriding behavior. Further investigation is needed to confirm this finding.

First, this study was performed in a single ED with a homegrown EHR. The ED part of an academic referral center receives patients with conditions of high severity, and the majority of physicians are trainees of its residency program. Furthermore, the EHR, DARWIN, has only been implemented in a few hospitals in Korea to date, which should be considered while generalizing our results.

Second, the alert override was the only outcome measured. The outcome of the alert override or the reason for override was not recorded in the database. Thus, it was not possible to determine the clinical appropriateness of the alerts or overrides, or the consequences for patients. Therefore, there is a need for a follow-up interview study or an institution-level investigation of adverse drug events.

Third, the effect of the minimally interrupted CDSS was not demonstrated in this ED. For clarity, a comparative study is required. Such a comparison would require a control group with more interruptive alerts on similar targets. The nature of the CDSS makes it difficult to perform a randomized controlled trial.

Finally, we did not scale all potential factors related to alert overrides in previous studies, such as alert fatigue, alert severity, and workload. In particular, to estimate the size of the effects of alert fatigue, there is a need for further observational studies using devices such as eye trackers that can quantitatively measure whether the physician paid attention to the CPOE alert.

In this retrospective study, we described alert override patterns with a medication CDSS in an academic ED. We found a relatively low rate of overrides and also assessed the influence of multiple contributing factors on these rates. This study could aid CDSS implementers by providing knowledge regarding physicians’ alert overriding behaviors as well as empirical evidence that contradicts conventional notions.