Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 20.07.20 in Vol 8, No 7 (2020): July

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/15653, first published Jul 26, 2019.

This paper is in the following e-collection/theme issue:

    Review

    Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review

    1Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan

    2International Center for Health Information Technology (ICHIT), Taipei Medical University, Taipei, Taiwan

    3Research Center of Big Data and Meta-Analysis, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan

    4Department of Dermatology, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan

    5TMU Research Center of Cancer Translational Medicine, Taipei Medical University, Taipei, Taiwan

    Corresponding Author:

    Yu-Chuan (Jack) Li, MD, PhD

    Graduate Institute of Biomedical Informatics

    College of Medical Science and Technology

    Taipei Medical University

    No. 250 Wuxing Street

    Taipei, 110

    Taiwan

    Phone: 886 2 27361661 ext 7600

    Email: jaak88@gmail.com


    ABSTRACT

    Background: The clinical decision support system (CDSS) has become an indispensable tool for reducing medication errors and adverse drug events. However, numerous studies have reported that CDSS alerts are often overridden. The increase in override rates has raised questions about the appropriateness of CDSS application along with concerns about patient safety and quality of care.

    Objective: The aim of this study was to conduct a systematic review to examine the override rate, the reasons for the alert override at the time of prescribing, and evaluate the appropriateness of overrides.

    Methods: We searched electronic databases, including Google Scholar, PubMed, Embase, Scopus, and Web of Science, without language restrictions between January 1, 2000 and March 31, 2019. Two authors independently extracted data and crosschecked the extraction to avoid errors. The quality of the included studies was examined following Cochrane guidelines.

    Results: We included 23 articles in our systematic review. The range of average override alerts was 46.2%-96.2%. An average of 29.4%-100% of the overrides alerts were classified as appropriate, and the rate of appropriateness varied according to the alert type (drug-allergy interaction 63.4%-100%, drug-drug interaction 0%-95%, dose 43.9%-88.8%, geriatric 14.3%-57%, renal 27%-87.5%). The interrater reliability for the assessment of override alerts appropriateness was excellent (kappa=0.79-0.97). The most common reasons given for the override were “will monitor” and “patients have tolerated before.”

    Conclusions: The findings of our study show that alert override rates are high, and certain categories of overrides such as drug-drug interaction, renal, and geriatric were classified as inappropriate. Nevertheless, large proportions of drug duplication, drug-allergy, and formulary alerts were appropriate, suggesting that these groups of alerts can be primary targets to revise and update the system for reducing alert fatigue. Future efforts should also focus on optimizing alert types, providing clear information, and explaining the rationale of the alert so that essential alerts are not inappropriately overridden.

    JMIR Med Inform 2020;8(7):e15653

    doi:10.2196/15653

    KEYWORDS



    Introduction

    Rationale

    A computerized provider order entry (CPOE) system is often integrated with a clinical decision support system (CDSS) to reduce patient harm and error rates [1]. A CDSS has immense potential for fostering patient safety and quality of care by reducing the adverse drug effects (ADEs) rate [1-3]. However, the current CDSS generates too many alerts, which are often overridden (approximately 90% to 95%), sometimes inappropriately. Concern related to inappropriate overrides reached a peak [4,5] with recognition of the potential to increase the risk of harm to patients. Multiple studies have reported that a high frequency of clinically irrelevant alerts (repetitive alerts with minimal clinical value), mediocre functionality (minimal integration among various departments and lack of alerts prioritization), and erroneous assessment by physicians are the main reasons for inappropriate overrides [4,6,7]. However, the growing number of inappropriate overrides often silently puts patients at risk of fatal ADEs [8,9].

    To date, significant efforts have been taken to make sound clinical decisions and provide high-quality services. Indeed, lower specificity (high false-positives) and ambiguous alert contents (no clear information provided on why alerts were triggered in the systems) are still associated with excessive overrides and alert fatigue [4,10]. A CDSS with higher sensitivity and lower specificity could also contribute to the substantial number of inappropriate alerts [11,12]. Recent findings suggest that applying hard-stop alerts might be an efficient and helpful tool to reduce inappropriate overrides; however, such a tool must be judiciously implemented to achieve improved usability and receptivity of systems [13]. To increase the alert acceptance rate and reduce overrides, a system should be implemented in such a way that enables prioritizing alerts based on grade and potential harm, analyze the physician response, provide clear recommendations, and explain why the alert is triggered [14].

    Goal of Investigation

    Since the override rate has been increasing, it is necessary to ascertain the types of alerts that are most frequently triggered, calculate the rate at which they are overridden (ie, reject the alerts), and to determine the reasons for overrides and the appropriateness of the reasons. Gaining a better understanding of these issues can provide meaningful insight into how alerts can be delivered in a relevant way (ie, converting a hard alert to a soft alert or turning off clinically irrelevant alerts or those with low clinical value).


    Methods

    Overview

    We conducted a systematic review in accordance with the Meta-analysis of Observational Studies in Epidemiology guidelines [15] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) standard [16]. The overview of the study process is given in Figure 1.

    Figure 1. Overview of the study process.
    View this figure

    Electronic Databases Search

    We conducted a systematic search in electronic databases, including the PubMed, Embase, Scopus, Google Scholar, and Web of Science databases, between January 1, 2000 and April 30, 2019. The search was performed by two authors (MI and TP) using the keywords “alert fatigue,” “override alerts,” “computerized physician order entry,” “decision support system,” “medication-related CDS,” “CDSS,” and “CPOE.” There was no language and data restriction applied in the initial search. We also scanned the references of review articles and conference proceedings.

    Eligibility Criteria

    The titles and abstracts of all retrieved studies were screened independently by two expert authors (MI and TP) to find the most relevant articles. They selected potentially eligible full-text articles. The full-text articles were considered as appropriate for inclusion in the systematic review by these same two experts after screening the full text and documenting the reasons for exclusion of inappropriate/ineligible articles. Any disagreement that arose in this screening process was resolved by the principal investigator of the study (YL). Articles were considered for inclusion if they met the following criteria: (1) published in English with desired outcomes reported, (2) evaluated override alerts along with reasons for those overrides, and (3) reported the override rate and the appropriateness of the override reasons.

    We excluded studies if they were published in the form of a review, report, short communication, letter to editor, methodology, or editorial.

    Data Extraction

    For studies that fulfilled the inclusion criteria, two authors (MI and TP) conducted data abstraction using a predefined, standardized protocol. Review Manager software (RevMan-5, Cochrane, UK) was also used to check the accuracy of the included studies. The following information was collected from the included studies: (1) methods, including setting, data analyzed, study design, study period, type of alerts, appropriateness criteria, inclusion and exclusion criteria; (2) results, including number of alerts, percent of override alerts, percentage of different types of alert overrides, percentage of overall override alerts, reasons for those overrides, characteristics of alert types, rate of appropriateness, rate of appropriateness for each override alert subtype, and rate of adverse effects; and (3) discussion, including the main findings, suggestions, intended recommendations, and limitations.

    Outcome Parameters

    The following three primary outcomes were considered in our analysis: (1) characterize the types of alerts and their override rate; (2) the reasons for an override for different types of alerts assessed for inpatient and outpatient settings; and (3) the rate of the appropriateness of the override reasons.


    Results

    Literature Selection

    Our systematic search identified 360 titles and abstracts of potentially eligible studies for inclusion. Of these, 240 articles were excluded due to duplication and 88 of the remaining 120 articles were excluded based on predefined eligibility criteria during screening of titles and abstracts. The remaining 32 articles were processed for full-text review. Among these, a total of 23 relevant studies met all inclusion criteria [4,5,7,8,11,17-34]. Figure 2 shows all inclusion and exclusion criteria based on the PRISMA guidelines.

    Figure 2. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for study selection.
    View this figure

    Study Characteristics

    The study characteristics of the included 23 articles are presented in Table 1. In this systematic review, six studies were based on a retrospective observational study design, five studies were cross-sectional, five studies were prospective observational studies, and seven studies only mentioned an observational study design. The settings included the intensive care unit (n=4), primary teaching hospital (n=5), academic medical center (n=5), tertiary care teaching hospital (n=3), university pediatric hospital (n=2), and others (n=4). There were 12 different types of alerts (drug-allergy, drug-drug interaction, drug-class, class-class, drug-dose, drug-duplication, drug-laboratory, drug-disease, drug-pregnancy, geriatric, age-based suggestion, renal, and formulary substitution) discussed in the included studies. The maximum override rate was 96.2%. ADEs were evaluated in 5 of the 23 studies [7,20,21,32,33].

    Table 1. Characteristics of included studies.
    View this table

    Appropriateness Criteria

    All of the included studies developed criteria for evaluating the appropriateness of overrides for each alert type for both inpatient and outpatient settings. To validate the appropriateness framework, they used a chart along with previously published articles and clinical experience of the multidisciplinary group (physicians, pharmacists, and nurses). All studies used specific criteria for different types of alerts, which were modified until reaching a final agreement. They considered override alerts as appropriate if the reasons reported by the physicians were acceptable according to their study’s framework and also verified based on review of relevant guidelines. For example, if a clinician prescribed a drug and a dose alert was displayed, the appropriate override reasons mentioned were “will monitor as recommended,” “will adjust the dose,” and “patient has already tolerated” based on previous data, indicating that monitoring is beneficial to patients (Figure 3). Subsequently, the multidisciplinary group carefully checked and verified all of the override reasons in their chart review. They extensively verified by reviewing guidelines, such as checking for dose/renal function/drug monitoring criteria, previously prescribed tolerable medication combinations, and accepted/refused medication. The included studies mentioned that pharmacists, nurses, training health care personal, and clinicians checked and verified the appropriateness of override reasons. Any disagreements among them were resolved with discussion.

    Figure 3. Example of a drug-drug interaction alert (left) and drug-allergy interaction alerts (right).
    View this figure

    Override Rate and Appropriateness of Overrides

    All 23 studies described the alert override rate and the appropriateness of overrides according to alert type (drug-allergy, drug-drug, dose, drug-class, class-class, drug duplication, drug-laboratory, drug-disease, drug-pregnancy, geriatric, renal-dose, age-based suggestion, renal, formulary substitution). The average override alerts ranged from 46.2% to 96.2% (Table 1). However, the range of override rates varied according to alert type (drug-allergy 46%-95%, drug-drug interaction 56.3%-95.6%, dose 82%-96.8%, geriatric 2.1%-87.1%, and renal 74.4%-97.1%). Moreover, the overall appropriateness rate ranged from 29.4% to 100%, which also varied according to alert types (drug-allergy 63.4%-100%, drug-drug interaction 0%-95%, dose 43.9%-88.8%, geriatric 14.3%-57%, renal 27%-87.5%). However, interrater reliability for the assessment of override alerts appropriateness was excellent (kappa=0.79-0.97). Table 2 summarizes the rates of override alerts and the appropriateness of override alerts by type.

    Table 2. Rate of override alerts and appropriateness of override alerts by type.
    View this table

    Reasons for Overrides

    All 23 included studies evaluated the reasons for overriding the alerts in the CDSS. The most common reasons for overriding drug-allergy alerts were “will monitor,” “patients tolerated before,” “patient took previously without an allergic reaction,” “low risk across sensitivity,” “no reasonable alternatives,” and other (with or without a free-text reason provided). The most common reasons for overriding drug-drug interaction alerts were “will monitor as recommended,” “will adjust the dose as recommended,” “patients have already tolerated this combination,” “clinically irrelevant,” and “benefit assessed to be greater than the risk.” Moreover, the common reasons for overriding dose alerts were “will adjust the dose as recommended,” “benefit outweighs risk,” and “patients tolerated before” (Table 3).

    Table 3. Override reason by alert type.
    View this table

    ADEs

    Five studies compared ADEs based on an appropriate and inappropriate override. Wong et al [17] demonstrated a significantly increased rate of ADEs in inappropriate override dose alerts compared with appropriate override dose alerts. The rate of ADEs per 100 override dose alerts was 1.3 and 5.0 for appropriate and inappropriate override dose alerts, respectively. Wong et al [20] also evaluated the potential and definite ADEs in 5 different types of alerts reported, demonstrating that the average potential and definite ADEs were higher in alerts that were considered to be inappropriately overridden than appropriate override alerts (16.5 vs 2.74 per 100 overridden alerts, P<.001). However, the rate of ADEs was always higher for inappropriate override alerts (drug-allergy interaction: 11.5 vs 0.6; drug-drug interaction: 11.4 vs 2.0; dose: 17.6 vs 11.1; geriatric: 11.1 vs 0; and renal: 30.8 vs 0). The logistic regression model showed that inappropriate override alerts were significantly associated with an increased risk of ADEs (odds ratio 6.14, 95% CI 4.63-7.71, P<.001) and an increased intensive care unit length of stay (2.25 days, 95% CI 0.52-3.98, P=.01). Moreover, 3 studies also reported that inappropriate override was associated with an increased risk of ADEs [21,32,33].


    Discussion

    Main Findings

    This is the first systematic review that evaluated the current scenario of a CDSS by measuring the rate with which alerts are overridden, described the reasons for override alerts at the time of prescribing, and the appropriateness of overrides. A significant proportion of alerts in the CDSS were overridden (96.2%), and the override rate varied dramatically according to alert types. The rate of appropriate overrides was high (nearly 100%) and they also varied significantly according to alert types. For example, renal, geriatric, and drug-drug interaction alert overrides had low appropriateness rates, whereas drug-allergy, drug-duplication, drug-formulary, and drug class alert overrides had higher appropriateness rates. Inappropriate overrides were associated with an increased risk of ADEs when compared with appropriately overridden alerts. However, the reasons provided for overriding alerts varied extensively depending on alert types. Refinement of these alert types has immense potential to improve the acceptance rate and patient safety. Furthermore, the clinical team should evaluate the appropriateness of overrides based on the given clinical context to optimize alert types and frequencies and ultimately improve their clinical relevancy while reducing alert fatigue.

    Clinical Implications

    A CPOE integrated with a CDSS is designed to improve patient safety and reduce preventable errors by generating pop-up alerts at the point of order entry. A frequent complaint about CPOEs is firing up too many alerts, which are frequently not clinically relevant or have very low clinical value [35]. An excessive number of alerts in the CPOE desensitizes physicians (hampers the mental state, consumes too much time), leading them to override both appropriate and clinically irrelevant alerts [36]. A system with low sensitivity (ie, more false-negatives) and low specificity (ie, more false-positives), ambiguous information content, and an overwhelming number of alerts (both relevant and irrelevant alerts) induce alert fatigue [35]. Inappropriate override always leads to potential ADEs and increased morbidity [37]. A study evaluating drug-drug interaction alert overrides and how override alerts lead to preventable ADEs reported 22 serious ADEs over the 3-month study period [32].

    In our study, a high number of alerts were overridden, especially for dose, drug-drug interaction, and drug-allergy interaction alerts. The findings of our study also suggest that a higher number of these alerts can lead to alert fatigue. There are two ways to combat alert fatigue. First, the system should set a higher threshold for triggering alerts. Second, the most frequent alerts should be categorized and the system updated regularly (overrides tend to increase over time). We also evaluated the appropriateness of alert overrides, demonstrating that the rate of the appropriateness of overrides varied according to the different types of alerts. Evaluating the appropriateness of override alerts is difficult but the range of interrater reliability for assessment was high. Among the dose recommendation alerts that were overridden, only 43.9%-85% were found to be appropriately overridden. The range of appropriately overridden renal and drug-drug interaction alerts was 12%-87.5% and 0%-84%, respectively. Among the overridden drug-allergy interaction recommendation alerts, approximately 83.5%-100% were appropriately overridden. Moreover, the vast majority of drug-duplicate (82%-99%), drug-class (88%), and formulary (82.8%) override alerts were appropriate, indicating that these groups can be the primary targets for rectification to stop alert fatigue by reducing or converting (hard-stop alert to soft/passive alerts) the number of alerts. However, the higher rates of inappropriate overrides of the renal, drug-drug interaction, and geriatric alert types indicate the need for further intervention.

    Our findings also provide a variety of reasons for overriding alerts. The majority of physicians provided the reasons for overrides as “will monitor as recommended,” “patients have tolerated it before,” “will adjust the dose,” and “maximum time,” leaving the free-text box blank. In some cases, physicians do not write any reason for the overrides; however, it is important to clearly outline the override reasons to best invest in the patient’s condition and care. Indeed, these findings raise concern about patient safety and quality of care. For example, failure to monitor several drug levels such as digoxin after initiation of verapamil (drug-drug interaction) can cause serious harm for the patient [38]. Other common reasons physicians gave for the override were “no reasonable alternative,” “physician aware,” “patients have already tolerated this combination,” and other (without free-text reason provided). However, there was no confirmation that physicians were actually aware of the potential harm and had monitored the patient’s condition before overriding. Moreover, in the case of drug-allergy, approximately two-thirds of alerts that showed a reaction of “anaphylaxis” were overridden by physicians and with the reasons provided including patients have taken previously without an allergic reaction” and “low-risk cross-sensitivity.” However, a patient with a true allergy can experience severe anaphylaxis. For example, a reaction between vancomycin and red man syndrome was found to be inappropriately overridden with the reason given that the “patient has taken previously without allergic reaction/patient has tolerated previously,” but severe ADE was observed (development of a patchy macular rash) [21]. Therefore, it is essential to know the patient’s history of anaphylaxis to reduce serious recurrence (approximately 35% of patients experience recurrence) [39]. Topaz et al [23] reported that only about one-tenth of the alerts showed potential life-threatening effects that were a definite match between the allergy and prescribed drugs, although others were due to either the “cross-sensitivity or allergy group.” Several studies confirmed that the hospitalization of patients with anaphylaxis has been increasing in both the United States [40] and the United Kingdom [41]. It is therefore important to evaluate these types of override alerts and the reasons given by the physician for the override. Moreover, future studies are needed to develop an effective knowledge management system that can provide more accurate and relevant drug-allergy interaction alerts for improving patient safety.

    Recommendations to Improve the CDSS

    The findings of our study provide a clear picture of the overall situation of current CDSSs by summarizing the existing literature. These findings can help policymakers and researchers to improve existing CDSSs by conducting an in-depth analysis of existing CDSS features. Having provided a collection of evidence-based information and removing unimportant alerts, a novel system also requires rigorous evaluation to determine the optimum rate of sensitivity and specificity for reducing patient harm. No system can achieve 100% sensitivity and specificity in a real-world setting. However, a logical and effective symmetry between sensitivity and specificity can make the system more flexible and safer. The sensitivity and specificity should be increased without sacrificing the other through the combination of patient factors and using futuristic algorithms. Osheroff et al [42] demonstrated that “five rights” (right information, to the right person, in the right CDS intervention format, through the right channel, and at the right time in the workflow) should be taken into consideration when alerts will be popped up in the system. Several recommendations are provided below to design a sophisticated CDSS by reducing alert fatigue.

    First, increase the positive predictive value for dose recommendation alerts by incorporating patient-specific factors (eg age, other medication orders, renal impairment history) [7].

    Second, optimize alert types and frequencies to increase their clinical relevance so that important alerts are not inappropriately overridden [43].

    Third, override alerts can be revised if they are not clinically important, and the system will be updated for reducing alert fatigue.

    Fourth, turn off alerts that are not clinically important/inaccurate or of only minor importance [8,44].

    Fifth, it is essential to categorize the most frequent interruptive alerts; for serious alerts such as drug-drug interactions and renal, the dose should be displayed as interruptive, whereas minor/low-risk alerts can be presented in a noninterruptive manner [13].

    Sixth, all types of alerts should contain clear and concise information [45,46] and provide exact information on why the alert is important for the situation [47].

    Seventh, identify a list of medications that patients previously showed no allergic reaction to or tolerated in the past so that physicians are not inundated with highly irrelevant alerts. Alerts to previously tolerated medications might be presented in a noninterruptive fashion [48-50].

    Eighth, systems should pay more attention to the storage of override reasons data (eg, dose-range, allergy) [50,51], and encourage providers to provide accurate override reasons [52,53].

    Ninth, identify the malfunctions and pattern of malfunctions in the CDSS [54].

    Tenth, it is essential to remove the repetitive and duplicate nature of alerts in the CDSS [21,55].

    Eleventh, it is important to understand the system behavior and patterns of physicians in accepting and rejecting the alerts [18].

    Twelfth, the system can trigger an alert based on the specialty of physicians (eg, do not provide too many renal alerts for kidney specialists and those with many years of experience in this field) [30,56].

    Thirteenth, a drug-drug interaction alert can be presented in an “alert tiering” based on the level of severity. For example, level-1, level-2, and level-3 will be considered as life-threatening, less serious, and least serious, respectively. For level-1 and level-2, hard-stop alerts will be applied, whereas a passive alert (no need for physicians action) can be applied [57].

    Fourteenth, it is important to use hard-stop alerts for drug-drug interactions, renal, and geriatric alerts that might harm patients and to use soft-stop alerts for a formulary, drug-allergy, and drug-class alerts that have a lower risk for patient harm [57].

    Fifteenth, review alerts periodically and improve according to clinical importance [58-60].

    Sixteenth, always encourage physicians to provide override reasons. Learn from the override reasons and place maximum effort to improve the system [61].

    Seventeenth, when designing the system, form a multidisciplinary committee consisting of physicians, pharmacists, information technology specialists, and quality administrators [62-64].

    Finally, do not establish a silo alert system (always integrate multidepartment data) [62].

    Strengths and Limitations

    There are several strengths of our study that should be mentioned. First, this is the very first systematic review that summarizes the overall override rate, reasons for overriding the alerts, and the appropriateness of the reasons. Second, we have also provided an override rate and the proportion of appropriateness according to various types of alerts. These data can help policymakers in determining the area that they should place more focus to reduce alert fatigue. Finally, we have provided recommendations to optimize alert types and to improve the clinical relevance of alerts while suppressing alert fatigue for the CDSS that is often injudiciously overridden.

    Our study also has several limitations. First, we could not determine the bias of the included studies because of the heterogeneous nature of the studies. Second, we could not provide the percentage of ADEs when alerts were inappropriately overridden owing to data scarcity. Finally, some studies used a random sampling method of alert overrides reviewed for appropriateness that was very trivial compared with the entire alerts fired up in the CDSS, and the accuracy of such reviews was completely reliant on information contained in the patients’ charts. However, this may vary from study to study.

    Conclusion

    The findings of our study show that a high proportion of alerts are overridden and the rate of appropriateness varies widely according to alert type. Although the CDSS is an extremely effective tool for reducing patient harm and improving quality of care, it could also diminish patient safety if information technology vendors and health care professionals do not appropriately design the clinical interface. Future research should be focused on how to obtain meaningful information for analyzing these override reasons and how to integrate patient-specific factors to reduce alert fatigue, resulting in a more efficient, safe, and effective system.

    Acknowledgments

    We would like to thank our colleague who is a native English speaker for editing our manuscript. This research is funded in part by the Ministry of Education (MOE) under grants MOE 108-6604-001-400 and DP2-109-21121-01-A-01, and by the Ministry of Science and Technology (MOST) under grants MOST 108-2823-8-038-002 and 109-2222-E-038-002-MY2.

    Conflicts of Interest

    None declared.

    References

    1. Wright A, McEvoy DS, Aaron S, McCoy AB, Amato MG, Kim H, et al. Structured override reasons for drug-drug interaction alerts in electronic health records. J Am Med Inform Assoc 2019 Oct 01;26(10):934-942 [FREE Full text] [CrossRef] [Medline]
    2. Berner E. Agency for Healthcare Research and Quality. 2009 Jun. Clinical decision support systemstate of the art (AHRQ Publication No 09-0069-EF)   URL: https://digital.ahrq.gov/sites/default/files/docs/page/09-0069-EF_1.pdf [accessed 2019-05-08]
    3. Burgos F, Melia U, Vallverdú M, Velickovski F, Lluch-Ariet M, Caminal P, et al. Clinical decision support system to enhance quality control of spirometry using information and communication technologies. JMIR Med Inform 2014;2(2):e29 [FREE Full text] [CrossRef] [Medline]
    4. Nanji KC, Slight SP, Seger DL, Cho I, Fiskio JM, Redden LM, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014;21(3):487-491 [FREE Full text] [CrossRef] [Medline]
    5. Zenziper Straichman Y, Kurnik D, Matok I, Halkin H, Markovits N, Ziv A, et al. Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients. Int J Med Inform 2017 Nov;107:70-75. [CrossRef] [Medline]
    6. Lin C, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs' Computerized Patient Record System. J Am Med Inform Assoc 2008;15(5):620-626 [FREE Full text] [CrossRef] [Medline]
    7. Wong A, Rehr C, Seger DL, Amato MG, Beeler PE, Slight SP, et al. Evaluation of Harm Associated with High Dose-Range Clinical Decision Support Overrides in the Intensive Care Unit. Drug Saf 2019 Apr;42(4):573-579. [CrossRef] [Medline]
    8. Rehr CA, Wong A, Seger DL, Bates DW. Determining Inappropriate Medication Alerts from "Inaccurate Warning" Overrides in the Intensive Care Unit. Appl Clin Inform 2018 Apr;9(2):268-274 [FREE Full text] [CrossRef] [Medline]
    9. Peterson JF, Bates DW. Preventable medication errors: identifying and eliminating serious drug interactions. J Am Pharm Assoc (Wash) 2001;41(2):159-160. [CrossRef] [Medline]
    10. Seidling HM, Phansalkar S, Seger DL, Paterno MD, Shaykevich S, Haefeli WE, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011;18(4):479-484 [FREE Full text] [CrossRef] [Medline]
    11. Slight SP, Seger DL, Nanji KC, Cho I, Maniam N, Dykes PC, et al. Are we heeding the warning signs? Examining providers' overrides of computerized drug-drug interaction alerts in primary care. PLoS One 2013;8(12):e85071 [FREE Full text] [CrossRef] [Medline]
    12. Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013 Oct 01;13:111 [FREE Full text] [CrossRef] [Medline]
    13. Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc 2018 Nov 01;25(11):1556-1566 [FREE Full text] [CrossRef] [Medline]
    14. McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012;19(3):346-352 [FREE Full text] [CrossRef] [Medline]
    15. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000 Apr 19;283(15):2008-2012. [CrossRef] [Medline]
    16. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009 Aug 18;151(4):264-269. [CrossRef] [Medline]
    17. Wong A, Seger DL, Slight SP, Amato MG, Beeler PE, Fiskio JM, et al. Evaluation of 'Definite' Anaphylaxis Drug Allergy Alert Overrides in Inpatient and Outpatient Settings. Drug Saf 2018 Mar;41(3):297-302. [CrossRef] [Medline]
    18. Cho I, Lee Y, Lee J, Bates DW. Wide variation and patterns of physicians' responses to drug-drug interaction alerts. Int J Qual Health Care 2019 Mar 01;31(2):89-95. [CrossRef] [Medline]
    19. Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018 May 01;25(5):476-481. [CrossRef] [Medline]
    20. Wong A, Amato MG, Seger DL, Rehr C, Wright A, Slight SP, et al. Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit. BMJ Qual Saf 2018 Sep;27(9):718-724. [CrossRef] [Medline]
    21. Wong A, Amato MG, Seger DL, Slight SP, Beeler PE, Dykes PC, et al. Evaluation of medication-related clinical decision support alert overrides in the intensive care unit. J Crit Care 2017 Jun;39:156-161. [CrossRef] [Medline]
    22. Slight SP, Beeler PE, Seger DL, Amato MG, Her QL, Swerdloff M, et al. A cross-sectional observational study of high override rates of drug allergy alerts in inpatient and outpatient settings, and opportunities for improvement. BMJ Qual Saf 2017 Mar;26(3):217-225 [FREE Full text] [CrossRef] [Medline]
    23. Topaz M, Seger DL, Slight SP, Goss F, Lai K, Wickner PG, et al. Rising drug allergy alert overrides in electronic health records: an observational retrospective study of a decade of experience. J Am Med Inform Assoc 2016 May;23(3):601-608. [CrossRef] [Medline]
    24. Her QL, Amato MG, Seger DL, Beeler PE, Slight SP, Dalleur O, et al. The frequency of inappropriate nonformulary medication alert overrides in the inpatient setting. J Am Med Inform Assoc 2016 Sep;23(5):924-933. [CrossRef] [Medline]
    25. Topaz M, Seger DL, Lai K, Wickner PG, Goss F, Dhopeshwarkar N, et al. High Override Rate for Opioid Drug-allergy Interaction Alerts: Current Trends and Recommendations for Future. Stud Health Technol Inform 2015;216:242-246 [FREE Full text] [Medline]
    26. Ahn EK, Cho S, Shin D, Jang C, Park RW. Differences of Reasons for Alert Overrides on Contraindicated Co-prescriptions by Admitting Department. Healthc Inform Res 2014 Oct;20(4):280-287 [FREE Full text] [CrossRef] [Medline]
    27. Cho I, Slight SP, Nanji KC, Seger DL, Maniam N, Dykes PC, et al. Understanding physicians' behavior toward alerts about nephrotoxic medications in outpatients: a cross-sectional analysis. BMC Nephrol 2014 Dec 15;15:200 [FREE Full text] [CrossRef] [Medline]
    28. Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the Meaningful Use era: no evidence of progress. Appl Clin Inform 2014;5(3):802-813 [FREE Full text] [CrossRef] [Medline]
    29. Mille F, Schwartz C, Brion F, Fontan J, Bourdon O, Degoulet P, et al. Analysis of overridden alerts in a drug-drug interaction detection system. Int J Qual Health Care 2008 Dec;20(6):400-405. [CrossRef] [Medline]
    30. Moltu C, Stefansen J, Svisdahl M, Veseth M. Negotiating the coresearcher mandate - service users' experiences of doing collaborative research on mental health. Disabil Rehabil 2012;34(19):1608-1616. [CrossRef] [Medline]
    31. Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13(1):5-11 [FREE Full text] [CrossRef] [Medline]
    32. Hsieh TC, Kuperman GJ, Jaggi T, Hojnowski-Diaz P, Fiskio J, Williams DH, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004;11(6):482-491 [FREE Full text] [CrossRef] [Medline]
    33. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003 Nov 24;163(21):2625-2631. [CrossRef] [Medline]
    34. Jani YH, Barber N, Wong ICK. Characteristics of clinical decision support alert overrides in an electronic prescribing system at a tertiary care paediatric hospital. Int J Pharm Pract 2011 Oct;19(5):363-366. [CrossRef] [Medline]
    35. Khajouei R, Jaspers MWM. The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med 2010;49(1):3-19. [CrossRef] [Medline]
    36. Cash JJ. Alert fatigue. Am J Health Syst Pharm 2009 Dec 01;66(23):2098-2101. [CrossRef] [Medline]
    37. Ohta Y, Sakuma M, Koike K, Bates DW, Morimoto T. Influence of adverse drug events on morbidity and mortality in intensive care units: the JADE study. Int J Qual Health Care 2014 Dec;26(6):573-578. [CrossRef] [Medline]
    38. Cañas F, Tanasijevic MJ, Ma'luf N, Bates DW. Evaluating the appropriateness of digoxin level monitoring. Arch Intern Med 1999 Feb 22;159(4):363-368. [CrossRef] [Medline]
    39. Mullins RJ. Anaphylaxis: risk factors for recurrence. Clin Exp Allergy 2003 Aug;33(8):1033-1040. [CrossRef] [Medline]
    40. Ma L, Danoff TM, Borish L. Case fatality and population mortality associated with anaphylaxis in the United States. J Allergy Clin Immunol 2014 Apr;133(4):1075-1083 [FREE Full text] [CrossRef] [Medline]
    41. Turner PJ, Gowland MH, Sharma V, Ierodiakonou D, Harper N, Garcez T, et al. Increase in anaphylaxis-related hospitalizations but no increase in fatalities: an analysis of United Kingdom national anaphylaxis data, 1992-2012. J Allergy Clin Immunol 2015 Apr;135(4):956-963 [FREE Full text] [CrossRef] [Medline]
    42. Osheroff J. Improving Medication Use and Outcomes with Clinical Decision Support: A Step by Step Guide. Chicago: HIMSS; 2009.
    43. Coleman JJ, Hodson J, Ferner RE. Deriving dose limits for warnings in electronic prescribing systems: statistical analysis of prescription data at University Hospital Birmingham, UK. Drug Saf 2012 Apr 01;35(4):291-298. [CrossRef] [Medline]
    44. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff (Millwood) 2011 Dec;30(12):2310-2317. [CrossRef] [Medline]
    45. Kawamoto K, Lobach DF. Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior. AMIA Annu Symp Proc 2003:361-365 [FREE Full text] [Medline]
    46. Feldstein A, Simon SR, Schneider J, Krall M, Laferriere D, Smith DH, et al. How to Design Computerized Alerts to Ensure Safe Prescribing Practices. Joint Commis J Qual Safet 2004 Nov;30(11):602-613. [CrossRef]
    47. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-147 [FREE Full text] [CrossRef] [Medline]
    48. Seidling HM, Schmitt SPW, Bruckner T, Kaltschmidt J, Pruszydlo MG, Senger C, et al. Patient-specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care 2010 Oct;19(5):e15. [CrossRef] [Medline]
    49. Eschmann E, Beeler PE, Schneemann M, Blaser J. Developing strategies for predicting hyperkalemia in potassium-increasing drug-drug interactions. J Am Med Inform Assoc 2017 Jan;24(1):60-66. [CrossRef] [Medline]
    50. Seidling HM, Al Barmawi A, Kaltschmidt J, Bertsche T, Pruszydlo MG, Haefeli WE. Detection and prevention of prescriptions with excessive doses in electronic prescribing systems. Eur J Clin Pharmacol 2007 Dec;63(12):1185-1192. [CrossRef] [Medline]
    51. Coleman JJ, Nwulu U, Ferner RE. Decision support for sensible dosing in electronic prescribing systems. J Clin Pharm Ther 2012 Aug;37(4):415-419. [CrossRef] [Medline]
    52. Topaz M, Goss F, Blumenthal K, Lai K, Seger DL, Slight SP, et al. Towards improved drug allergy alerts: Multidisciplinary expert recommendations. Int J Med Inform 2017 Jan;97:353-355 [FREE Full text] [CrossRef] [Medline]
    53. Cahill KN, Johns CB, Cui J, Wickner P, Bates DW, Laidlaw TM, et al. Automated identification of an aspirin-exacerbated respiratory disease cohort. J Allergy Clin Immunol 2017 Mar;139(3):819-825 [FREE Full text] [CrossRef] [Medline]
    54. Wright A, Hickman TT, McEvoy D, Aaron S, Ai A, Andersen JM, et al. Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016 Nov;23(6):1068-1076 [FREE Full text] [CrossRef] [Medline]
    55. van der Sijs H, Mulder A, van Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf 2009 Oct;18(10):941-947. [CrossRef] [Medline]
    56. Krall MA, Sittig DF. Clinician's assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp 2002:400-404 [FREE Full text] [Medline]
    57. Tilson H, Hines LE, McEvoy G, Weinstein DM, Hansten PD, Matuszewski K, et al. Recommendations for selecting drug-drug interactions for clinical decision support. Am J Health Syst Pharm 2016 Apr 15;73(8):576-585 [FREE Full text] [CrossRef] [Medline]
    58. Phansalkar S, Edworthy J, Hellier E, Seger DL, Schedlbauer A, Avery AJ, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010;17(5):493-501 [FREE Full text] [CrossRef] [Medline]
    59. McEvoy DS, Sittig DF, Hickman T, Aaron S, Ai A, Amato M, et al. Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 2017 Mar 01;24(2):331-338 [FREE Full text] [CrossRef] [Medline]
    60. Smithburger PL, Kane-Gill SL, Benedict NJ, Falcione BA, Seybert AL. Grading the severity of drug-drug interactions in the intensive care unit: a comparison between clinician assessment and proprietary database severity rankings. Ann Pharmacother 2010 Nov;44(11):1718-1724. [CrossRef] [Medline]
    61. Phansalkar S, Desai AA, Bell D, Yoshida E, Doole J, Czochanski M, et al. High-priority drug-drug interactions for use in electronic health records. J Am Med Inform Assoc 2012;19(5):735-743 [FREE Full text] [CrossRef] [Medline]
    62. Riedmann D, Jung M, Hackl WO, Stühlinger W, van der Sijs H, Ammenwerth E. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak 2011 May 25;11:35 [FREE Full text] [CrossRef] [Medline]
    63. Seidling HM, Klein U, Schaier M, Czock D, Theile D, Pruszydlo MG, et al. What, if all alerts were specific - estimating the potential impact on drug interaction alert burden. Int J Med Inform 2014 Apr;83(4):285-291. [CrossRef] [Medline]
    64. Yang C, Lo Y, Chen R, Liu C. A Clinical Decision Support Engine Based on a National Medication Repository for the Detection of Potential Duplicate Medications: Design and Evaluation. JMIR Med Inform 2018 Jan 19;6(1):e6 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    ADE: adverse drug effects
    CDSS: clinical decision support system
    CPOE: computerized provider order entry
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses


    Edited by G Eysenbach, K Fortuna; submitted 26.07.19; peer-reviewed by G Schiff, K Fuji, M Hellaby; comments to author 07.03.20; revised version received 13.03.20; accepted 30.03.20; published 20.07.20

    ©Tahmina Nasrin Poly, Md.Mohaimenul Islam, Hsuan-Chia Yang, Yu-Chuan (Jack) Li. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 20.07.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.