Maintenance Note

On Friday, August 31, 2018 at 12:00 pm Eastern Time, JMIR will be completing a server migration to improve site stability and user experience. We expect to be back online Friday, August 31, 2018 at 5:00 pm Eastern Time. Should any problems arise our technical team will be using the weekend to resolve them, and users will be able to access our site by Sunday, September 2, 2018 at 1:00pm Eastern Time.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 24.01.18 in Vol 6, No 1 (2018): Jan-Mar

This paper is in the following e-collection/theme issue:

    Review

    Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review

    1Division of Pharmacy, University Hospitals of Geneva, Geneva, Switzerland

    2School of Pharmaceutical Sciences, University of Geneva, University of Lausanne, Geneva, Switzerland

    3Division of Medical Information Sciences, University Hospitals of Geneva, Geneva, Switzerland

    4School of Medicine, University of Geneva, Geneva, Switzerland

    Corresponding Author:

    Delphine Carli, PharmD, PhD

    Division of Pharmacy

    University Hospitals of Geneva

    Rue Gabrielle-Perret-Gentil 4

    Geneva, 1211

    Switzerland

    Phone: 41 786532871

    Fax:41 223726255

    Email:


    ABSTRACT

    Background: Computerized decision support systems have raised a lot of hopes and expectations in the field of order entry. Although there are numerous studies reporting positive impacts, concerns are increasingly high about alert fatigue and effective impacts of these systems. One of the root causes of fatigue alert reported is the low clinical relevance of these alerts.

    Objective: The objective of this systematic review was to assess the reported positive predictive value (PPV), as a proxy to clinical relevance, of decision support systems in computerized provider order entry (CPOE).

    Methods: A systematic search of the scientific literature published between February 2009 and March 2015 on CPOE, clinical decision support systems, and the predictive value associated with alert fatigue was conducted using PubMed database. Inclusion criteria were as follows: English language, full text available (free or pay for access), assessed medication, direct or indirect level of predictive value, sensitivity, or specificity. When possible with the information provided, PPV was calculated or evaluated.

    Results: Additive queries on PubMed retrieved 928 candidate papers. Of these, 376 were eligible based on abstract. Finally, 26 studies qualified for a full-text review, and 17 provided enough information for the study objectives. An additional 4 papers were added from the references of the reviewed papers. The results demonstrate massive variations in PPVs ranging from 8% to 83% according to the object of the decision support, with most results between 20% and 40%. The best results were observed when patients’ characteristics, such as comorbidity or laboratory test results, were taken into account. There was also an important variation in sensitivity, ranging from 38% to 91%.

    Conclusions: There is increasing reporting of alerts override in CPOE decision support. Several causes are discussed in the literature, the most important one being the clinical relevance of alerts. In this paper, we tried to assess formally the clinical relevance of alerts, using a near-strong proxy, which is the PPV of alerts, or any way to express it such as the rate of true and false positive alerts. In doing this literature review, three inferences were drawn. First, very few papers report direct or enough indirect elements that support the use or the computation of PPV, which is a gold standard for all diagnostic tools in medicine and should be systematically reported for decision support. Second, the PPV varies a lot according to the typology of decision support, so that overall rates are not useful, but must be reported by the type of alert. Finally, in general, the PPVs are below or near 50%, which can be considered as very low.

    JMIR Med Inform 2018;6(1):e3

    doi:10.2196/medinform.7170

    KEYWORDS



    Introduction

    Computerized patient records and computerized provider order entry (CPOE) systems are recognized as major tools in efforts to improve the safety and efficiency of care. Computerized patient records are the cornerstone of information sharing among care providers, and increasingly with patients; they contribute to improving the continuum of care and patient safety. The way CPOE improves processes rests on 3 pillars. The first pillar is formal structured order entry, which improves both completeness and readability. The second embeds CPOE into complete care processes such as medication loops or clinical pathways. The third pillar is the decision support capability during the ordering process, such as the provision of extensive information on the drugs being prescribed or the links made between the current order and other elements of the patient’s record such as problems, laboratory results, and other drugs or diagnoses. Numerous studies have reported the positive effects of clinical decision support systems (CDSS) on patient outcomes such as fewer duplicate orders, dosage errors, drug interactions, and missed or delayed actions using reminders, to name a few [1-4]. The benefits of CPOE have already been demonstrated in the improved cost-efficiency of care, either directly, by lowering adverse events and duplicate orders, or indirectly, by reducing lengths of stay [5,6]. Nevertheless, the burden of alerts and reminders must not be too high or alert fatigue could cause clinicians to override both important and unimportant alerts, thus jeopardizing the improvements in safety that a CDSS should be expected to bring [7]. In other words, the CDSS’s specificity (Sp) must be high. A few studies have reported on the unintended effects of CDSS in CPOE [8-10] and their occasional dramatic consequences on patient safety. These were related to delays in reporting adverse events, and thus therapy, leading to specific infectious or thrombotic complications in treatment [11] or to the cancellation of QT interval-alert generation after proposed measures to reduce alert overload [12]. This is not a marginal problem. For example, a 2013 study published by Yeh, analyzing more than 1 million prescriptions from outpatient settings in Taiwan, reported a 91.5% override rate on the approximately 11,000 drug-drug interaction alerts proposed [13]. Understanding the reasons why clinicians override CDSS in CPOE has since received a lot of attention [14,15]. In recent years, numerous studies have been published on the topic of alert improvements for CPOE. These addressed the theoretical background, such as models and frameworks [16], data representation [17] or behavioral theories [18], usability and interfaces [19,20], perceptions and expectations [21], simulation [22], effectiveness monitoring [23,24], and decision support Sp [25], among other issues.

    This study focuses on the predictive value of CPOE alerts. One can consider the CDSS in CPOE to be akin to any other decision support instrument in medicine: a tool with positive predictive values (PPVs) and negative predictive values (NPVs). As mentioned above, some previous studies have focused on evaluating the predictive value of decision support in CPOE, and the PPVs reported were usually below 20% and as low as 5% [26,27]. A study by van der Sijs et al stated that 49% to 96% of alerts were overridden [28] and identified a range of human factors responsible:

    • alert fatigue due to a poor signal-to-noise ratio as a result of a low PPV
    • usability issues such as bad ergonomics, misinterpretation, or unnoticed alerts
    • disagreements with guidelines
    • physicians’ belief in their own knowledge
    • lack of time

    Further understanding has been provided by questionnaires and focus groups that allowed physicians to evaluate the most important factors for useful, easy-to-use alerts [29,30]. These showed that drug-related alerts were rated more useful than alerts reminding the clinician of the state of the patient’s health or disease. Shah et al suggested that an approach based on a careful selection of alerts so as to improve the relevancy, severity, likelihood, and strength of clinical evidence would improve the acceptance of alerts [31]. Bates et al put forward “Ten commandments for effective clinical decision support” such as speed of the information system, anticipation of clinician needs and provide information to clinicians at the time they need it, integration suggestions with practice, offer an alternative, change of direction rather than stop or management, and maintenance of knowledge-based systems [32].

    As stated, most alerts are overridden. Although numerous authors speak about the number of alerts, or the pertinence of alerts, we have been interested in trying to assess clearly the PPV of alerts, and thus the rate of true and false positive alerts. In doing this review, three inferences were drawn. First, very few papers report direct or enough indirect elements that support the use or the computation of PPV, which is a gold standard for all diagnostic tools in medicine, which is why it should be systematically reported for decision support. Second, the PPV varies a lot according to the typology of decision support and would have to be reported by the type of alert. Third is that, in general, the PPVs can be considered as very low—below 50% or near 50%.

    Due to the high expectations health care professionals have for CDSS in CPOE, as well as the related costs and potential unintended consequences, we decided to carry out a systematic review of the literature on CPOE, CDSS, and predictive value, and their associations with alert fatigue. We start from the assumption that a low PPV would explain why majority of alerts are overridden. We framed this systematic review to determine the real PPV of CPOE alerts.


    Methods

    Selection Criteria

    We targeted publications evaluating clinically relevant alert in computerized patient records implementing CPOE.

    Search Strategy

    A search of the literature was made using PubMed for work published between February 2009 and March 2015, using the following queries: (CPOE[all fields] OR “Medical Order Entry Systems”[all fields] OR “Alert Systems”[all fields] OR “Order Entry”[all fields] OR “Decision support Systems”[all fields]) AND (sensitivity[All Fields] OR sensibility[All Fields] OR predictive[All Fields]) OR (fatigue[All Fields] OR overload[All Fields] OR overcharge[All Fields] OR burden[All Fields] OR override[All Fields] OR overalerting[All Fields] OR ignore[All Fields]). The following meanings were searched for decision support: CPOE, medical order entry systems, alert systems, order entry, and decision support systems. The following meanings were searched for relevance: sensitivity, sensibility, predictive, fatigue, overload, overcharge, burden, override, over alerting, and ignore.

    The following limits were applied to all queries: English language, only papers available in full text, assessing medication, and numerical data available.

    We excluded qualitative studies, user-satisfaction or opinion surveys, physician adherence studies, and analyses of the impact of human factors.

    Selection of Relevant Publications

    First, the 3 reviewers (DC, GF, and CL) selected references independently based on their titles and according to the review study’s inclusion and exclusion criteria. When results were discordant, the final choice was made by consensus. Next, they independently read and assessed the abstracts of all the papers identified. When no abstract was available, full-text papers were retrieved and reviewed so that only relevant papers were retained. Again, the 3 reviewers solved any disagreements by consensus. In the absence of an agreement, the abstract was provisionally included for consideration subject to reading the full text.

    Abstracts that were rated as relevant to the research question were kept, and all full-text papers were retrieved. Then, each retrieved paper’s reference section was searched for additional relevant literature that might be included.

    Of the reviewers, 2 (DC and GF) assessed the quality of the papers selected by using a standardized evaluation process based on the exclusion and inclusion criteria. For papers to be selected for the final review, the levels of predictive value, sensitivity (Se; ability to generate alerts in potentially dangerous situations), or Sp (inability to prevent irrelevant alerts) were retrieved or calculated if possible. Se was defined as the number of patients with an adverse drug event (ADE) detected by an alert, out of the total number of patients with a positive ADE. Sp was defined as the number of patients without an ADE and with no warning alert, out of the total number of patients without an ADE. The PPV was defined as the number of relevant medication alerts (true positives) out of the total number of alerts (sum of true and false positives). Evaluation disagreements between the 2 reviewers were resolved by the third reviewer (CL).


    Results

    Selection of Studies

    The database search retrieved 928 matching references. A first evaluation based on MEDLINE summary allowed identifying 402 potentially interesting papers. Then, a second deeper analysis based on abstracts and applying the inclusion and exclusion criteria resulted in the exclusion of 311 articles, thus reducing the initial set to 91 reports. Out of these, 26 full-text papers were retrieved, reviewed, and included in the next phase of the review. The additional search through the selected studies’ reference sections resulted in 20 additional potentially relevant papers. Of these, 4 were included in our analysis. The review selection process is summarized in Figure 1.

    Description of Studies

    Including the additional search references, the final sample of 17 studies that met our eligibility criteria, as listed in Table 1, were published between 1998 and 2015. The papers predominantly analyzed interruptive alerts (n=7/8 notified). Various alert targets were used and are described in Table 2. The main ones described were drug-lab interactions (n=11), drug-dosage interactions (n=8), drug-drug interactions (n=6), duplicate orders (n=3), and drug–allergy interactions (n=3).

    These papers report the predictive value or Se and Sp of the alerts studied. As shown in Table 3, four papers did not report any PPV, although this study’s authors were able to calculate it for two of those papers. The PPV found in the papers were usually low and heterogeneous, mostly between 20% and 40%. Despite the diversity of target alerts, alert notifications, study designs, and study periods of the papers included in this review, it seems that PPVs were higher for drug-lab interactions (2.3%-83%) than they were for drug-dosage interactions (8%-13.8%), or drug-drug interactions (1.6%-48%). Furthermore, advanced CDSS [49] showed higher PPV than the more basic ones (17%-97%).

    The Types of Alert Influencing PPV

    In general, PPV increased when the risk increased. For example, PPV was higher for drug-dosage interactions than for drug-lab interactions. This is probably because of the higher risk of experiencing an ADE [48]. Furthermore, the PPV was lower in prevention (the opportunity to prevent ADEs) than in detection (evaluate or treat possible existing ADEs): 24% versus 97% [46]. Indeed PPV is related to the prevalence (Prev) unlike Se and Sp, which are only related to the test as defined as defined as follows: PPV=(Se×Prev)÷(Se×Prev+(1−Sp)×(1−Prev)). Therefore, in prevention settings, the prevalence of disease is likely to be very low, so the PPV will also therefore be low. Additionally, it was shown that the PPV of alerts targeting drug-lab interactions varied with the choice of the alarm signal. Indeed, for a laboratory value lower than the maximum defined value, the PPV of the alert was 36% (95% CI 29-43). If an alert was activated after at least a 50% decrease in the value between the last two laboratory results, the PPV increased to 83% (95% CI 62-104). For two consecutive decreases, with at least a 25% difference between the third most recent and the most recent platelet count, the PPV was 40% (95% CI 32-48) [50].

    Furthermore, it has been shown that the PPV of safety alerts aimed at high-risk patients was higher (PPV=14%) than when dealing with initiation of a drug (PPV=6%), ongoing use of a drug (PPV=6%), advice (PPV=7%), and medication used to treat an ADE (PPV=0%) [28]. In summary, the PPV of alerts is usually very low. However, several factors seem to improve PPV.

    Contextual Information Improves PPV

    The PPV of advanced alerts is higher than for basic alerts because they are more specific. Advanced CDSS, such as using patients’ characteristics and laboratory test results, have a higher PPV than basic ones. For example, Eppenga et al showed that using information from the laboratory and a few other specific pieces of information increased the PPV from 12.2% to 23.3% (P<.05) and that PPV was higher in advanced systems than in basic ones (17% vs 5.8%, P<.05) [37]. Numerous factors can influence the PPV, mostly because they will have influence of the population considered for the alert. For example, not specifying the administration route can sometimes decrease the PPV, for example in some topical treatments. This is because the risk of developing an ADE can vary according to the administration route [50]. Further advances in dosing alert systems should aim to improve the Se of alerts. The Se of the system for identifying dosing errors increased from 54.1% (95% CI 47.8-60.3) to 60.3% (95% CI 54.0-66.3) in a customized dose range system (P=.02). The system’s Se for underdosage was 49.6% without customization, and this increased to 60.3% with customization (P=.01) [47]. Furthermore, it has been highlighted that PPV differs according to patients’ characteristics and comorbidity: for alerts on the risk of developing hypoglycemia, the PPV was higher for patients with sulfonylureas in their drug regimens (45.7% vs 28.4%, P=.04) and for patients with three or more chronic medical conditions (35.7% vs 22.7%, P=.049). The PPV of an alert warning of the risk of developing hyperkalemia was higher for patients with serum creatinine >2.0 mg/dL (50.0% vs 16.0%, P=.01) [38].

    The PPV can vary according to the types of alerts. Among the 5 types of alerts with the best PPV (34.1%-73.3%), 3 were drug-lab interactions, which are advanced alerts. In parallel, of the 10 alerts described as being the least relevant (PPV between 0% and 4.5%), 8 were drug-drug interactions [37].

    Finally, the PPV varies according to the specific goal. A study of alerts aimed at identifying 4 types of ADE showed that some of them could have a lower PPV: the PPV was only 4.0% (95% CI 1.3-9.1) for hypokalemia versus 31.2% (95% CI 18.2-46.6) for hypoglycemia, 31.1% (95% CI 25.1-37.8) for hyperkalemia, and 20.6% (95% CI 11.7-32.1) for thrombocytopenia. Furthermore, the effect of an alert can differ according to the medical specialty. In a study by Riggio et al, a surgery department ordered laboratory tests earlier than general medicine department when alerts were activated, probably because surgeons were more aware of the importance of the platelet counts that were being observed in the study [33]. The PPV can also vary according to the alert’s pharmacological target. For example, anti-infective drugs are excluded from alerts concerning drug dosage interactions to limit the number of false positives because these drugs could present patients specific dosing adjustment and multiple indications [44].

    Figure 1. Flowchart describing the systematic literature review process.
    View this figure
    Table 1. Characteristics of the studies included in the paper.
    View this table
    Table 2. Characteristics of alerts included in the paper.
    View this table
    Table 3. Positive predictive value (PPV), sensitivity or specificty for studies included in the review.
    View this table

    Discussion

    Principal Findings

    The PPV found in the papers were rather low: 20% to 40%. Despite the heterogeneity of papers, it seems that several factors influence PPV. First, the PPV can vary with the types of alert such as the risk patients trying to be prevented. Furthermore, several factors seem to improve PPV such as contextual information. Indeed alerts that are more specific have a higher PPV than basic alerts specifying the administration route or patients’ characteristics for example. Moreover, PPV can differ according to alert’s pharmacological target or medical specialty.

    Even the most basic systems usually show good Se. They thereby allow medical professionals to detect drug-related problems more comprehensively: a pharmacy department increased the number of its clinical interventions by 15% after the introduction of a CDSS [47]. However, the impact of a true positive alert can be paradoxical. For example, patients presented no reduction in ADEs, time to therapeutic intervention, or time to laboratory testing in an alert group, and physicians waited 1.6 days longer before stopping a treatment inducing ADE in that alert group (P=.049) [49]. This result could be because of alert fatigue induced by a low PPV.

    This study has several limitations. First of all, we conducted our research using only PubMed, and carried no queries using EMBASE, Web of Science, or conference proceedings. The results are based on few reports, as only few studies reported all characteristics required to assess properly the contexts of decision support and their associated predictive values. There was a wide heterogeneity in how results were reported, completeness, and evaluation methodologies, thus limiting the reliability of pooling the PPV of alerts across publications. Because PPV varies with prevalence, the patient context, including population, hospital settings, and the like, has influence, and could not be considered. Thus, these results introduce some types of biases into the overall assessment.

    Studies about interruptive alerts had some homogeneity in their methodology, and studies on decision support were mostly about 3 types interactions: drug-lab, drug-drug, and drug-dosage. These 3 types of interactions were the easiest to implement, and there are several large databases available for each of them. In general, systems that do not take patients’ specific clinical information into account and use only external databases demonstrate the lowest PPV; systems that have a specific source of knowledge and use the greatest number of patients’ individual characteristics have the highest PPV.

    Conclusions

    The PPV of clinical decision support systems for CPOE, as reported in the literature, varies massively, from 5.8% to 83%, with the majority of results between 20% and 40%. Drug-drug interaction alerts have the lowest PPV, and drug-lab alerts have the highest.

    Our literature review leads us to suggest that the best strategy to use with a CPOE is to adapt and carefully optimize the database driving the knowledge for activating alerts. Furthermore, the CDSS should take into account as many of the patient’s characteristics as possible. The efficiency of the alerts, and thus their PPV, is more important than a very large database of knowledge that may generate lots of false positives, which reduce PPV and generate alert fatigue.

    Advanced alert systems should aim to improve PPV of alerts, while keeping a good Se. To reduce the number of false positive alerts, contextual data from different sources, such as the pharmacy, demographic data, or laboratory tests, should be integrated into the system.

    The US Institute of Medicine has suggested that systems should be designed so as to make it “hard for people to do the wrong thing and easy for people to do the right thing” [51]. However, with PPVs as low as those seen in the literature, it seems, unfortunately, that many computerized patient records tend to make it hard for people to do the right thing and easy for people to do the wrong thing.

    Conflicts of Interest

    None declared.

    References

    1. Abramson EL, Kaushal R. Computerized provider order entry and patient safety. Pediatr Clin North Am 2012 Dec;59(6):1247-1255. [CrossRef] [Medline]
    2. Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003 Jun 19;348(25):2526-2534. [CrossRef] [Medline]
    3. Radley DC, Wasserman MR, Olsho LE, Shoemaker SJ, Spranca MD, Bradshaw B. Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems. J Am Med Inform Assoc 2013 May 01;20(3):470-476 [FREE Full text] [CrossRef] [Medline]
    4. van Rosse F, Maat B, Rademaker CM, van Vught AJ, Egberts AC, Bollen CW. The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics 2009 Apr;123(4):1184-1190. [CrossRef] [Medline]
    5. Forrester SH, Hepp Z, Roth JA, Wirtz HS, Devine EB. Cost-effectiveness of a computerized provider order entry system in improving medication safety ambulatory care. Value Health 2014 Jun;17(4):340-349 [FREE Full text] [CrossRef] [Medline]
    6. Vermeulen KM, van Doormaal JE, Zaal RJ, Mol PG, Lenderink AW, Haaijer-Ruskamp FM, et al. Cost-effectiveness of an electronic medication ordering system (CPOE/CDSS) in hospitalized patients. Int J Med Inform 2014 Aug;83(8):572-580. [CrossRef] [Medline]
    7. Slight SP, Seger DL, Nanji KC, Cho I, Maniam N, Dykes PC, et al. Are we heeding the warning signs? Examining providers' overrides of computerized drug-drug interaction alerts in primary care. PLoS One 2013;8(12):e85071 [FREE Full text] [CrossRef] [Medline]
    8. Magid S, Forrer C, Shaha S. Duplicate orders: an unintended consequence of computerized provider/physician order entry (CPOE) implementation: analysis and mitigation strategies. Appl Clin Inform 2012;3(4):377-391 [FREE Full text] [CrossRef] [Medline]
    9. Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 2007 Oct 11:26-30 [FREE Full text] [Medline]
    10. Cowan L. Literature review and risk mitigation strategy for unintended consequences of computerized physician order entry. Nurs Econ 2013;31(1):27-31, 11. [Medline]
    11. Strom BL, Schinnar R, Aberra F, Bilker W, Hennessy S, Leonard CE, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010 Sep 27;170(17):1578-1583. [CrossRef] [Medline]
    12. van der Sijs H, Kowlesar R, Aarts J, Berg M, Vulto A, van Gelder T. Unintended consequences of reducing QT-alert overload in a computerized physician order entry system. Eur J Clin Pharmacol 2009 Sep;65(9):919-925 [FREE Full text] [CrossRef] [Medline]
    13. Yeh ML, Chang YJ, Wang PY, Li YC, Hsu CY. Physicians' responses to computerized drug-drug interaction alerts for outpatients. Comput Methods Programs Biomed 2013 Jul;111(1):17-25. [CrossRef] [Medline]
    14. Perna G. Clinical alerts that cried wolf. As clinical alerts pose physician workflow problems, healthcare IT leaders look for answers. Healthc Inform 2012 Apr;29(4):18, 20. [Medline]
    15. Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013;13:111 [FREE Full text] [CrossRef] [Medline]
    16. Wipfli R, Lovis C. Alerts in clinical information systems: building frameworks and prototypes. Stud Health Technol Inform 2010;155:163-169. [Medline]
    17. Luna D, Otero V, Canosa D, Montenegro S, Otero P, de Quirós FG. Analysis and redesign of a knowledge database for a drug-drug interactions alert system. Stud Health Technol Inform 2007;129(Pt 2):885-889. [Medline]
    18. Schedlbauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc 2009;16(4):531-538 [FREE Full text] [CrossRef] [Medline]
    19. Khajouei R, Jaspers MW. CPOE system design aspects and their qualitative effect on usability. Stud Health Technol Inform 2008;136:309-314. [Medline]
    20. Tsopra R, Jais JP, Venot A, Duclos C. Comparison of two kinds of interface, based on guided navigation or usability principles, for improving the adoption of computerized decision support systems: application to the prescription of antibiotics. J Am Med Inform Assoc 2014 Feb;21(e1):e107-e116 [FREE Full text] [CrossRef] [Medline]
    21. Jung M, Hoerbst A, Hackl WO, Kirrane F, Borbolla D, Jaspers MW, et al. Attitude of physicians towards automatic alerting in computerized physician order entry systems. A comparative international survey. Methods Inf Med 2013;52(2):99-108. [CrossRef] [Medline]
    22. van der Sijs H, van Gelder T, Vulto A, Berg M, Aarts J. Understanding handling of drug safety alerts: a simulation study. Int J Med Inform 2010 May;79(5):361-369. [CrossRef] [Medline]
    23. Reynolds G, Boyer D, Mackey K, Povondra L, Cummings A. Alerting strategies in computerized physician order entry: a novel use of a dashboard-style analytics tool in a children's hospital. AMIA Annu Symp Proc 2008 Nov 06:1108. [Medline]
    24. Zimmerman CR, Jackson A, Chaffee B, O'Reilly M. A dashboard model for monitoring alert effectiveness and bandwidth. AMIA Annu Symp Proc 2007 Oct 11:1176. [Medline]
    25. Smithburger PL, Buckley MS, Bejian S, Burenheide K, Kane-Gill SL. A critical evaluation of clinical decision support for the detection of drug-drug interactions. Expert Opin Drug Saf 2011 Nov;10(6):871-882. [CrossRef] [Medline]
    26. McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014;14(2):195-202 [FREE Full text] [Medline]
    27. Rommers MK, Zwaveling J, Guchelaar H, Teepe-Twiss IM. Evaluation of rule effectiveness and positive predictive value of clinical rules in a Dutch clinical decision support system in daily hospital pharmacy practice. Artif Intell Med 2013 Sep;59(1):15-21. [CrossRef] [Medline]
    28. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-147 [FREE Full text] [CrossRef] [Medline]
    29. Krall MA, Sittig DF. Subjective assessment of usefulness and appropriate presentation mode of alerts and reminders in the outpatient setting. Proc AMIA Symp 2001:334-338 [FREE Full text] [Medline]
    30. Krall MA, Sittig DF. Clinician's assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp 2002:400-404 [FREE Full text] [Medline]
    31. Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13(1):5-11 [FREE Full text] [CrossRef] [Medline]
    32. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10(6):523-530 [FREE Full text] [CrossRef] [Medline]
    33. Riggio JM, Cooper MK, Leiby BE, Walenga JM, Merli GJ, Gottlieb JE. Effectiveness of a clinical decision support system to identify heparin induced thrombocytopenia. J Thromb Thrombolysis 2009 Aug;28(2):124-131. [CrossRef] [Medline]
    34. Cash JJ. Alert fatigue. Am J Health Syst Pharm 2009 Dec 01;66(23):2098-2101. [CrossRef] [Medline]
    35. van der Sijs H, Bouamar R, van Gelder T, Aarts J, Berg M, Vulto A. Functionality test for drug safety alerting in computerized physician order entry systems. Int J Med Inform 2010 Apr;79(4):243-251. [CrossRef] [Medline]
    36. FitzHenry F, Doran J, Lobo B, Sullivan TM, Potts A, Feldott CC, et al. Medication-error alerts for warfarin orders detected by a bar-code-assisted medication administration system. Am J Health Syst Pharm 2011 Mar 01;68(5):434-441. [CrossRef] [Medline]
    37. Eppenga WL, Derijks HJ, Conemans JM, Hermens WA, Wensing M, De Smet PA. Comparison of a basic and an advanced pharmacotherapy-related clinical decision support system in a hospital care setting in the Netherlands. J Am Med Inform Assoc 2012;19(1):66-71 [FREE Full text] [CrossRef] [Medline]
    38. Moore C, Li J, Hung C, Downs J, Nebeker JR. Predictive value of alert triggers for identification of developing adverse drug events. J Patient Saf 2009 Dec;5(4):223-228. [CrossRef] [Medline]
    39. Fritz D, Ceschi A, Curkovic I, Huber M, Egbring M, Kullak-Ublick GA, et al. Comparative evaluation of three clinical decision support systems: prospective screening for medication errors in 100 medical inpatients. Eur J Clin Pharmacol 2012 Aug;68(8):1209-1219. [CrossRef] [Medline]
    40. Harinstein LM, Kane-Gill SL, Smithburger PL, Culley CM, Reddy VK, Seybert AL. Use of an abnormal laboratory value-drug combination alert to detect drug-induced thrombocytopenia in critically Ill patients. J Crit Care 2012 Jun;27(3):242-249. [CrossRef] [Medline]
    41. Zorina OI, Haueis P, Semmler A, Marti I, Gonzenbach RR, Guzek M, et al. Comparative evaluation of the drug interaction screening programs MediQ and ID PHARMA CHECK in neurological inpatients. Pharmacoepidemiol Drug Saf 2012 Aug;21(8):872-880. [CrossRef] [Medline]
    42. Beeler PE, Eschmann E, Rosen C, Blaser J. Use of an on-demand drug-drug interaction checker by prescribers and consultants: a retrospective analysis in a Swiss teaching hospital. Drug Saf 2013 Jun;36(6):427-434. [CrossRef] [Medline]
    43. Stultz JS, Nahata MC. Appropriateness of commercially available and partially customized medication dosing alerts among pediatric patients. J Am Med Inform Assoc 2014 Feb;21(e1):e35-e42 [FREE Full text] [CrossRef] [Medline]
    44. Stultz JS, Porter K, Nahata MC. Sensitivity and specificity of dosing alerts for dosing errors among hospitalized pediatric patients. J Am Med Inform Assoc 2014 Oct;21(e2):e219-e225 [FREE Full text] [CrossRef] [Medline]
    45. Dormann H, Criegee-Rieck M, Neubert A, Egger T, Levy M, Hahn EG, et al. Implementation of a computer-assisted monitoring system for the detection of adverse drug reactions in gastroenterology. Aliment Pharmacol Ther 2004 Feb 01;19(3):303-309. [Medline]
    46. Raschke RA, Gollihare B, Wunderlich TA, Guidry JR, Leibowitz AI, Peirce JC, et al. A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital. J Am Med Assoc 1998 Oct 21;280(15):1317-1320. [Medline]
    47. Silverman JB, Stapinski CD, Huber C, Ghandi TK, Churchill WW. Computer-based system for preventing adverse drug events. Am J Health Syst Pharm 2004 Aug 01;61(15):1599-1603. [Medline]
    48. Handler SM, Altman RL, Perera S, Hanlon JT, Studenski SA, Bost JE, et al. A systematic review of the performance characteristics of clinical event monitor signals used to detect adverse drug events in the hospital setting. J Am Med Inform Assoc 2007;14(4):451-458 [FREE Full text] [CrossRef] [Medline]
    49. Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007;14(1):29-40 [FREE Full text] [CrossRef] [Medline]
    50. Harinstein LM, Kane-Gill SL, Smithburger PL, Culley CM, Reddy VK, Seybert AL. Use of an abnormal laboratory value-drug combination alert to detect drug-induced thrombocytopenia in critically Ill patients. J Crit Care 2012 Jun;27(3):242-249. [CrossRef] [Medline]
    51. Institute of Medicine (US) Committee on Quality of Health Care in America. In: Kohn LT, Corrigan JM, Donaldson MS, editors. To Err Is Human: Building a Safer Health System. Washington (DC): National Academies Press (US); 2000.


    Abbreviations

    ADE: adverse drug event
    CDSS: clinical decision support systems
    CPOE: computerized provider order entry
    PPV: positive predictive value
    Se: Sensitivity
    Sp: Specificity


    Edited by G Eysenbach; submitted 04.04.17; peer-reviewed by A Elmessiry, K Singh; comments to author 05.06.17; revised version received 25.08.17; accepted 16.09.17; published 24.01.18

    ©Delphine Carli, Guillaume Fahrni, Pascal Bonnabry, Christian Lovis. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 24.01.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.