This paper is in the following e-collection/theme issue:

Decision Support for Health Professionals (1185) Clinical Information and Decision Making (1407) Safety and Error Prevention in Health (202) Computerized Provider Order Entry (CPOE) (23) Electronic Health Records (1053) Pharmacovigilance (134)

Published on in Vol 8, No 7 (2020): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/18758, first published .
Barriers and Facilitators to Implementation of Medication Decision Support Systems in Electronic Medical Records: Mixed Methods Approach Based on Structural Equation Modeling and Qualitative Analysis

Barriers and Facilitators to Implementation of Medication Decision Support Systems in Electronic Medical Records: Mixed Methods Approach Based on Structural Equation Modeling and Qualitative Analysis

Barriers and Facilitators to Implementation of Medication Decision Support Systems in Electronic Medical Records: Mixed Methods Approach Based on Structural Equation Modeling and Qualitative Analysis

Authors of this article:

Se Young Jung1, 2 Author Orcid Image ;   Hee Hwang1, 3 Author Orcid Image ;   Keehyuck Lee1, 2 Author Orcid Image ;   Ho-Young Lee1, 4 Author Orcid Image ;   Eunhye Kim1 Author Orcid Image ;   Miyoung Kim1 Author Orcid Image ;   In Young Cho2 Author Orcid Image
Article Authors Cited by (18) Tweetations (1) Metrics

    Original Paper

    1Office of eHealth Research and Businesses, Seoul National University Bundang Hospital, Seongnam, Republic of Korea

    2Department of Family Medicine, Seoul National University Bundang Hospital, Seongnam, Republic of Korea

    3Department of Pediatrics, Seoul National University Bundang Hospital, Seongnam, Republic of Korea

    4Department of Nuclear Medicine, Seoul National University Bundang Hospital, Seongnam, Republic of Korea

    *these authors contributed equally

    Corresponding Author:

    Keehyuck Lee, MD, MBA

    Office of eHealth Research and Businesses

    Seoul National University Bundang Hospital

    Dolma-ro 172, Bundang-gu

    Seongnam, 13605

    Republic of Korea

    Phone: 82 317878992

    Email: chrisruga@naver.com


    Background: Adverse drug events (ADEs) resulting from medication error are some of the most common causes of iatrogenic injuries in hospitals. With the appropriate use of medication, ADEs can be prevented and ameliorated. Efforts to reduce medication errors and prevent ADEs have been made by implementing a medication decision support system (MDSS) in electronic health records (EHRs). However, physicians tend to override most MDSS alerts.

    Objective: In order to improve MDSS functionality, we must understand what factors users consider essential for the successful implementation of an MDSS into their clinical setting. This study followed the implementation process for an MDSS within a comprehensive EHR system and analyzed the relevant barriers and facilitators.

    Methods: A mixed research methodology was adopted. Data from a structured survey and 15 in-depth interviews were integrated. Structural equation modeling was conducted for quantitative analysis of factors related to user adoption of MDSS. Qualitative analysis based on semistructured interviews with physicians was conducted to collect various opinions on MDSS implementation.

    Results: Quantitative analysis revealed that physicians’ expectations regarding ease of use and performance improvement are crucial. Qualitative analysis identified four significant barriers to MDSS implementation: alert fatigue, lack of accuracy, poor user interface design, and lack of customizability.

    Conclusions: This study revealed barriers and facilitators to the implementation of MDSS. The findings can be applied to upgrade MDSS in the future.

    JMIR Med Inform 2020;8(7):e18758

    doi:10.2196/18758

    Keywords

    clinical decision support system; electronic health record; medication safety; Computerized Provider Order Entry (CPOE) 


    Background

    In 2009, based on evidence that electronic health records (EHR) can improve healthcare quality, the US government enacted the Health Information Technology for Economic and Clinical Health (HITECH) Act [1]. Over the past decade, the healthcare industry has experienced a tremendous digital revolution initiated by the government’s efforts to implement EHRs [2,3]. As of 2017, over 90% of general medical and surgical hospitals in the US use certified EHR systems, thus generating an enormous amount of electronic medical information daily [2,4]. Analysis of big data gathered from EHRs can generate real-time evidence that helps end-users take better care of their patients [5]. Clinical decision support systems (CDSS) are a typical example of value provided to EHR users [6]. Such systems intervene in real-time to help users make appropriate decisions based on up-to-date information from EHRs. Medication decision support systems (MDSS), a well-known and frequently used type of CDSS, reduce adverse drug events (ADE), some of the most common causes of iatrogenic injuries in hospitals [7]. ADEs are generally defined as anticipated or unanticipated side effects resulting primarily from medication errors, attributable to human errors. The most common types of medication errors include the use of contraindicated drugs and overdosing [8]. An MDSS checks for problems based on CDSS data and alerts users in advance of potentially preventable errors. However, despite the high adoption rate of EHRs in the US, ADEs are still a significant problem [9]. The situation is similar in South Korea. ADEs have not been reduced dramatically in South Korea, although the adoption rate of EHRs is around 90% as of 2017 [10,11].

    Prior Research

    Efforts have been made to reduce medication errors to minimize the frequency of ADEs. Previous studies have demonstrated that implementing an MDSS in the EHR system improves patient care and overall outcomes by reducing medication errors [12-19]. However, repeated false alerts from MDSSs can decrease healthcare professionals’ productivity by interrupting their workflow [14,20-22]. Furthermore, if doctors are frequently interrupted by false alerts, they are less likely to adopt MDSS recommendations [23].

    Research has shown that physicians override about 90% of drug allergy and high-severity drug interaction warning notifications [20,21,24,25]. Two methods could be adopted to improve MDSS. One is to enhance the precision of the MDSS algorithms to reduce unhelpful notifications. Machine learning techniques are being widely considered to provide personalized, accurate notifications [15]. The other method is to support users by understanding the factors associated with MDSS feasibility and usability, which requires an understanding of what factors users consider important in clinical settings. To date, many studies have explored the effectiveness of MDSS, but only a few have analyzed their feasibility and usability.

    Aim

    This study aimed to analyze factors related to the adoption of MDSS. A mixed-methods research approach was taken to both quantitatively measure factors necessary for the successful implementation of MDSS and to qualitatively gather and reflect on the opinions of end users. The study encompassed the entire process of implementing an MDSS into a comprehensive EHR system and analyzed the relevant barriers and facilitators. Based on the results, some ideas are suggested to support users and upgrade MDSS, resolving issues already well established by previous studies.


    Design

    A mixed research methodology was adopted [26]. Data from a structured survey and 15 in-depth interviews were integrated, and structural equation modeling (SEM) was conducted to yield a quantitative analysis of the factors related to user adoption of MDSS. A qualitative analysis based on semistructured interviews with physicians also collected various opinions about MDSS implementation. To objectively report results, the qualitative analysis followed the Consolidated Criteria for Reporting Qualitative (COREQ) Research Guidelines [27].

    Setting and Participants

    The study was conducted at Seoul National University Bundang Hospital (SNUBH), where the comprehensive, privately developed BESTCare electronic medical record (EMR) has been in use since 2003. The system has been accredited three times as a Health Information Management Systems and Society Analytics EMR Adoption Model Stage 7 since 2010. BESTCare implements a proprietary MDSS concurrently with a prescription drug monitoring program run by the South Korean government; thus, BESTCare users are already familiar with MDSS.

    A taskforce team of 12 attending physicians, two pharmacists, three nurses, and three engineers was formed to improve medication safety. The team decided to introduce a third-party MDSS to BESTCare. In November 2016, the team analyzed MDSS previously released in the market and decided to implement the Medi-Span solution. The task force analyzed mapping codes, mapping contents, and filters, and designed the user interface and overall system architecture (Table 1).

    The taskforce team designed the overall system architecture of the MDSS (Figures 1 and 2) and designed alert screens to display messages efficiently (Figure 3).

    After implementing the MDSS in April 2017, we conducted a structured survey of physicians between May 2017 and October 2017 and employed SEM to analyze the factors facilitating successful implementation. Focused group interviews were also conducted to collect direct opinions from end-users. For the qualitative analysis, study participants were selected through purposive sampling [28], aiming to include participants with in-depth knowledge of the work process involving the EHR system and MDSS. Textbox 1 presents the items included in the semistructured interview questionnaire.

    Table 1. Basic considerations for integration of a commercialized MDSS into the EMR system.
    MappingWhat standard codes should be used to interface Medi-Span with BESTCare?
    ContentsWhat functions should be implemented to improve medication safety?
    • Drug interactions
      • ‎
    • Drug allergies
      • ‎
    • Drug disease contraindications
      • ‎
    • Duplicate therapy
      • ‎
    • Dose screening and drug order
      • ‎
    • Route contraindications
      • ‎
    • Pregnancy/lactation contraindications
      • ‎
    • Gender/age contraindications
      • ‎
    FiltersWhat filters should we integrate? How can we control users’ authorization to override MDSS alerts?
    AlertsHow can we show alerts efficiently?
    User interface/experienceHow can we improve user experience and user interface designs?
    ‎
    Figure 1. MDSS system architecture and configuration. EMR: electronic medical record; HIS: hospital information system.
    View this figure
    ‎
    Figure 2. MDSS function list. API: application programming interface; GPI: generic product identifier.
    View this figure
    ‎
    Figure 3. Screenshot of the MDSS user interface. 1) Override requirements: red alerts indicate that users must view an alert message and select a reason for overriding it, whereas orange alerts indicate that users must confirm the alert. 2) Origin of alerts: BESTCare MDSS, Medi-span CDSS, and South Korean national prescription drug monitoring program. 3) Classification of alerts using icons, allowing users to see the notification easily.
    View this figure

    Semistructured interview questionnaire items.

    Questions:

    • What was your first impression of the MDSS implemented in this hospital?
    • Did you have experience with other MDSSs before?
    • How long did it take for you to get used to the MDSS?
    • Were there any barriers to implementation of the MDSS?
    • What do you think would help to better implement the MDSS?
    • How is the navigation when using the MDSS?
    • Are there any problems with the MDSS that need to be resolved?
    • Do you think the MDSS is customized well for the EHR workflow? Is there anything missing?
    • What features or functions do you want to add to the MDSS?
    • Do you have any recommendations for the MDSS to improve your work experience?
    • Is there anything else you want to mention regarding the MDSS?
    Textbox 1. Semistructured interview questionnaire items.

    Data Collection

    MYK, a registered nurse, conducted the survey and face-to-face semistructured interviews. IYC, a medical doctor, also led the interviews and took notes. Both interviewers received training on qualitative interviews. The interviews lasted 20 to 60 minutes and were recorded in a closed office or conference room. Nobody was present besides the participants and researchers. During the sessions, MYK followed the semistructured interview questionnaire covering topics related to the implementation of the MDSS (Textbox 1). The researchers followed interview guidelines based on previous research and approved by members of the eHealth research team at SNUBH.

    Data Analysis

    For SEM, the survey adopted the technology acceptance model (TAM) and the unified theory of acceptance and use of technology (UTAUT), both of which have been widely adopted to analyze user willingness to accept new technologies [15,29-32]. The models were modified to create a structural equation model optimized for this study Figure 4. Performance expectancy, effort expectancy, and facilitating conditions were expected to have a positive influence on attitude, and attitude was expected to have a positive influence on intention to use. The TAM includes two variables impacting behavioral intentions to use, and the UTAUT includes three behavioral variables and one variable that influences actual use, all of which influence the overall process. Theoretically, facilitating conditions should influence actual use. However, based on previous studies, we hypothesized that facilitating conditions would instead moderate intention to use due to difficulties in measuring actual use. Social influence from senior colleagues was omitted in order to simplify the model, as the study’s focus was on factors related only to user expectations and support from the hospital.

    ‎
    Figure 4. The analytical model used in this study, modified from the technology acceptance model (TAM) and the unified theory of acceptance and use of technology (UTAUT).
    View this figure

    Ethics

    The research protocol was approved by the Institutional Review Board of Human Research of Seoul National University Bundang Hospital (Protocol No. B-1709-420-303).


    Participant Demographics

    Table 2 presents the demographic characteristics of all SEM survey respondents. Of the 80 professionals invited to take the survey, 61 responded. Most were residents who use the EMR and MDSS more actively than any other position on the hospital staff.

    For qualitative analysis, 15 people out of 80 participants were interviewed. Table 3 presents the interviewees’ demographic characteristics.

    A reliability test was performed to confirm the consistency of the survey items for SEM analysis. Cronbach α exceeded .8 for all variables except facilitating conditions. Thus, the survey items were confirmed to be consistent and reliable (Table 4).

    Table 2. Demographic characteristics of SEM survey respondents.
    Categories/itemsNumberPercentage
    Gender


    		Male2236.07

    		Female3963.93
    Age (years)


    		20-291321.31

    		30-394573.77

    		40-4923.28

    		50 and above11.64
    Department


    		Internal/family medicine4268.85

    		Pediatrics813.11

    		Surgery58.20

    		Other69.84
    Length of service (years)


    		<12439.34

    		1-33049.18

    		3-546.56

    		5-1000

    		>1034.92
    Position


    		Professor23.28

    		Fellow46.56

    		Resident5590.16
    Table 3. Demographic characteristics of focused interview participants.
    Categories/itemsNumberPercentage
    Gender


    		Male1173

    		Female427
    Age (years)


    		20-2927

    		30-391280

    		40-49113
    Department


    		Internal/family medicine1386

    		Pediatrics17

    		Surgery17
    Length of service


    		<117

    		1-31279

    		3-517

    		5-1017
    Position


    		Professor320

    		Resident1280
    Table 4. Reliability analysis.
    ConstructNumber of itemsCronbach α
    Performance expectancy3.82
    Effort expectancy3.90
    Attitude2.86
    Facilitating conditions3.70
    Intention to use3.95

    Quantitative Analysis

    Table 5 shows the total number of red and orange alerts presented by the MDSS each month from April 2017 to March 2018. A total of 185,441 red alerts (65.82%) were overridden.

    Table 6 presents the usability test results. The overall mean score was 3.38. Study participants generally agreed with all statements except “I feel confident using Medi-Span,” which resulted in a positive score (above 3) but was not statistically significant.

    Figure 5 presents the overall results of the SEM analysis. For the research model, χ2 was 93.51 (df=60, P<.01), the Tucker-Lewis index (TLI) was 0.971, the comparative fit index (CFI) was 0.987, and the root mean square error of approximation (RMSEA) was 0.067. Because TLI and CFI exceeded 0.9, RMSEA was below 0.1, and the P value of the model was statistically significant, the model was confirmed to be appropriate for analyzing end-user intentions to use the MDSS. The associations between latent variables were positive and statistically significant, confirming the influence of performance expectancy on attitude, effort expectancy on performance expectancy, and attitude on the intention to use.

    Table 5. Number of red and orange alerts presented by the MDSS each month.
    MonthRed alerts (N)Orange alerts (N)
    April 20177893805
    May 2017397929,037
    June 201725,90394,023
    July 201723,86873,986
    August 201727,46874,070
    September 201724,40169,403
    October 201722,72962,991
    November 201725,53672,996
    December 201728,44576,197
    January 201827,72882,671
    February 201837,62468,926
    March 2018325078,162
    Total1,126,724281,720
    Table 6. Usability test results.
    ItemsaMean (95% CI)
    I feel like I use Medi-Span frequently.3.43 (3.23, 3.62)
    Medi-Span is unnecessarily complicated to use.a2.74 (3.08, 3.45)
    Medi-Span is easy to use.3.46 (3.31, 3.61)
    I need technical support to use Medi-Span.a2.70 (2.47, 2.93)
    Medi-Span integrates various functions well.3.41 (3.26, 3.56)
    Medi-Span is not consistent in terms of usability.a2.48 (2.31, 2.63)
    I think most people learn how to use Medi-Span quickly.3.51 (3.34, 3.68)
    It is bothersome to use Medi-Span.a2.59 (2.31, 2.67)
    I feel confident using Medi-Span.2.93 (2.78, 3.08)
    It takes a long time to get used to Medi-Span.b2.61 (2.45, 2.77)
    Total score3.38 (3.27, 3.48)

    aEach item was rated on a 5-point Likert scale, with a score of 3 or higher indicating agreement with the statement (for questions with negative wording, a score of 3 or below indicated a positive response).

    bQuestions with negative wording were reverse-scored to calculate the mean total score.

    ‎
    Figure 5. Results of the SEM model. *P<.001.
    View this figure

    Qualitative Analysis

    Alert Fatigue

    Alert fatigue is a well-known problem associated with MDSS [20]. Participants in this study also mentioned alert fatigue several times.

    It is inconvenient because there are many alerts for drugs commonly used in hematology-oncology, and it is not possible to dismiss the extreme caution alert, which frequently appears in older patients.

    Lack of Accuracy

    Accuracy is an essential factor affecting users’ trust in an MDSS. If false alerts pop up repeatedly, users will tire of the interruption and may fail to take heed when a valid alert is given. Accuracy is also closely related to alert fatigue because poor accuracy results in a higher number of unnecessary alerts. This study’s participants also mentioned accuracy frequently.

    For example, when co-prescribing morphine and clopidogrel, the same message about drug-drug interaction occurs several times, and the alert override has to be selected several times.

    Poor User Interface Design

    South Korean medical staff are accustomed to using English at work. However, poor user interface design can display too much English information on one screen, making it difficult for professionals to see every message in a busy hospital setting. Particularly in emergent situations, poor user interface design presenting excessive and unnecessary English information can be problematic. Therefore, it is crucial to design a user interface that provides essential messages only. In this study, two participants mentioned that the context of English information was difficult to understand quickly.

    It’s hard to understand the alert messages because they’re in English and include a lot of content.

    Lack of Customizability

    Participants highlighted the need for functionality in the MDSS to customize types of alerts according to user preference. To reduce the rate of overrides and alert fatigue, an MDSS must be easily customizable.

    It would be nice to have the ability to set specific drugs and specific doses as a basis for alerts for each department or doctor.


    Principal Findings

    This study employed a mixed-methods approach to analyze barriers and facilitators to the implementation of MDSS. Barriers were identified based on the results of SEM and qualitative analysis, and facilitators were identified based on SEM.

    The quantitative analysis found an average usability rating of 3.38 out of 5, indicating acceptable usability of the MDSS. SEM analysis revealed that effort expectancy had a positive effect on performance expectancy, performance expectancy had a positive effect on attitude, and attitude had a positive effect on the intention to use. Thus, user expectations regarding ease of use may not directly affect their attitude. If users can utilize the system easily, they expect it to result in performance improvement, which in turn affects their attitude toward and intentions to use the system.

    The qualitative analysis identified four significant barriers to implementation of an MDSS: alert fatigue, lack of accuracy, poor user interface design, and lack of customizability. To our knowledge, this is the first study to analyze barriers, facilitators, and usability of MDSS implementation based on a mixed-methods approach.

    Barriers

    Alert Fatigue

    A previous study revealed that medication safety alert fatigue could be reduced through interaction design and clinical role tailoring [33]. The results of our SEM analysis revealed that effort expectancy (ie, user expectations regarding ease of use) affects performance expectancy, which in turn affects intentions to use the system. If the problem of frequent alert fatigue is neglected, the usability of the MDSS will suffer, which will affect user performance expectancy and foster a negative attitude toward the use of the MDSS. In particular, previous studies have shown that the busy working environment of interns or residents can aggravate the adverse effects of alert fatigue [22].

    Poor User Interface Design

    Users clearly want to improve their work performance by using the MDS system. If they can utilize the MDSS to its fullest extent, they can expect to increase their work efficiency and performance. User interface design and experience are crucial to facilitate full utilization. If the system is difficult to use (ie, navigation is unintuitive), doctors will tend to dismiss or ignore messages from the MDSS. Previous studies have shown that user interface design is an essential factor in the successful implementation of a CDSS [34-36]. BESTCare has an integrated interface design, allowing users to easily and intuitively predict the next necessary action. The same interface design was adopted and upgraded to implement a third-party MDSS, helping users to quickly grasp the information presented by MDSS and move on to the next required action. However, the qualitative analysis found that redundant information hindered the user interface of the MDSS. Thus, the volume and layout of the displayed information must be considered in addition to the screen design.

    Lack of Accuracy

    A previous study found that 52.6% of MDSS alerts in outpatient clinics were overridden, 53% of which were appropriate [21]. Another study showed that 73.3% of patient allergy, duplicate drug, and drug interaction alerts were overridden in an inpatient clinic, only about 60% of which were appropriate [20]. Even if only 40% of overrides are considered inappropriate, this can significantly increase the risk of medication errors and potentially leading to ADEs. The override rate was 65.82% in the MDSS evaluated in this study, similar to previous results. Accuracy is related to performance expectancy. If accuracy remains consistently low, users will begin to lose hope of improving performance, resulting in a negative cycle of higher alert overrides.

    Poor Customizability

    BESTCare, the EHR integrated with the Medi-Span based MDSS used in this study, has an alert-related authority control function that meets the standards of the EHR certification program run by the Office of the National Coordinator for Health Information Technology. The professionals who participated in this study’s in-depth interview were dissatisfied with this integrated management system and wanted the ability to customize and adjust the alerts they received. MDS systems are usually introduced to prevent ADEs. Therefore, a centralized, integrated management system is necessary for consistency and stability. However, end-user satisfaction will increase if they can adjust the level of alerts provided without sacrificing this overall stability.

    Facilitators

    Effort Expectancy

    This study’s quantitative analysis found that doctors generally considered the MDSS easy to use. Before development, both hospital staff and developers expressed concern about integrating the new Medi-Span MDSS on top of the two MDSS already integrated into the EHR system because the integration of three MDSS within the EHR could result in an excessive number of alerts. However, dedicated trial and error by the taskforce team ensured the usability of the system. As revealed in the in-depth interview, the users’ wishes regarding MDSS usability can never be fully satisfied. However, the taskforce team’s activities to improve usability acted as important facilitators for the successful introduction of the MDSS.

    Performance Expectancy

    According to this study’s SEM analysis, end-user effort expectancy had a positive effect on their expectations of performance improvement. MDSS platforms must provide users with feedback on their actions in response to alerts and performance improvement outcomes in order to reduce overrides for valid alerts. Previous studies have noted that gaining user trust is crucial for the proper implementation and maintenance of a new system [37-39]. Clinical indicators regarding performance improvement and regular result reporting may be an excellent way to promote the use of the MDSS and gain trust. For example, public disclosure of antibiotic use rates effectively lowered the use of antibiotics for upper respiratory infections in South Korea [40]. Likewise, public disclosure about ADEs prevented by using the MDSS may help reduce alert override rates. Another option is to create a method by which users can provide feedback on false alerts. If doctors can provide feedback about false alerts instead of using the MDSS passively, the system can be dynamically upgraded to gain trust.

    Limitations and Future Research

    This study’s main limitation is that the system was implemented in only one hospital. External validation in other hospitals is needed to help generalize the study results. Nevertheless, this research demonstrated the effects of interaction between user expectations regarding ease of use and performance improvement on their attitude toward using an MDSS, which can inform the practices of system designers and policymakers in charge of MDSS development.

    Conclusion

    This study revealed barriers and facilitators to the implementation of MDSS. The study’s findings can be used as a reference to upgrade MDSSs effectively. Further studies are needed to evaluate specific ways to gain MDSS users’ trust.

    Conflicts of Interest

    There are no conflicts of interest that could influence the findings of this research.

    1. Blumenthal D. Stimulating the Adoption of Health Information Technology. N Engl J Med 2009 Apr 09;360(15):1477-1479. [CrossRef]
    2. Adler-Milstein J, Jha AK. HITECH Act Drove Large Gains In Hospital Electronic Health Record Adoption. Health Affairs 2017 Aug;36(8):1416-1422. [CrossRef]
    3. GOLD M, McLAUGHLIN C. Assessing HITECH Implementation and Lessons: 5 Years Later. The Milbank Quarterly 2016 Sep 13;94(3):654-687. [CrossRef]
    4. Percent of Specialty Hospitals that Possess Certified Health IT. The Office of the National Coordinator for Health Information Technology.   URL: https://dashboard.healthit.gov/quickstats/pages/specialty-hospital-ehr-adoption.php [accessed 2020-02-13]
    5. Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2014 Feb 7;2(1). [CrossRef]
    6. Wasylewicz A, Scheepers-Hoeks A. Clinical Decision Support Systems. Kubben P, Dumontier M, Dekker A. eds. Fundamentals of Clinical Data Science. Cham (CH): : Springer 2018 Dec 22:2018. [CrossRef] [Medline]
    7. Kohn L, Corrigan J, Donaldson M. To err is human: building a safer health system. National academy press Washington, DC 2000 Jan 01:2000. [CrossRef] [Medline]
    8. Zafar A, Hickner J, Pace W. An adverse drug event and medication error reporting system for ambulatory care (MEADERS). 2008 Nov 06 Presented at: AMIA Annu Symp Proc ;?43; 2008; Washinton p. 839.
    9. Harris Y, Hu DJ, Lee C, Mistry M, York A, Johnson TK. Advancing Medication Safety: Establishing a National Action Plan for Adverse Drug Event Prevention. Jt Comm J Qual Patient Saf 2015 Aug;41(8):351-360. [CrossRef] [Medline]
    10. Park Y, Han D. Current Status of Electronic Medical Record Systems in Hospitals and Clinics in Korea. Healthc Inform Res 2017 Jul;23(3):189-198 [FREE Full text] [CrossRef] [Medline]
    11. Cho M, Kang DY, Kang H. Adverse drug reactions. J Korean Med Assoc 2019;62(9):472. [CrossRef]
    12. Jia P, Zhang L, Chen J, Zhao P, Zhang M. The Effects of Clinical Decision Support Systems on Medication Safety: An Overview. PLoS ONE 2016 Dec 15;11(12):e0167683. [CrossRef]
    13. Salmasian H, Tran TH, Chase HS, Friedman C. Medication-indication knowledge bases: a systematic review and critical appraisal. J Am Med Inform Assoc 2015 Sep 02:ocv129. [CrossRef]
    14. Payne T, Hines L, Chan R, Hartman S, Kapusnik-Uner J, Russ AL, et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015 Nov;22(6):1243-1250. [CrossRef] [Medline]
    15. Esmaeilzadeh P, Sambasivan M, Kumar N, Nezakati H. Adoption of clinical decision support systems in a developing country: Antecedents and outcomes of physician's threat to perceived professional autonomy. Int J Med Inform 2015 Aug;84(8):548-560. [CrossRef] [Medline]
    16. Felkey BG, Fox BI. Consider the benefits of a fully integrated medication use process. Hosp Pharm 2014 Jan;49(1):101-102 [FREE Full text] [CrossRef] [Medline]
    17. Tawadrous D, Shariff SZ, Haynes RB, Iansavichus AV, Jain AK, Garg AX. Use of clinical decision support systems for kidney-related drug prescribing: a systematic review. Am J Kidney Dis 2011 Dec;58(6):903-914. [CrossRef] [Medline]
    18. Nieuwlaat R, Connolly SJ, Mackay JA, Weise-Kelly L, Navarro T, Wilczynski NL, CCDSS Systematic Review Team. Computerized clinical decision support systems for therapeutic drug monitoring and dosing: a decision-maker-researcher partnership systematic review. Implement Sci 2011 Aug 03;6(1):90 [FREE Full text] [CrossRef] [Medline]
    19. Hemens BJ, Holbrook A, Tonkin M, Mackay JA, Weise-Kelly L, Navarro T, CCDSS Systematic Review Team. Computerized clinical decision support systems for drug prescribing and management: a decision-maker-researcher partnership systematic review. Implement Sci 2011 Aug 03;6(1):89 [FREE Full text] [CrossRef] [Medline]
    20. Nanji K, Seger D, Slight S, Amato MG, Beeler PE, Her QL, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018 May 01;25(5):476-481. [CrossRef] [Medline]
    21. Nanji KC, Slight SP, Seger DL, Cho I, Fiskio JM, Redden LM, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014 May 01;21(3):487-491 [FREE Full text] [CrossRef] [Medline]
    22. Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R, with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017 Apr 10;17(1):36 [FREE Full text] [CrossRef] [Medline]
    23. Tsai C, Wang S, Hsu M, Li Y. Do false positive alerts in naïve clinical decision support system lead to false adoption by physicians? A randomized controlled trial. Comput Methods Programs Biomed 2016 Aug;132:83-91. [CrossRef] [Medline]
    24. Isaac T, Weissman JS, Davis RB, Massagli M, Cyrulik A, Sands DZ, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009 Feb 09;169(3):305-311. [CrossRef] [Medline]
    25. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003 Nov 24;163(21):2625-2631. [CrossRef] [Medline]
    26. Slootman M. A Mixed-Methods Approach. In: Slootman M. ed. Ethnic Identity, Social Mobility and the Role of Soulmates. Cham: : Springer International Publishing 2018. 41?. Switzerland: Springer, Cham; Sep 22, 2018:57.
    27. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007 Dec 16;19(6):349-357. [CrossRef] [Medline]
    28. Robinson OC. Sampling in Interview-Based Qualitative Research: A Theoretical and Practical Guide. Qualitative Research in Psychology 2013 Nov 18;11(1):25-41. [CrossRef]
    29. Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 1989 Sep;13(3):319. [CrossRef]
    30. Venkatesh, Morris, Davis, Davis. User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly 2003;27(3):425. [CrossRef]
    31. Holden RJ, Karsh B. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010 Feb;43(1):159-172 [FREE Full text] [CrossRef] [Medline]
    32. Gagnon M, Desmartis M, Labrecque M, Car J, Pagliari C, Pluye P, et al. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J Med Syst 2012 Feb 30;36(1):241-277 [FREE Full text] [CrossRef] [Medline]
    33. Hussain M, Reynolds T, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc 2019 Oct 01;26(10):1141-1149. [CrossRef] [Medline]
    34. Campion TR, Waitman LR, Lorenzi NM, May AK, Gadd CS. Barriers and facilitators to the use of computer-based intensive insulin therapy. Int J Med Inform 2011 Dec;80(12):863-871 [FREE Full text] [CrossRef] [Medline]
    35. Kanstrup A, Christiansen M, Nøhr C. Four principles for user interface design of computerised clinical decision support systems. Stud Health Technol Inform 2011;166:65-73. [Medline]
    36. Yuan MJ, Finley GM, Long J, Mills C, Johnson RK. Evaluation of user interface and workflow design of a bedside nursing clinical decision support system. Interact J Med Res 2013 Jan 31;2(1):e4 [FREE Full text] [CrossRef] [Medline]
    37. Koskela T, Sandström S, Mäkinen J, Liira H. User perspectives on an electronic decision-support tool performing comprehensive medication reviews - a focus group study with physicians and nurses. BMC Med Inform Decis Mak 2016 Jan 22;16(1):6 [FREE Full text] [CrossRef] [Medline]
    38. Alexander GL. Issues of trust and ethics in computerized clinical decision support systems. Nurs Adm Q 2006;30(1):21-29. [CrossRef] [Medline]
    39. Amoedo A, Martnez-Costa MDP, Moreno E. An analysis of the communication strategies of Spanish commercial music networks on the web: http://los40.com, http://los40principales.com, http://cadena100.es, http://europafm.es and http://kissfm.es. radio journal: international studies in 2009 Feb 01;6(1):5-20 [FREE Full text] [CrossRef]
    40. Yun JM, Shin DW, Hwang S, Cho J, Nam YS, Kim JH, et al. Effect of public disclosure on antibiotic prescription rate for upper respiratory tract infections. JAMA Intern Med 2015 Mar 01;175(3):445-447. [CrossRef] [Medline]

    ‎
    ADE: adverse drug event
    API: application programming interface
    ATT: attitude
    AU: actual use
    CFI: comparative fit index
    COREQ: Consolidated Criteria for Reporting Qualitative Research Guidelines
    EE: effort expectancy
    EHR: electronic health record
    EMR: electronic medical record
    FC: facilitating conditions
    GPI: generic product identifier
    HIS: hospital information system
    ITU: intention to use
    MDSS: medication decision support system
    PE: performance expectancy
    PEOU: perceived ease of use
    PU: perceived usefulness
    RMSEA: root mean square error of approximation
    SEM: structural equation modeling
    SI: social influence
    SNUBH: Seoul National University Bundang Hospital
    TAM: technology acceptance model
    TLI: Tucker-Lewis index
    UTAUT: unified theory of acceptance and use of technology

    Edited by G Eysenbach; submitted 16.03.20; peer-reviewed by J Aarts, S Sarbadhikari; comments to author 23.04.20; revised version received 02.05.20; accepted 14.05.20; published 22.07.20

    Copyright

    ©Se Young Jung, Hee Hwang, Keehyuck Lee, Ho-Young Lee, Eunhye Kim, Miyoung Kim, In Young Cho. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 22.07.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.