In 2009, based on evidence that electronic health records (EHR) can improve healthcare quality, the US government enacted the Health Information Technology for Economic and Clinical Health (HITECH) Act [1]. Over the past decade, the healthcare industry has experienced a tremendous digital revolution initiated by the government’s efforts to implement EHRs [2,3]. As of 2017, over 90% of general medical and surgical hospitals in the US use certified EHR systems, thus generating an enormous amount of electronic medical information daily [2,4]. Analysis of big data gathered from EHRs can generate real-time evidence that helps end-users take better care of their patients [5]. Clinical decision support systems (CDSS) are a typical example of value provided to EHR users [6]. Such systems intervene in real-time to help users make appropriate decisions based on up-to-date information from EHRs. Medication decision support systems (MDSS), a well-known and frequently used type of CDSS, reduce adverse drug events (ADE), some of the most common causes of iatrogenic injuries in hospitals [7]. ADEs are generally defined as anticipated or unanticipated side effects resulting primarily from medication errors, attributable to human errors. The most common types of medication errors include the use of contraindicated drugs and overdosing [8]. An MDSS checks for problems based on CDSS data and alerts users in advance of potentially preventable errors. However, despite the high adoption rate of EHRs in the US, ADEs are still a significant problem [9]. The situation is similar in South Korea. ADEs have not been reduced dramatically in South Korea, although the adoption rate of EHRs is around 90% as of 2017 [10,11].

Efforts have been made to reduce medication errors to minimize the frequency of ADEs. Previous studies have demonstrated that implementing an MDSS in the EHR system improves patient care and overall outcomes by reducing medication errors [12-19]. However, repeated false alerts from MDSSs can decrease healthcare professionals’ productivity by interrupting their workflow [14,20-22]. Furthermore, if doctors are frequently interrupted by false alerts, they are less likely to adopt MDSS recommendations [23].

Research has shown that physicians override about 90% of drug allergy and high-severity drug interaction warning notifications [20,21,24,25]. Two methods could be adopted to improve MDSS. One is to enhance the precision of the MDSS algorithms to reduce unhelpful notifications. Machine learning techniques are being widely considered to provide personalized, accurate notifications [15]. The other method is to support users by understanding the factors associated with MDSS feasibility and usability, which requires an understanding of what factors users consider important in clinical settings. To date, many studies have explored the effectiveness of MDSS, but only a few have analyzed their feasibility and usability.

This study aimed to analyze factors related to the adoption of MDSS. A mixed-methods research approach was taken to both quantitatively measure factors necessary for the successful implementation of MDSS and to qualitatively gather and reflect on the opinions of end users. The study encompassed the entire process of implementing an MDSS into a comprehensive EHR system and analyzed the relevant barriers and facilitators. Based on the results, some ideas are suggested to support users and upgrade MDSS, resolving issues already well established by previous studies.

A mixed research methodology was adopted [26]. Data from a structured survey and 15 in-depth interviews were integrated, and structural equation modeling (SEM) was conducted to yield a quantitative analysis of the factors related to user adoption of MDSS. A qualitative analysis based on semistructured interviews with physicians also collected various opinions about MDSS implementation. To objectively report results, the qualitative analysis followed the Consolidated Criteria for Reporting Qualitative (COREQ) Research Guidelines [27].

The study was conducted at Seoul National University Bundang Hospital (SNUBH), where the comprehensive, privately developed BESTCare electronic medical record (EMR) has been in use since 2003. The system has been accredited three times as a Health Information Management Systems and Society Analytics EMR Adoption Model Stage 7 since 2010. BESTCare implements a proprietary MDSS concurrently with a prescription drug monitoring program run by the South Korean government; thus, BESTCare users are already familiar with MDSS.

A taskforce team of 12 attending physicians, two pharmacists, three nurses, and three engineers was formed to improve medication safety. The team decided to introduce a third-party MDSS to BESTCare. In November 2016, the team analyzed MDSS previously released in the market and decided to implement the Medi-Span solution. The task force analyzed mapping codes, mapping contents, and filters, and designed the user interface and overall system architecture (Table 1).

The taskforce team designed the overall system architecture of the MDSS (Figures 1 and 2) and designed alert screens to display messages efficiently (Figure 3).

After implementing the MDSS in April 2017, we conducted a structured survey of physicians between May 2017 and October 2017 and employed SEM to analyze the factors facilitating successful implementation. Focused group interviews were also conducted to collect direct opinions from end-users. For the qualitative analysis, study participants were selected through purposive sampling [28], aiming to include participants with in-depth knowledge of the work process involving the EHR system and MDSS. Textbox 1 presents the items included in the semistructured interview questionnaire.

Questions:

MYK, a registered nurse, conducted the survey and face-to-face semistructured interviews. IYC, a medical doctor, also led the interviews and took notes. Both interviewers received training on qualitative interviews. The interviews lasted 20 to 60 minutes and were recorded in a closed office or conference room. Nobody was present besides the participants and researchers. During the sessions, MYK followed the semistructured interview questionnaire covering topics related to the implementation of the MDSS (Textbox 1). The researchers followed interview guidelines based on previous research and approved by members of the eHealth research team at SNUBH.

For SEM, the survey adopted the technology acceptance model (TAM) and the unified theory of acceptance and use of technology (UTAUT), both of which have been widely adopted to analyze user willingness to accept new technologies [15,29-32]. The models were modified to create a structural equation model optimized for this study Figure 4. Performance expectancy, effort expectancy, and facilitating conditions were expected to have a positive influence on attitude, and attitude was expected to have a positive influence on intention to use. The TAM includes two variables impacting behavioral intentions to use, and the UTAUT includes three behavioral variables and one variable that influences actual use, all of which influence the overall process. Theoretically, facilitating conditions should influence actual use. However, based on previous studies, we hypothesized that facilitating conditions would instead moderate intention to use due to difficulties in measuring actual use. Social influence from senior colleagues was omitted in order to simplify the model, as the study’s focus was on factors related only to user expectations and support from the hospital.

The research protocol was approved by the Institutional Review Board of Human Research of Seoul National University Bundang Hospital (Protocol No. B-1709-420-303).

Table 2 presents the demographic characteristics of all SEM survey respondents. Of the 80 professionals invited to take the survey, 61 responded. Most were residents who use the EMR and MDSS more actively than any other position on the hospital staff.

For qualitative analysis, 15 people out of 80 participants were interviewed. Table 3 presents the interviewees’ demographic characteristics.

A reliability test was performed to confirm the consistency of the survey items for SEM analysis. Cronbach α exceeded .8 for all variables except facilitating conditions. Thus, the survey items were confirmed to be consistent and reliable (Table 4).

Table 5 shows the total number of red and orange alerts presented by the MDSS each month from April 2017 to March 2018. A total of 185,441 red alerts (65.82%) were overridden.

Table 6 presents the usability test results. The overall mean score was 3.38. Study participants generally agreed with all statements except “I feel confident using Medi-Span,” which resulted in a positive score (above 3) but was not statistically significant.

Figure 5 presents the overall results of the SEM analysis. For the research model, χ² was 93.51 (df=60, P<.01), the Tucker-Lewis index (TLI) was 0.971, the comparative fit index (CFI) was 0.987, and the root mean square error of approximation (RMSEA) was 0.067. Because TLI and CFI exceeded 0.9, RMSEA was below 0.1, and the P value of the model was statistically significant, the model was confirmed to be appropriate for analyzing end-user intentions to use the MDSS. The associations between latent variables were positive and statistically significant, confirming the influence of performance expectancy on attitude, effort expectancy on performance expectancy, and attitude on the intention to use.