This study received approval from the University of Alberta Health Research Ethics Board (Pro00083538). A waiver of informed consent was also approved as part of our study by the Health Research Ethics Board.

The study was conducted in the Comprehensive Academic Outpatient Center at the University of Alberta Hospital, which provides care for patients with IBD in the Greater Edmonton region as well as rural and remote communities across Alberta, Canada. It also serves a small number of patients with IBD from Saskatchewan, Northwest Territories, and British Columbia.

The clinic’s preexisting system was an enterprise EMR based on the 2014 version of Epic EMR (Epic Systems), which was being used for outpatient medical care in Edmonton, Alberta. This system was customized and branded locally as eCLINICIAN. Medication lists, allergies, and health problems are recorded and shared between users as part of clinical documentation, order entry, and planning. The system was implemented for gastroenterology outpatient care in March 2014.

As Epic is a general-purpose EMR, it includes built-in CDS functionality. For example,this includes generic functionality, such as alerting users when duplicate orders exist. More specialty-specific CDS features are often customized at the request and guidance of end users.

Functionality can be administered through a number of tools, including those referred to by Epic as “Flowsheets” (documentation tables), “Best Practice Advisories (BPAs)” (alerts) [17], and “SmartSets” (ie, grouping of orders and clinical content) [18].

These tools, particularly BPAs and SmartSets, are clinical data and test result driven; they can be triggered by unique combinations of provider characteristics, patient demographics, test results, clinical problems, as well as current and requested medications.

The system interruption and intervention uses BPA appearing in the clinician’s navigator workflow. The BPA is triggered by the existence of IBD in the patient problem list or visit diagnosis fields. The BPA (Figure 1) prompts the clinician to complete clinical symptom indices—modified Harvey Bradshaw Index (mHBI) [19] for Crohn disease or partial Mayo (pMayo) score [20] for ulcerative colitis—for the encounter. If the score is indicative of a disease flare, the BPA instructs the user to activate a corresponding SmartSet.

The SmartSet offers ordering and printing of appropriate lab panels, stool cultures, and other investigations, including imaging, procedures, and medication prescriptions. All recommendations were designed to be consistent with established IBD care guidelines and the flare protocol for the clinic. For example, during a flare encounter, the IBD flare lab panel and fecal calprotectin (FCP) tests are automatically selected for ordering (they can still be deselected by the provider). A snapshot of the SmartSet portion of the CDSS is shown in Figure 2.

The study used a pre- and postimplementation interrupted time series (ITS) design, the interruption being the enhanced CDSS used within the EMR. Each data point represented 1 month of clinical encounters. For each intervention period, there was a total of 18 data points, 9 before and 9 after the intervention. Multimedia Appendix 1 presents an elaboration on the rationale for using an ITS design.

Physicians at the participating clinic were not guaranteed to have outpatient clinics on a weekly basis due to their service rotation; therefore, it was decided to aggregate the data points by month instead of by week. This avoided the potential week-to-week variation and ensured an adequate number of individual patient encounters (IBD flares) for each data point.

The Quality Criteria for ITS Designs checklist was used in the study design and assessment of appropriateness [21], and the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) guidelines were used for health informatics evaluations [22,23].

All IBD care providers at the university-based outpatient clinic were included in the study and invited to use the CDSS, including 7 IBD specialist clinicians, 1 IBD nurse practitioner, and 4 IBD specialist nurses. The term “IBD practitioner” will be used to collectively refer to IBD specialists and IBD nurse practitioners.

To be included in the data set, patients had to be under the care of the IBD providers; aged ≥18 years; and diagnosed with either Crohn disease or ulcerative colitis confirmed by imaging, pathology, or endoscopy report. They also had to be experiencing a flare of the disease during the included encounter, as defined by clinical scores (mHBI >5; pMayo >2) or noted symptoms in combination with physician judgment. Only initial encounters in a flare episode spanning multiple encounters were included.

The intervention was implemented and evaluated in 2 continuous periods (Figure 3). First, a pilot version was trialed by IBD nurses (Implementation Period 1), and then, the polished version was implemented across all providers in the division (ie, clinicians, nurse practitioners, and IBD nurses) as Implementation Period 2. The pilot version was trialed beginning in September 2017 and included the following 3 SmartSets available within the BPA, corresponding to different positions along the care path of a patient with flaring IBD: suspected flare, 2 to 4 weeks into the flare, and 16 weeks’ postflare assessments. Feedback was gathered informally from providers (Multimedia Appendix 2) to inform further improvement to the CDSS.

After collecting feedback from the pilot, further changes were made to the CDSS. Aside from minor modifications to update included lab tests, the most significant change was the consolidation of the 3 separate SmartSets into 1, targeting the “suspected flare,” the first step in the care pathway. The activation of the BPA in the initial CDSS was entirely manual and relied on the provider entering a specific visit diagnosis. However, in the full version, the BPA was set to automatically trigger based on the presence of an IBD diagnosis in the patient’s problem list. This change was expected to improve the adoption and ease of use of the SmartSet for flare encounters.

The full implementation of the CDSS began on October 10, 2018. An instructional memo with paper-based workflow and educational material was sent to each provider (Multimedia Appendix 3). Over the course of 1 month, each participant was given the opportunity to ask questions about using the system and access to use the system in the sandbox environment. A demonstration of the system was also presented at weekly clinical rounds, with an opportunity to ask questions.

Process indicators were used to measure the proportion of adherent IBD practitioner flare encounters. These indicators include completion of clinical scores (mHBI or pMayo); laboratory testing, such as standard lab panel, FCP, stool cultures, and Clostridium difficile toxin (only if diarrhea is present); and of vitamin D or calcium in conjunction with corticosteroid prescription, patient information given and documented, and modification of maintenance therapy. A secondary outcome was the adoption or acceptance of the CDSS measured by application rate (ratio of CDSS uses to CDSS available for activation).

Potential encounters in the pre- and postintervention periods were initially identified by querying the eCLINICIAN EMR database for encounters with the included IBD providers, where patients had documentation of IBD in their problem list or diagnosis field (International Classification of Diseases coding). A sampling method was used to exclude encounters with specific reasons for visit deemed unlikely to constitute a flare based on exploratory analysis of the data set. Examples of excluded reasons for the visit included “medication refill,” “medical insurance coverage,” and “review results” (a more detailed description of the sampling method is available in a previous publication [10]). Encounters were then screened manually for inclusion and exclusion eligibility by one of the authors (RTS) and a research assistant.

Data for primary outcome measures were also queried and extracted from the EMR database, in collaboration with the eCLINICIAN reporting team in Alberta Health Services (AHS). The various database codes and IDs as well as the final SQL queries used to extract data are included in the Multimedia Appendix 4.

Descriptive statistics were calculated to determine patient characteristics, with data presented as counts and proportions for categorical variables, mean (SD) values for normally distributed continuous variables, and median (IQR) values for nonnormally distributed continuous variables. Proportions were compared by using the Pearson χ² test [24].

A segmented regression analysis was performed for each primary outcome variable to determine the level and slope in the preintervention period as well as the change in level and slope in the postintervention period regarding the mean percentage of adherent encounters [25]. Autocorrelation in the residuals was tested using the Durbin-Watson test.

Data analysis was performed using IBM SPSS Statistics (version 23; IBM Corp) and R 3.5.1 (RStudio Inc) [26]. A 95% CI was used in all analyses unless otherwise specified.

The sample size was first calculated for pre- and postimplementation cohorts based on logistic regression (Multimedia Appendix 5). With a power of 0.80 and a type I error set to 5%, the sample size required was approximately 634 for small effects and 145 for medium effects [27]. This assumes equal sample sizes (N) in the comparison groups and an initial proportion of adherence to each guideline component of approximately 70%, chosen based on a recent study by Jackson et al [11]. The sample size was calculated using G*Power 3.2.9.2 [28].

There is no standard method for determining power in time series analyses. However, a simulation-based power calculation displayed that with N=16 (8 data points in the preintervention period and 8 data points in the postintervention period), there is a 70% chance to detect an effect size of 0.5 or more, and over 90% chance to detect an effect size of 1 or more, at an alpha level of .05 [29]. It is also generally recommended in the literature to have over 100 observations per data point [25,30].

Figure 4 shows the study’s flow diagram. The complete, extracted data set includes 31,726 encounters from January 1, 2017, to June 30, 2019. When considering only clinic visits (7655), orders (16,485), and telephone (5220) encounter types, the data set totals 29,360 (92.5%) encounters. There was an average of 998 encounters per month, with a minimum of 735 (December 2018) and a maximum of 1202 (May 2017) encounters. Of note, there is an overlap between both implementation periods (Figure 3), and thereby, a number of flare encounters appear in both analyses.

From September 2017 to June 2019, the CDSS was activated a total of 214 times across 214 encounters with 207 patients. Of these, 16 encounters were excluded from analysis due to, upon review, not being used appropriately for a flare or suspected flare encounter with a patient with IBD. This left 198 encounters, which are detailed in Table 1. More detailed demographics of providers using the system are included in Multimedia Appendix 6.

Implementation Period 1 included data from January 2017 to June 2018 (18 months), where September 2017 and beyond were labelled as the active intervention months (postintervention). Of the total 623 confirmed flare encounters, 502 occurred during Implementation Period 1 (Figure 3). Table 2 compares outcome measures before and after the intervention using chi-square tests. Notably, there was a substantial increase in the proportion of flare encounters with completed clinical scores from 3.5% (8/228) to 24.1% (66/274) post intervention. There was also an increase in the proportion of flare encounters with FCP ordered, from 16.7% (38/228) to 27% (74/274).

ITS analysis was done for outcomes that were significant in the before-and-after analyses (Figure 5). For clinical score completion rates, there was no slope change (estimated β −1.22, 95% CI −4.44 to 2.01; P=.43), but there was a level increase (estimated β 19.0, 95% CI 2.39-35.60; P=.03). For calprotectin testing, there was no slope change (estimated β −2.45, 95% CI −6.21 to 1.32; P=.19) or level change (estimated β 14.77, 95% CI −4.63 to 34.17; P=.13).

Implementation Period 2 included data from January 2018 to June 2019 (18 months), where October 2018 and beyond were postintervention months. Of the total 623 confirmed flare encounters, 492 occurred during Implementation Period 2 (Figure 3). Table 3 compares outcome measures before and after the intervention using chi-square tests. There were increases in the proportion of flare encounters with completed flare labs (109/229, 47.6% to 173/263, 65.8%), CRP ordered (147/229, 64.2% to 207/263, 78.7%), calprotectin ordered (64/229, 27.9% to 98/263, 37.3%), and stool cultures ordered (125/229, 54.6% to 176/263, 66.9%).

The ITS analysis for significant outcomes is shown in Figure 6, and accompanying β values for slope change and level change with 95% CIs are shown in Table 4. For Period 2, there were no slope or level increases that reached significance at P=.05, although CRP testing and stool culture testing would be significant for a level increase at P=.10.

In this study, we evaluated the effectiveness of a CDSS that aimed to standardize protocols for patients with IBD experiencing an acute disease flare. An increase in several practices was demonstrated following the CDSS implementation, including increased FCP use. Completion of clinical scores also increased during Implementation Period 1 (before-and-after analysis and ITS analysis) and remained increased throughout Implementation Period 2.

We did not reach significance in slope changes or level changes in any ITS analysis in Period 2. This could be due to the sample size, which may also account for the large variance seen in some data points. There were, however, some encouraging upward trends in flare lab testing, particularly CRP (P<.10 in the ITS analysis) and stool cultures.

In characterizing the adoption of this CDSS by the application rate, an interesting finding was that the CDSS was used more by IBD nurses compared to nurse practitioners. This could represent the nurses’ increased experience with the CDSS from the pilot phase and our CDSS focus on decisions related to patients experiencing a disease flare. In the University of Alberta clinic, patients are instructed to call the IBD nurse flare line if they experience changes in symptoms, and so nurses are often the first point of contact in the flare clinical pathway. This is supported by our data showing flare encounters are primarily telephone encounters. Other research has shown that flares are unlikely to coincide with scheduled clinic appointments, which aligns with the current uptake in remote monitoring and rapid access clinics [31-33].

Our observed CDSS use by specialized IBD nurses is in contrast to several other studies that have demonstrated that nurses are less likely to use CDSSs when making decisions about care they are experienced and confident in delivering, especially in the case of telephone triage decisions [34-36]. Our results could be a product of the integration of the nurses’ feedback after the pilot phase, a strategy that may have increased the utility of the CDSS for nurses. This highlights recommendations from other research that emphasize the importance of engaging all stakeholders but especially end users in the CDSS design [37,38].

There are several limitations to this research. Although the ITS design allows for better characterization of temporal changes compared to before-and-after analyses, it is still possible that other changes, such as clinic structure and release or dissemination of guidelines, could have led to the changes observed. However, apart from the intervention activation and the released memo and instructions for use that were disseminated, to our knowledge, there were no other educational campaigns, institutional changes, or major publications promoting the specific care guidelines investigated by the study. There were subtle changes in staff, for example, the joining of a new IBD physician and the leaving of another. However, there were no changes in IBD nurse staff, who were the primary users of the CDSS.

In contrast to the advantage of our 2-phased design regarding the opportunity for feedback from nurses, the design may have hindered our ability to demonstrate change. As we used the same group of IBD nurses in the pilot (Phase 1) and implementation (Phase 2) periods, our baseline use prior to the beginning of Phase 2 had already started. This may have accelerated the observed uptake speed of the CDSS by practitioners and could have also led to an underestimation of the changes before and after Implementation Phase 2.

Sample size is another limitation. In an ITS analysis, it is recommended to have a minimum of 16 data points and 100 observations per data point [25,29,30]. Although we met the data point requirement, the number of flares per month was consistently under 50. Future studies should aim to include more data points, which may require multisite participation. Unfortunately, at the time of this study, the EMR software was only deployed at a single site.

We only captured data from orders that were tied to the encounter. If a decision was made to not order labs for any reason (eg, they were recently completed), they would not be captured by our extraction. As a consequence, estimates of protocol adherence could be deflated.

Finally, it is important to note that for process measures that depend on manual data entry, such as clinical score completion, this research method can only determine whether a process was documented as completed but not necessarily whether it was actually completed. This may have resulted in underestimates of protocol adherence.

The currently available CDSS in this study was limited in its ability to support complex multiprovider pathways and tie together multiple visits along a pathway. More advanced CDSS workflows should be investigated in future versions of the CDSS software and evaluated for effectiveness.

Triggering logic for CDSSs should also be precisely targeted. For example, a CDSS should determine whether a patient has had a test done within a certain time span, and if not, prompt the user to order it. The reverse should also be possible; if a test has been recently ordered (eg, Clostridium difficile, which can only be tested once every 2 weeks), the CDSS could automatically deselect or prompt the user to remove this order to save downstream resources. This was not possible with the resources available in our CDSS environment.

In extracting data for analysis, a significant challenge was identifying flare encounters based on EMR data. The problem stems from a lack of discrete data identifying patients with active diseases (clinical scores were not regularly documented as discrete data). Future research should seek to develop a case definition for disease flare through administrative provincial data sets. This could include quantitative metrics, such as CRP and FCP, that predict the likelihood of flare, but it could also include the integration of a case-finding algorithm that uses natural language processing to parse clinical notes. This strategy has been explored in several other diseases and has been shown to significantly improve case detection [39]. Some work has been done in IBD to identify phenotypic information from clinic notes using natural language processing [40].

The methodology used in this research should be expanded to investigate the effects of improved versions of CDSS for IBD on other community clinics and nonacademic practices throughout Alberta. Cluster-randomized designs or stepped-wedge designs could be explored since multiple clinics could be available for randomization.

This study did not investigate the impact on patient outcomes, which would require a longer follow-up period (ideally 2 or more years). Nonetheless, long-term patient outcomes for the CDSS are of great importance [9] and should be explored in the future.

Through our study, we designed and implemented, in 2 phases, a CDSS for IBD disease flare embedded in existing EMR software and evaluated the impact of the CDSS on provider adoption of clinical guidelines and local best practices. We have shown moderate adoption and acceptance of this system by providers, particularly by IBD nurses, as measured by the system application rate. Findings from the first phase support the hypothesis that the CDSS improved the use of FCP and the documentation of clinical scores. Findings from the second phase support further improvement in ordering flare lab panels, CRP, and stool cultures, as shown in before-and-after analysis and multivariate analysis. In addition, potential improvements in workflow integration were identified through qualitative questionnaires and feedback forms; areas for future research have also been established.