Following user-centered design (UCD) principles [40], a multidisciplinary team including 5 physicians, 2 pharmacists, and several EHR builders, with expertise in implementation science, informatics, behavioral economics, and health services research, designed a fully automated, EHR-embedded, interruptive, provider-facing CDS. The intervention was beta-tested by several ED clinicians in a practice setting for 6 months before the systemwide rollout. CDS did not interface with any technologies beyond the EHR and fired within typical workflow to recommend and facilitate the addition of a naloxone prescription before the e-signing of any high-risk opioid analgesic prescription order (Multimedia Appendix 1). High-risk criteria were adapted from the 2016 CDC guidelines for chronic pain and defined as any opioid prescription (1) resulting in >90 morphine milligram equivalents per day, (2) for a patient with an opioid use disorder (OUD) diagnosis, or (3) prior opioid OD [41]. CDS searched for Systematized Nomenclature of Medicine Clinical Terms in the “Problem List Diagnosis,” “Encounter Diagnosis,” and “Hospital Problem Diagnosis” lists. Alerts were suppressed if the patient had an active naloxone prescription or if the patient was discharged to hospice, given patients on end-of-life care are excluded from CDC guidelines [41]. Naloxone prescriptions stayed on the patient’s medication list for 1 year.

Key design principles followed were that CDS be intuitive, trigger only when indicated, and default to a preselected naloxone order that was the least expensive option in the health care system’s retail pharmacies [24,34,35]. Any provider with prescribing privileges could encounter the alert. Default selection was chosen to decrease work (clicking “Accept” added naloxone to the existing order) and because “opt-out” approaches increase the uptake of target clinical behaviors [42-45].

Accepting CDS was the path of least resistance. However, consistent with nudge theory, clinicians could bypass CDS by (1) selecting “Do Not Order” then “Accept” (2 clicks); (2) selecting prepopulated bypass options (“Doesn’t meet criteria,” going to “Hospice/SNF,” “Already has naloxone”) then “Accept” (2 clicks);” or (3) commenting (≥2 clicks) [46]. “Already has naloxone” was included to account for naloxone outside the EHR. Clinicians were returned to their prior workflow after any action.

Clinicians were educated on CDS via departmental meetings and email. Educational materials included (1) CDS rationale, (2) instructions for use, and (3) suggested patient communication. No ongoing education was provided, and no changes were made to CDS after implementation.

This was a retrospective, observational study of ED visits in a large, not-for-profit university-affiliated, nongovernmental health care system. Located in the Rocky Mountain Region, the system has >500,000 total ED visits per year and includes 12 EDs—1 urban-academic level 1 trauma center, 2 urban community hospitals (1 a level 1 trauma center), 2 suburban community level 2 trauma centers, and 7 community free-standing EDs. The study was approved and informed consent was waived by the Colorado Multiple Institutional Review Board (COMIRB). The Guidelines and Checklist for the Reporting on Digital Health Implementations (iCHECK-DH) were followed (Multimedia Appendix 2) [47].

Naloxone coprescribing was defined as a clinician prescribing opioids and naloxone during the same ED visit. Deidentified patient characteristics (age, sex, race, ethnicity, preferred language, and insurance), CDS data (reasons for firing, number of firings per visit, clinician actions, and bypass reasons), and clinical variables (whether naloxone was prescribed via CDS and the prescription was filled) were extracted monthly from the shared EHR (Epic Systems). Research data governance was linked to EHR data governance. Clinicians entered data into Epic Hyperspace and CDS responses were automatically registered in real time. Extract, transform, and load processes transferred all patient data into relational databases hosted on private virtually protected servers nightly, and a Microsoft SQL Server Management Studio query was run to further clean and filter research data into Microsoft Excel.

The RE-AIM framework was used to determine the impact of CDS [39]. More explicitly, reach was measured by examining the proportion of ED visits where CDS was triggered and whether patients’ characteristics influenced opioid prescribing (and high-risk opioid prescribing, ie, CDS triggering) and naloxone coprescribing. Effectiveness (primary outcome) was assessed by evaluating the number of ED discharges with naloxone coprescriptions per week across the system before and after CDS implementation. Effectiveness was also measured by quantifying the naloxone prescription fill rate (naloxone prescription fills per naloxone orders via CDS vs other workflows) at a 24-hour ED outpatient retail pharmacy in the largest urban academic ED. This subgroup analysis was performed to determine whether increased naloxone orders translated to more naloxone reaching patients and to compare whether patients prescribed naloxone via CDS were more likely to fill their prescriptions than patients prescribed naloxone via other workflows. All prescriptions written at this ED defaulted to the ED’s outpatient pharmacy—unless specifically requested by the patient—thus prescription fill data were available in the pharmacy EHR. Adoption was defined as the number of naloxone prescriptions from CDS per number of CDS firings. Due to EHR limitations, we could not measure CDS suppression. The process of implementation is described. Finally, maintenance was judged by whether CDS was maintained after the study period and by modeling changes in adoption over time.

All data releases were cleared by a Research Services Manager who ensured the data being released were compliant with the Health Insurance Portability and Accountability Act (HIPAA) and the corresponding institutional review board exemption (#23-0458). No continuing review was required because this was secondary research and all data were deidentified. Results were shared via secure email. Individual informed patient consent was waived and no compensation was offered, given no patient participation or protected health information was shared.

There is a documented need for rigorous, pragmatic evaluation when implementing new CDS [34]. Interrupted time series analyses are suggested for CDS evaluation because they control for confounding secular trends [34,48-50]. We used a Bayesian structural time-series model controlling for the number of ED visits to evaluate the impact of CDS on naloxone coprescribing (CausalImpact package; version 1.3.0; Brodersen et al) [51], Mann-Kendall tests to model longitudinal changes in CDS adoption, and chi-square tests to compare the proportions of individuals who triggered CDS and were prescribed either an opioid or opioid with naloxone across demographic categories. Equity of RE-AIM outcomes was evaluated based on patient characteristics because prior research has demonstrated an increased likelihood of opioid prescribing for White patients and increased naloxone prescribing (and coprescribing) for Black and Latine patients [52-58]. Otherwise, frequencies and percentages are reported for categorical variables. All statistical analyses were conducted in R (version 4.2.2; R Core Team) [59].

All ED visits with a discharge opioid prescription between March 2013 and July 2023 were included. Effectiveness was assessed by comparing weekly aggregated counts of ED visits, opioid prescriptions, and naloxone coprescriptions between the pre- (March 2013-November 2019) and postimplementation periods (November 2019-July 2023). Adoption was assessed using only post-period data. The accuracy of synthetic control models, like the Bayesian structural time-series model used, is generally improved by including more pre-period data [51]. Therefore, we extracted enough data to provide a 2:1 pre-to post-period ratio. We did not perform a prospective power calculation. Patients younger than 18 years or older than 90 years old and those who were admitted to the hospital were excluded from both periods.

After implementation, between November 2019 and July 2023, there were 1,994,994 eligible ED discharges. Of these, 5.19% (103,606/1,994,994) included an opioid analgesic prescription and 0.83% (16,566/1,994,994) of prescriptions met high-risk criteria. Most visits included female (n=1,083,973, 54.33%), White (n=1,357,153, 68.03%), non-Latine (n=1,519,584, 76.17%), English-speaking (n=1,866,744, 93.57%), and publicly insured (n=1,146,781, 57.48%) patients (Table 1). White, non-Latine, English-speaking, privately insured patients were prescribed opioids a greater proportion of the time compared to Black, Latine, non–English-speaking, and publicly insured patients (P<.001).

CDS fired in 0.83% (16,566/1,994,994) of all ED visits. A total of 15.99% (16,566/103,606) of visits with a discharge opioid prescription met high-risk criteria and triggered CDS. CDS fired multiple times in 13.17% (2182/16,566) of visits (mean 1; median 1); ED clinicians interacted with CDS 19,246 times overall. Visits triggering CDS most often involved patients who were female (n=8670, 52.34%), White (n=12,724, 76.81%), non-Latine (n=13,818, 83.41%), spoke English (n=15,934, 96.18%), and had Medicaid or Medicare (n=10,193, 61.53%; Table 1). However, adjusting for the number of visits with an opioid prescription (a prerequisite for CDS triggering), CDS was more likely to trigger in visits with male, Latine, English speaking, and publicly insured patients (P<.001).

Before CDS implementation, clinicians coprescribed naloxone in 0.05% (156/318,216) of ED visits when an opioid analgesic was prescribed. After CDS implementation, ED clinicians coprescribed naloxone in 3.49% (3616/103,606) of ED visits when an opioid analgesic was prescribed. In the postimplementation period, 85.09% (3077/3616) of naloxone coprescriptions originated from CDS.

Using the number of ED visits as a covariate, the CausalImpact package predicted 0.80 (95% CI 0.55‐1.05) ED visits with naloxone coprescriptions per week and 150 (95% CI 100‐200) ED visits with naloxone coprescriptions in the postimplementation period. After CDS go-live there was an immediate increase in the number of ED visits with naloxone coprescriptions each week—18.9 ED visits with naloxone coprescriptions observed on average weekly and 3616 ED visits with naloxone coprescriptions in the entire postimplementation period (Figure 1). In other words, CDS increased ED naloxone coprescribing by 18.1 (95% CI 17.85‐18.34) naloxone coprescriptions per week or 2327% (95% CI 1702‐3335). Black and non-Latine patients were more likely to have naloxone coprescribed when CDS triggered compared to White and Latine patients (P<.001).

During the postimplementation period, there were 4541 naloxone coprescriptions with opioid analgesics written at the urban, academic ED (mean 1.2; median 1 per visit) and 3308 (72.85%) were ordered from a CDS alert. Patients filled 49.42% (2134/4318) of their opioid prescriptions and 43.80% (1989/4541) of naloxone coprescriptions. Patients coprescribed naloxone via CDS filled their prescriptions less often than patients coprescribed naloxone via other workflows (35.64%, 1179/3308 vs 65.69%, 810/1233).

ED clinicians adopted CDS, following the recommendation to coprescribe naloxone in 34.36% (6613/19,246) of alerts. Clinicians at the academic ED adopted CDS at a higher rate 61.62% (2005/3254) than at community EDs 34.70% (4608/13,280).

This CDS was implemented simultaneously at every ED and no changes were made to CDS postimplementation. All EDs used the same EHR, and it took a CDS builder 70 hours (including meetings, communications, and build time) to design and implement CDS.

According to the Mann-Kendall test, CDS adoption increased over time (τ=0.454; P<.001). Because no changes were made to CDS, there were no obvious sustainability costs beyond what our health system regularly paid for EHR access. CDS is still active and currently being scaled to outpatient clinics. Sustainability decisions are made by local governance based on naloxone prescribing because it was defined as a CDC best practice. CDS are reviewed ad hoc based on technical issues and yearly otherwise.

The implementation process benefitted from the makeup of the study team, who were able to provide local context for design, identify key workflow needs, address local barriers, and serve as champions during implementation. Beta testing and CDS-specific data analytics were prioritized to identify technical and efficiency issues early. Having data analytics built and collecting data during testing was key for providing estimates on workflow interruptiveness and informing iterative improvements. For example, monitoring revealed CDS initially only searched the current visit diagnosis, failing to identify histories of OUD and OD. The trigger algorithm was changed to include any EHR documented history of OUD or OD before going live, with a significant increase in case identification. Additionally, because clinicians told champions that CDS were firing “too late,” CDS were modified to trigger when clinicians entered as opposed to signed orders, facilitating clinician-patient communication before prescribing.

This project began as quality improvement, which was important for local buy-in. Also, the health system is funded, and therefore, owns the intervention. It would have been ideal to prospectively track implementation to elucidate system and per-patient costing and inform decisions about CDS maintenance. Future studies should formally evaluate patient-centered outcomes to confirm CDS as an effective and equitable implementation strategy.

A minimally interruptive CDS was readily adopted, showed a sustained effect, and significantly increased the number of ED naloxone coprescriptions. These findings support CDS as an effective implementation strategy to increase clinician uptake of naloxone best practices.

The high rate of adoption supports the need for user-centered CDS development, monitoring, and evaluation as the impact of CDS is often limited by low adoption and frequent workflow interruptions resulting in “alert fatigue” (the desensitization to important safety warnings) [24,60-64]. A Cochrane review of 122 CDS trials showed that CDS, on average, only increases the proportion of patients receiving desired care by 5.8% (95% CI 4.0% to 7.6%) [36]. The impact is variable, with the top quartile of reported improvements ranging from 10% to 62% [36]. With an adoption rate of 34.36% and a 2327% increase in the number of ED visits with naloxone coprescriptions, this CDS falls well within the top quartile of CDS improvements [36]. Interestingly, adoption increased over time. This finding differs from most other CDS literature reporting a decrease in adoption over time [65], and mirrors one other CDS study that reported a similar effect after UCD [66], perhaps suggesting that UCD improves initial and sustained adoption [66].

ED clinicians face increasingly complex workflow challenges that require validated solutions [67,68]. Previous evaluations of naloxone coprescribing CDS have not always aligned with best practices for designing, conducting, and reporting CDS interventions [28-31,34]. Prior studies have not discussed the rationale for CDS design (such as choice architecture) and have excluded key operational details (supplements and alert screenshots), making it challenging to reproduce or scale CDS [28-31,34]. The default order design of our CDS may have contributed to CDS acceptability by making choice architecture less burdensome to clinicians [34]. CDS adoption may also reflect actions in line with clinicians’ and patients’ positive attitudes toward naloxone prescribing and use [15,34,62].

Our Bayesian structural time series model, without a statutory mandate, offers robust evidence to support claims that CDS increases ED naloxone coprescribing. Our methods address the gap from prior studies that relied on pre-post designs and inferential statistics (logistic regression, t tests, and χ² tests) [30,31], which increase the risk of confounding by organizational policies, regulations, or reimbursement rules [34].

The fact that White, non-Latine, English-speaking, and privately insured patients were significantly more likely to have an opioid prescribed is concerning but consistent with prior literature [52-58]. Demographic differences in opioid and naloxone prescribing have been widely reported [52-58]. It is notable that Black and non-Hispanic patients were more likely to have naloxone coprescribed after CDS was triggered. This is the first study to report demographic differences in clinicians’ responses to CDS designed to increase naloxone coprescribing. Although, Black and Latine patients are coprescribed naloxone more often at baseline. Thus, it is possible CDS increased naloxone coprescribing equally and simply failed to reduce the influence of racial and ethnic bias on opioid or naloxone prescribing [57,58]. Other CDS designers should consider these differences when implementing and evaluating CDS to ensure they do not inadvertently maintain or widen existing disparities.

No clinical outcomes were measured, so we do not know if practice changes impacted care such as ED readmissions. No statutory mandates were implemented during this study, but we cannot be sure local educational efforts were not made to encourage naloxone coprescribing. Larger trends in opioid prescribing were not examined but are unlikely to have impacted the rate of naloxone coprescribing.

The availability of a 24-hour ED pharmacy at the academic site was another potential operational confounder in measuring naloxone fill rates. Discharged patients had to pass the pharmacy to exit the ED. This is an important consideration for sustainability since we do not compare naloxone coprescribing versus take-home naloxone (THN). THN has been reported to improve naloxone distribution by removing the need to stop at a pharmacy and may alleviate patient costs but shift medication costs to systems or public health organizations. Prior work, evaluating THN programs, has reported naloxone distribution rates as high as 87.3% [69]. However, THN programs are resource intensive [69,70], thus, might still be improved by CDS that improve the recognition of patients at risk for OD [71].

An EHR-based CDS encouraging ED naloxone coprescribing with opioid analgesics increased alert-based naloxone orders and overall system rates of naloxone coprescribing. The CDS had a low rate of interruption, a high rate of adoption [36], and significantly increased ED naloxone coprescribing across 12 EDs. There were no obvious sustainability costs beyond what the health system regularly paid for EHR access. These findings support claims that health care system leaders should consider CDS as an implementation strategy to address the significant gap in naloxone coprescribing [28-33].