The study took place at Wake Forest Baptist Medical Center (WFBMC), a tertiary level 1 trauma center and level 1 pediatric trauma center with 885 beds in Winston-Salem, North Carolina. WFBMC implemented an EHR system (Epic Systems) in 2012. This study focuses on the pediatric intensive care unit (PICU) with 12 beds, the neonatal ICU (NICU) with 40 beds, and the adult medical ICU with 172 beds.

Order and MAR data were extracted from the EHR for 11 medications prescribed at WFBMC: dobutamine, dopamine, epinephrine, fentanyl, insulin, intravenous (IV) fluids, lipids, milrinone, morphine, total parenteral nutrition (TPN), and vasopressin. The medications were originally selected by the investigative team (EK, KM, YN) because they were the continuously infused medications associated with the highest harm in the NICU setting. Structural differences in the format of 2 of the medication orders between the sites were taken into account during data extraction. At Cincinnati Children’s Hospital Medical Center (CCHMC), all TPN and IV fluids are contained in orders under 1 parent order for each medication/fluid category. At WFBMC, there is no single parent order, and additional mapping of the individual fluid and TPN orders was necessary. After accounting for this difference, the data from WFBMC were retrospectively extracted for the calendar year 2018 (January 1, 2018, to December 31, 2018). To compare system outputs, NICU data from CCHMC were also retrospectively extracted over the same period.

MED.Safe is an automated software package that analyzes medication use information in EHRs to identify medication administration discrepancies [12,20,21]. The MED.Safe package was originally developed by CCHMC with the purpose of monitoring high-risk IV medications in the NICU setting.

The analyzed information includes (1) medication orders that document medication doses (or infusion rates) prescribed to the patients, (2) structured order modifications (audits) that adjust the original doses/rates via computerized physician order entry, (3) MARs that document actual doses/rates administered to patients, and (4) free-text physician to nurse communication orders that deliver complex dose/rate adjustment during patient care. The free-text communications were parsed with a set of regular expression–based natural language processing algorithms to identify discrete dose/rate changes. The output consists of matching ordered medication doses with those recorded on the MAR in chronologic order. Using the extracted information, the detector module identifies discrepant doses/rates between MARs and other data sources using a set of logic-based rules. The detector was built upon our earlier research on MAE detection, where the logic-based rules were abstracted from standard care practices, refined by neonatologists, and implemented by programmers. By analyzing the dynamic EHR information, the detector determines the latest dose/rate prescribed to a patient and matches it with an MAR dose/rate to determine whether a match or discrepancy is present. MED.Safe allows users to map data elements required by the computerized algorithms to the site’s EHR instance data model. Once the mapping is complete, MED.Safe automatically extracts data from the EHR instance, executes the discrepancy detection algorithms, and visualizes chronological ordering of the medication use data and the identified discrepancies (if any). It also generates descriptive statistics of the medication use data including numbers of orders, audits, MARs, and discrepancies for the studied medications.

The investigative team (EK, BR, AM) executed the MED.Safe software package developed at CCHMC on the local WFBMC EHR data followed by a rigorous analysis of algorithm outputs. This step was completed entirely at WFBMC with guidance from the CCHMC study team (KM and YN). Analysis of the outputs was performed with the intent of learning the context within which the discrepancy detection algorithms were a good “fit” and performed accurately, and where they seemed to be inaccurate and needed customization for the new clinical environment. Figure 1 presents an overview of the study, and the individual methodological steps are further described in the following sections.

Table 1 presents the distribution of medical use data for each ICU department at WFBMC. A total of 10,304 orders, 2647 audits, and 268,446 MARs were created during the study period. The NICU placed the most orders, made the most order modifications (audits), and created the most MAR entries. By contrast, the adult medical ICU had the least in all 3 categories, reflecting the fact that the MED.Safe system was originally designed for a pediatric population (the CCHMC NICU). Multimedia Appendices 1 and 2 present more specific breakdowns by medication and department, which suggested that IV fluids, TPN, lipids, and fentanyl were the most ordered medications and had the highest MARs in each of the investigated departments. The WFBMC NICU was the only investigated department without use of vasopressin and morphine; the other departments had orders and subsequent audits and MARs for all 11 medications studied. Additionally, the WFBMC NICU had almost 3 times the number of MARs when compared to the CCHMC NICU despite having only about half as many orders and audits. This was found to be the result of a practice of documenting rate verifications on the MAR much more frequently than the practice in the CCHMC NICU.

The audit/order and MAR/(order + audit) ratios are presented in Multimedia Appendices 3-5 to compare the differences in prescribing habits and order fluidity between WFBMC and CCHMC. Figure 2 compares the audit/order ratios between all WFBMC ICUs, WFBMC NICU (NICU subset of all WFBMC ICUs), and CCHMC NICU. The ratios differed substantially between the 3 data sets across the studied medications. The CCHMC NICU had higher audit/order ratios for 7 of the 11 medications. For example, dopamine at CCHMC had an audit/order ratio of 3.0, whereas that medication at WFBMC had an audit/order ratio of 0.9.

Figure 3 and Multimedia Appendices 3-5 present MAR/(order + audit) ratios between WFBMC departments and CCHMC NICU. The WFBMC NICU and all ICUs at WFBMC had comparable ratios. When compared to the CCHMC NICU, the ratios for WFBMC were higher for each studied medication. The average ratio for WFBMC NICU was 23.6 and the average for CCHMC was 4.4. The MAR/(order + audit) ratio for milrinone in the WFBMC NICU was higher than the other medications and departments. This is a result of WFBMC NICU’s practice to verify the rate of milrinone approximately every hour for the entire duration of the medication.

Table 2 presents the discrepancy rate output by MED.Safe for each studied medication. Compared to the baseline discrepancy rates from CCHMC NICU, 5 out of 9 medications used at WFBMC NICU (excluding vasopressin and morphine that did not have orders) showed close discrepancy rates, with less than 1% difference. Epinephrine had similar discrepancy rates, with less than 3% difference. However, the discrepancy rates for insulin, dobutamine, and dopamine were exceptionally large, with over 5% difference. Compared to WFBMC NICU, the discrepancy rates at all WFBMC ICUs tended to deviate more from CCHMC NICU.

Figure 4 further depicts the relationship between site, discrepancy rate, and medication. A circle size represents the number of orders for a medication during the study period while plotting the discrepancy rate by medication and institutional site. For nearly all medications, the CCHMC NICU had lower discrepancy rates when compared to WFBMC sites and a larger number of orders when compared to the WFBMC NICU specifically. We observed that the outliers in discrepancy rates (epinephrine, dopamine, dobutamine, and insulin) were often due to a small number of orders as represented by the small circle radius.

We further investigated the medications with discrepancy rates that substantially deviated from the CCHMC baseline. Three primary causes for the deviation of discrepancy rates were identified: (1) range-based dosing (a common prescribing practice); (2) a data extraction, transforming, and loading issue causing initial order values to be overwritten in the data (a technical data processing issue); and (3) verbal ordering practices (site-specific prescribing practice).

At WFBMC, some medication orders are written as a dosing range (eg, insulin 1-10 Units/hr, with an associated titration protocol) rather than as a discrete dose (eg, insulin 1 Units/hr, titrate by 0.5 Units/hr). Because MED.Safe expects a determinate dose for high-risk IV medications per guidelines at CCHMC, the dosing range practice resulted in very high levels of discrepancies for some medications (eg, insulin) at WFBMC, as seen in Table 2. Figure 5 demonstrates an example system output for an order with a dosing range, including the order, audit, and MARs for a single patient spanning 2 calendar days. After reviewing the patient chart, it was discovered that the original order in the EHR was set to a range of 1-10 Units/hr and was changed to 1-20 Units/hr approximately 6 hours later. However, the MED.Safe system expected a discrete dose for insulin and converted the dosing range to a single value, accepting only the lower-bound range value as an order dose/rate input despite the original physician order for 1-10 Units/hr. Consequently, it marked all of the MAR dose/rate values as causing discrepancies in this single patient. This is a technical limitation of the system design. If the system had been able to accommodate dosing ranges in orders, it should have analyzed the MARs appropriately and avoided false-positive alerts.

The second cause of deviation is related to an issue where original order doses/rates were overwritten or replaced by each new audit value, a consequence of the data extraction, transforming, and loading operations of the EHR software. We previously reported on this phenomenon in detail; it is the result of how the proprietary EHR system updates and stores audited order values in the retrospective database [22]. Figure 6 presents an example of this phenomenon. The original order value should be “5.0 Units/hr” (as evidenced by the first audit that changed dose from 5 to 4) but was listed as “3.0 Units/hr” that reflected the last dose modification (the second audit). Consequently, the first MAR was marked as discrepant. This issue resulted in inflated discrepancy rates because the first MAR could always be marked as discrepant if the original order value was no longer presented in our data. This data extraction, transforming, and loading pattern was confirmed by the team’s suspicions upon inspecting order values in the real-time production EHR system and comparing them to the retrospective data extracts. Astute readers may also notice that only the first MAR was considered discrepant by the system in Figure 6. This is because the system implements a “check the value with previous MAR data” logic that overrides subsequent discrepancy calls when the MAR values do not change in order to avoid overcalling discrepancies. As such, the first is considered a discrepancy, while subsequent consecutive MARs do not trigger a discrepancy to be called, by design.

Lastly, there were discrepancies associated with changes to dosage (manifested as MAR documentations) that occurred greater than 30 minutes before the order was entered into the EHR. Such might occur as a result of an emergency during which a verbal order at the bedside is performed but not timely documented in the EHR. As such, the system implemented a 30-minute time window to account for these known lags in documentation due to verbal ordering while meeting the institutional expectations. This phenomenon is depicted in Figure 7, where the rate was changed to “4.0 Units/hr” 76 minutes before the order was modified. By reviewing the patient chart, we confirmed that the dose was changed via a verbal order and the administration was correct. However, the system marked the corresponding MAR as a discrepancy given that there was no audit or new order entered into the EHR for over 30 minutes after the administration. As a quick sensitivity analysis, we modified the algorithms to accept orders within a 60-minute time window; a comparison of discrepancy rates demonstrated a minimal impact, with rates changing less than 0.142% across all medications.

The system found discrepancies in medication administration that were attributed to both technical and clinical factors, which contributed to the initial poor fit of discrepancy detection on some medications at the implementation site (WFBMC). To overcome these barriers to successful implementation, the algorithms should be customized to adapt to the local institution. As an initiative, we customized the algorithms with a software update to solve 1 of the 3 major sources of false-positive discrepancies: order/audit value overwrites (the second issue identified in phase 3).

The investigative team (all study authors) implemented a patch to MED.Safe to recover the original order values from the sequences of medication use data. We then re-executed the updated system on the data used in the initial analysis to study its effects. Figure 8 and Table 3 demonstrate its effects in decreasing the output discrepancy rates for fentanyl, dobutamine, epinephrine, milrinone, and IV fluids. The other medications retained their discrepancy rates prior to the update, implying that they were not affected by order/audit value overwriting errors. As a result of this update, discrepancy rates from the WFBMC NICU became comparable to those from the CCHMC NICU for 5 of 9 medications with orders. The remaining medications maintained rates approximately twofold higher than the baseline CCHMC rates. Although this customization corrected for order/audit value overwriting errors, false-positive discrepancies persist as a result of delayed documentation of verbal orders and dosing range issues.

The ability to effectively implement the MED.Safe package at a second site is the first critical step toward creating a scalable and impactful solution for detecting and mitigating medication errors. This study investigated the feasibility and success of implementation for MED.Safe at a second site distinct from the origin of the software. The system outputs, such as descriptive statistics from local EHR data and discrepancy rates, served as a means to understand the institutional clinical workflows and prescribing patterns, assess the system generalizability, and help develop site-specific customizations. It is our hope that this study will serve as a guide for future institutions to efficiently assess the applicability of MED.Safe and lead to its implementation in an effort that maximizes medication safety in clinical settings.

Consideration of the clinical policies and workflows surrounding medication ordering, auditing, and MARs was vital in determining the feasibility of MED.Safe implementation at WFBMC. We observed that the NICU, PICU, and adult medical ICU were fundamentally different in their prescribing and auditing patterns (Table 1 and Multimedia Appendices 1-5). The WFBMC NICU had the most orders, audits, and MARs for the studied medications, reflecting the fact that MED.Safe was originally designed for an NICU setting that did not include common adult vasopressors such as norepinephrine. The adult medical ICUs had far less medication orders despite greater bed count. This was partially due to the fact that norepinephrine would have contributed 1466 orders to the total order count in this environment if an algorithm was available in MED.Safe to detect discrepancies; if included, the descriptive statistics would have more closely correlated with the bed count across the units. Regardless, the descriptive statistics output by the system allowed us to quickly understand, at the aggregate level, how prevalent the medications and MAR documentations were in different clinical environments and where the system may be the most useful. For instance, we found from the descriptive statistics that the NICU did not have vasopressin and morphine orders. As such, the algorithms for those medications not prescribed would not have any utility in the NICU and implementing MED.Safe there would yield no benefit. Beyond the basic descriptive characteristics, the comparison between audit/order ratios at WFBMC and CCHMC (Figure 2 and Multimedia Appendices 1-5) allowed us to understand the differences in prescribing workflows between the institutions. The lower audit/order ratios at WFBMC in comparison to CCHMC lead us to believe that WFBMC tends to create new orders for medication dose/rate changes, whereas CCHMC modifies existing orders for such changes more frequently. The more frequent use of order dose range intervals in combination with practices of documenting MAR rate to verify values very frequently may have contributed to the higher MAR/(order + audit) ratios at the WFBMC NICU despite fewer orders and audits overall (compared to CCHMC NICU). Our findings highlight potential practice differences across institutions, which may change the distribution of discrepancy rates, introduce additional opportunities to identify errors, or suggest the need for customizations to the MED.Safe system.

In phase 2, we executed the discrepancy detection algorithms of the software and analyzed the output discrepancy rates at WFBMC (Table 2). The rates at WFBMC aligned well with the ones at CCHMC for the majority of the studied medications. However, the rates at WFBMC varied widely, ranging from 0% to 59%, compared to CCHMC rates that ranged from 0% to 4.3%. The results suggested that the algorithms generalized well to the data and clinical practices for some medications but fit poorly for the others. Further inspection for the poorly performing medications in phase 3 identified 3 phenomena that contributed to the inflated discrepancy rates: range-based dosing, order/audit value overwriting in the data, and verbal ordering practices.

WFBMC uses dosing ranges to allow for bedside adjustment of a medication so long as the dosing is in range of the order and follows ancillary instructions, protocols, or policies. Such practice is common in adult medication prescribing, particularly in the administration of insulin, where dosing might shift within a given range depending on the trend of blood glucose values or intake of food. However, the algorithms were not equipped to deal with ordering ranges because at CCHMC site-specific practices required that an order dose/rate should be determinate and an audit (modification) be documented each time a dose/rate was changed. Consequently, WFBMC had comparatively fewer audits and more discrepancies for values within the acceptable dosing range. This difference in site-specific practices resulted in high discrepancy rates for insulin (59.3% at WFBMC NICU versus 4.3% at CCHMC NICU). A quick glance at the descriptive data and discrepancy rates generated by the algorithms will cue future customizations as to the cause of the high rates and shortcut much of the time spent in exploration and validation.

Second, the investigative team (all study authors) determined that the institutional EHR was overwriting the original order values with each new audit. The overwriting resulted in a notable amount of false-positive discrepancies on the first MARs. We were able to overcome this EHR-derived technical limitation with a software update that recovered the original order dose/rate by reasoning through from the sequences of order-audit data.

Lastly, a portion of discrepancies originated from dose/rate changes with delayed order documentation. This often occurs in emergency settings where verbal orders are first placed, while electronic orderings are documented after the care is delivered. The “grace period” for entering the electronic orders varies between institutions based on the site-specific clinical practices. Operating under verbal orders without proper documentation and procedure is high risk, and it creates a blind spot for errors that may have occurred but lacked the appropriate data for the system to detect them. The inability to identify medication errors during this elapsed time might lead to perpetuation of similar errors for an extended period, ultimately lessening the value of the system in identifying errors efficiently. A change in policy to eliminate the practice of verbal ordering is one potential solution, but this does not fit with the reality of clinical practice. Another solution is to adapt the system to the “grace period” that complies with local policies surrounding verbal ordering. For instance, the MED.Safe algorithms adopted a period of 30 minutes given the institutional expectations at CCHMC, which could be extended to 45-60 minutes to comply with WFBMC’s verbal ordering policies. In our quick sensitivity analysis we found that an extension to a 60-minute window, however, did not greatly reduce the discrepancy rate. This effect appears to be site specific as we have seen this change decrease rates to a greater degree at other sites. In the future, we will add this customizable feature to the software so that the grace period can be adjusted depending on the care setting and local policy. This will also allow an automated version of the sensitivity analysis. Ultimately, the system could be more flexible and customizable to fit each institution and even department that varies in health care policy and procedures surrounding the medication use life cycle.

In phase 4, we addressed the order/audit value overwriting issue through a software update. It reduced false-positive discrepancies output by the system for most of the studied medications. The remaining 2 medications (dobutamine and insulin) with discrepancy rates notably higher than baseline CCHMC rates are largely due to the range-based dosing issue. Further reduction in false-positive discrepancies can therefore be obtained by addressing the other 2 issues, range-based dosing and verbal ordering practices. Efforts to do so are planned for future work.

Our study suggested that it was feasible to implement MED.Safe in a setting external to the development environment. However, the software package did not account for all the differences in medication administration practices at the implementation site, with a resultant impact on its performance. The identified barriers to proper fitting of the system can be overcome through both clinical practice change/policy reform and the addition of algorithm customizations where appropriate. We were able to identify targets for algorithm customization to account for these practices and to address one of those issues efficiently. These efforts have greatly advanced our knowledge of the portability of the MED.Safe and have shown us what work is left to do in order to further improve its generalizability.

The implementation of the MED.Safe system at a second site was a feasible and efficient way to track medical administration discrepancies. Analysis of medication use data and discrepancy rates output by the system revealed local medication prescribing patterns, and comparison against implementation at the original site suggested areas of both good and poor fit. Overall fit was enhanced through the implementation of a software update. To maximize efficiency in accurately detecting and correcting medication errors, modifications must be made to both the MED.Safe software package and suboptimal clinical practices. Such modifications should increase the system’s customizability to the local clinical workflows and policies, ultimately improving its accuracy and generalization for external use.