This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.
As a result of the overwhelming proportion of medication errors occurring each year, there has been an increased focus on developing medication error prevention strategies. Recent advances in electronic health record (EHR) technologies allow institutions the opportunity to identify medication administration error events in real time through computerized algorithms. MED.Safe, a software package comprising medication discrepancy detection algorithms, was developed to meet this need by performing an automated comparison of medication orders to medication administration records (MARs). In order to demonstrate generalizability in other care settings, software such as this must be tested and validated in settings distinct from the development site.
The purpose of this study is to determine the portability and generalizability of the MED.Safe software at a second site by assessing the performance and fit of the algorithms through comparison of discrepancy rates and other metrics across institutions.
The MED.Safe software package was executed on medication use data from the implementation site to generate prescribing ratios and discrepancy rates. A retrospective analysis of medication prescribing and documentation patterns was then performed on the results and compared to those from the development site to determine the algorithmic performance and fit. Variance in performance from the development site was further explored and characterized.
Compared to the development site, the implementation site had lower audit/order ratios and higher MAR/(order + audit) ratios. The discrepancy rates on the implementation site were consistently higher than those from the development site. Three drivers for the higher discrepancy rates were alternative clinical workflow using orders with dosing ranges; a data extract, transfer, and load issue causing modified order data to overwrite original order values in the EHRs; and delayed EHR documentation of verbal orders. Opportunities for improvement were identified and applied using a software update, which decreased false-positive discrepancies and improved overall fit.
The execution of MED.Safe at a second site was feasible and effective in the detection of medication administration discrepancies. A comparison of medication ordering, administration, and discrepancy rates identified areas where MED.Safe could be improved through customization. One modification of MED.Safe through deployment of a software update improved the overall algorithmic fit at the implementation site. More flexible customizations to accommodate different clinical practice patterns could improve MED.Safe’s fit at new sites.
Patient safety is maximized when medical errors are efficiently detected and mitigated or prevented in the first place. The most common type of medical errors are medication errors, which are defined as any preventable event that may cause or lead to inappropriate medication use or patient harm while the medication is in the control of the health care professional, patient, or consumer [
The persistence of medication errors has led to a need for clinical informatics methods and technological interventions to improve medication error detection and prevention [
The availability of digitized EHRs and medication administration records (MARs) make it possible to perform algorithmic analysis of the data to detect MAEs quickly and efficiently [
In this study, we sought to implement MED.Safe, a software package of medication discrepancy detection algorithms, and benchmark the results to our earlier work at the development site to determine its portability and generalizability. We analyzed the system outputs at an external site, highlighting where and in what context the system performed well, and suggested customizations to further improve its performance. This analysis will provide the basis for further implementation and scaling of the current software package into other health care institutions.
The study took place at Wake Forest Baptist Medical Center (WFBMC), a tertiary level 1 trauma center and level 1 pediatric trauma center with 885 beds in Winston-Salem, North Carolina. WFBMC implemented an EHR system (Epic Systems) in 2012. This study focuses on the pediatric intensive care unit (PICU) with 12 beds, the neonatal ICU (NICU) with 40 beds, and the adult medical ICU with 172 beds.
Order and MAR data were extracted from the EHR for 11 medications prescribed at WFBMC: dobutamine, dopamine, epinephrine, fentanyl, insulin, intravenous (IV) fluids, lipids, milrinone, morphine, total parenteral nutrition (TPN), and vasopressin. The medications were originally selected by the investigative team (EK, KM, YN) because they were the continuously infused medications associated with the highest harm in the NICU setting. Structural differences in the format of 2 of the medication orders between the sites were taken into account during data extraction. At Cincinnati Children’s Hospital Medical Center (CCHMC), all TPN and IV fluids are contained in orders under 1 parent order for each medication/fluid category. At WFBMC, there is no single parent order, and additional mapping of the individual fluid and TPN orders was necessary. After accounting for this difference, the data from WFBMC were retrospectively extracted for the calendar year 2018 (January 1, 2018, to December 31, 2018). To compare system outputs, NICU data from CCHMC were also retrospectively extracted over the same period.
MED.Safe is an automated software package that analyzes medication use information in EHRs to identify medication administration discrepancies [
The analyzed information includes (1) medication orders that document medication doses (or infusion rates) prescribed to the patients, (2) structured order modifications (audits) that adjust the original doses/rates via computerized physician order entry, (3) MARs that document actual doses/rates administered to patients, and (4) free-text physician to nurse communication orders that deliver complex dose/rate adjustment during patient care. The free-text communications were parsed with a set of regular expression–based natural language processing algorithms to identify discrete dose/rate changes. The output consists of matching ordered medication doses with those recorded on the MAR in chronologic order. Using the extracted information, the detector module identifies discrepant doses/rates between MARs and other data sources using a set of logic-based rules. The detector was built upon our earlier research on MAE detection, where the logic-based rules were abstracted from standard care practices, refined by neonatologists, and implemented by programmers. By analyzing the dynamic EHR information, the detector determines the latest dose/rate prescribed to a patient and matches it with an MAR dose/rate to determine whether a match or discrepancy is present. MED.Safe allows users to map data elements required by the computerized algorithms to the site’s EHR instance data model. Once the mapping is complete, MED.Safe automatically extracts data from the EHR instance, executes the discrepancy detection algorithms, and visualizes chronological ordering of the medication use data and the identified discrepancies (if any). It also generates descriptive statistics of the medication use data including numbers of orders, audits, MARs, and discrepancies for the studied medications.
The investigative team (EK, BR, AM) executed the MED.Safe software package developed at CCHMC on the local WFBMC EHR data followed by a rigorous analysis of algorithm outputs. This step was completed entirely at WFBMC with guidance from the CCHMC study team (KM and YN). Analysis of the outputs was performed with the intent of learning the context within which the discrepancy detection algorithms were a good “fit” and performed accurately, and where they seemed to be inaccurate and needed customization for the new clinical environment.
The overall processes of the study, for executing MED.Safe at a second site.
To determine the fit and feasibility of MED.Safe at WFBMC, the investigative team (all study authors) analyzed the quantity and distribution of medication use data available. Descriptive statistics on medication orders, order modifications (ie, audits), and MARs generated by MED.Safe were aggregated by department (NICU, PICU, and adult medical ICU) and medication to study prescriber preferences and workflows. The analyzed MARs were restricted to actions including new bag, start, restart, rate verify, and rate change, to include administrations where potential administration errors could occur. Ratios comparing the numbers of audits, orders, and MARs were calculated for all ICUs at WFBMC and the NICU at CCHMC. The audit/order ratio represented the average number of times an order was modified during its life cycle, which implied prescribing patterns in a clinical environment (if prescribers frequently changed an order or kept a more stable prescribing habit). The MAR/(order + audit) ratio represented the average number of MARs documented by clinicians for each order or order modification, which suggested documentation patterns in a clinical unit.
After data element configuration, MED.Safe was executed against WFBMC’s clinical data repository to extract medication use data retrospectively. MED.Safe’s discrepancy detection algorithms were then performed for each WFBMC ICU department. We analyzed the results aggregated across the ICU departments and for WFBMC NICU solely and compared them with those from the development site (CCHMC) to determine specific settings (medications and clinical departments) that demonstrated the best fit and areas of improvement needed for the system. Results were visualized numerically and graphically to compare trends in discrepancy rates between WFBMC and CCHMC.
We assumed that good system generalizability to the WFBMC data would be expected to yield discrepancy rates similar to the baseline rates at CCHMC. Discrepancy rates substantially higher than the baselines were assumed to indicate a poor fit, which prompted further investigation to confirm this assumption and suggest areas of improvement.
If the discrepancy rate for a medication was higher than expected compared to the baseline, the system outputs were inspected manually to identify potential causes. The numbers of processed medication orders, audits, and MARs were interrogated to understand and examine the possible effect of local medication use patterns. For example, a specific type of order or MAR entry triggering discrepancies on more than 1 occasion might indicate a pattern of interest. These patterns were investigated, and the inspection was completed for each medication.
Manual analysis of the patterns identified in phase 3 was completed by the investigative team (all study authors) to pinpoint whether the source of discrepancy deviation was technical (caused by algorithm logic) or a result of clinical factors (a change of prescribing practices between sites that the system was not capable of capturing), in order to improve accuracy in MAE detection.
The technical barriers to good fit that were identified were addressed through the addition of a software update where feasible. The updated system was then re-executed on the same 2018 WFBMC data set. The updated system outputs were compared to the original system outputs in terms of order counts, order audit counts, MAR counts, and discrepancy rates to understand the impact of the customizations.
The audit/order and MAR/(order + audit) ratios are presented in
Distribution of medication orders, audits, and medication administration records in the WFBMC ICUs compared to the CCHMC NICU.
Distribution of data elements | WFBMCa adult medical ICUb | WFBMC PICUc | WFBMC NICUd | CCHMCe NICU |
Number of orders | 1950 | 1964 | 6390 | 12,603 |
Number of audits | 576 | 934 | 1137 | 4386 |
Number of MARsf | 38,787 | 62,780 | 166,879 | 56,715 |
aWFBMC: Wake Forest Baptist Medical Center.
bICU: intensive care unit.
cPICU: pediatric intensive care unit.
dNICU: neonatal intensive care unit.
eCCHMC: Cincinnati Children’s Hospital Medical Center.
fMAR: medication administration record.
Comparison of audit/order ratios between (A) CCHMC NICU, (B) WFBMC NICU, and (C) WFBMC All ICUs. CCHMC: Cincinnati Children’s Hospital Medical Center; ICU: intensive care unit; NICU: neonatal intensive care unit; WFBMC: Wake Forest Baptist Medical Center.
Comparison of MAR/(order + audit) ratios between the CCHMC NICU, the WFBMC NICU, and WFBMC All ICUs. CCHMC: Cincinnati Children’s Hospital Medical Center; ICU: intensive care unit; MAR: medication administration record; NICU: neonatal intensive care unit; WFBMC: Wake Forest Baptist Medical Center.
A comparison of medication administration discrepancy rates generated by MED.Safe at Wake Forest Baptist Medical Center and Cincinnati Children’s Hospital Medical Center during the study period.
Medication | Discrepancy rate at all ICUsa in WFBMCb, % | Discrepancy rate at NICUc in WFBMC, % | Discrepancy rate at NICU in CCHMCd, % |
Dobutamine | 7.9 | 19.8 | 0.0 |
Dopamine | 6.7 | 6.0 | 0.9 |
Epinephrine | 20.9 | 4.7 | 2.1 |
Fentanyl | 5.9 | 0.5 | 0.3 |
Insulin | 41.7 | 59.3 | 4.3 |
Intravenous fluids | 1.1 | 1.7 | 2.5 |
Lipids | 0.1 | 0.0 | 0.1 |
Milrinone | 1.1 | 0.3 | 0.0 |
Morphine | 6.7 | N/Ae | 0.1 |
Total parenteral nutrition | 1.4 | 1.4 | 1.3 |
Vasopressin | 2.1 | N/A | 2.3 |
aICU: intensive care unit.
bWFBMC: Wake Forest Baptist Medical Center.
cNICU: neonatal intensive care unit.
dCCHMC: Cincinnati Children’s Hospital Medical Center.
eN/A: not applicable. In 2018, no orders for continuous morphine or vasopressin were placed in the WFBMC NICU.
A comparison of discrepancy rates by medication and number of orders between (A) WFBMC All ICUs, (B) WFBMC NICU, and (C) CCHMC NICU. Circle radius correlates with the number of medication orders for the sites. CCHMC: Cincinnati Children’s Hospital Medical Center; ICU: intensive care unit; NICU: neonatal intensive care unit; WFBMC: Wake Forest Baptist Medical Center.
We further investigated the medications with discrepancy rates that substantially deviated from the CCHMC baseline. Three primary causes for the deviation of discrepancy rates were identified: (1) range-based dosing (a common prescribing practice); (2) a data extraction, transforming, and loading issue causing initial order values to be overwritten in the data (a technical data processing issue); and (3) verbal ordering practices (site-specific prescribing practice).
At WFBMC, some medication orders are written as a dosing range (eg, insulin 1-10 Units/hr, with an associated titration protocol) rather than as a discrete dose (eg, insulin 1 Units/hr, titrate by 0.5 Units/hr). Because MED.Safe expects a determinate dose for high-risk IV medications per guidelines at CCHMC, the dosing range practice resulted in very high levels of discrepancies for some medications (eg, insulin) at WFBMC, as seen in
Example of a dosing range order interpretation issue by the algorithm. In this example, orders placed with dosing ranges are not interpreted correctly by the system in place to detect medical administration discrepancies. The algorithms, in their current state, do not expect a dosing range and mark the MAR as a discrepancy if the value doesn’t match the first value in the order dose range. Subsequent titrations that would fall within the acceptable range of the order are erroneously identified as discrepancies by the algorithm. *The Order Dose/Rate in this figure represents the value that the algorithm parses from the original order. In the instance of orders being placed with a dose range (ie, 1-10 Units/hr), the algorithms only parse and use the first value of the dose range. MAR: medication administration record.
The second cause of deviation is related to an issue where original order doses/rates were overwritten or replaced by each new audit value, a consequence of the data extraction, transforming, and loading operations of the EHR software. We previously reported on this phenomenon in detail; it is the result of how the proprietary EHR system updates and stores audited order values in the retrospective database [
Example of an “order/audit value overwriting” issue leading to false positive calls from the system. Due to an ETL process, the original order value is repeatedly overwritten by the newer order audit values and ends up with the value of the last order audit record. When compared to the MAR documentations (which are correct), the false value in the order causes the algorithms to ‘detect’ a discrepancy, which is a false positive. ETL: extract, transform, load; MAR: medication administration record.
Lastly, there were discrepancies associated with changes to dosage (manifested as MAR documentations) that occurred greater than 30 minutes before the order was entered into the EHR. Such might occur as a result of an emergency during which a verbal order at the bedside is performed but not timely documented in the EHR. As such, the system implemented a 30-minute time window to account for these known lags in documentation due to verbal ordering while meeting the institutional expectations. This phenomenon is depicted in
Example of the delayed entry of a verbal order causing a discrepancy to be detected. A verbal order was given at the bedside and the medication was appropriately adjusted, but the order was not documented until after the MAR documentation was placed. The algorithms allow a 30-minute window for verbal orders to be entered before calling a discrepancy, but in this example the order audit for the verbal order rate was not entered until 76 minutes later. MAR: medication administration record.
The system found discrepancies in medication administration that were attributed to both technical and clinical factors, which contributed to the initial poor fit of discrepancy detection on some medications at the implementation site (WFBMC). To overcome these barriers to successful implementation, the algorithms should be customized to adapt to the local institution. As an initiative, we customized the algorithms with a software update to solve 1 of the 3 major sources of false-positive discrepancies: order/audit value overwrites (the second issue identified in phase 3).
The investigative team (all study authors) implemented a patch to MED.Safe to recover the original order values from the sequences of medication use data. We then re-executed the updated system on the data used in the initial analysis to study its effects.
A comparison of discrepancy rates between (A) CCHMC NICU, (B) WFBMC NICU using the updated MED.Safe, and (C) WFBMC using the original MED.Safe. CCHMC: Cincinnati Children’s Hospital Medical Center; IV: intravenous; NICU: neonatal intensive care unit; TPN: total parenteral nutrition; WFBMC: Wake Forest Baptist Medical Center.
Discrepancy rates of medication administration in the NICU before and after implementation of a software update at WFBMC in comparison to the site of development CCHMC.
Medication | Absolute change in discrepancy rate, % | Initial discrepancy rates in CCHMCc NICU, % | ||
Dobutamine | 19.8 | 19.5 | –0.3 | 0.0 |
Dopamine | 6.0 | 6.0 | 0.0 | 0.9 |
Epinephrine | 4.7 | 3.8 | –0.9 | 2.1 |
Fentanyl | 0.5 | 0.25 | –0.25 | 0.3 |
Insulin | 59.3 | 59.3 | 0.0 | 4.3 |
Intravenous fluids | 1.7 | 1.0 | –0.7 | 2.5 |
Lipids | 0.0 | 0.0 | 0.0 | 0.1 |
Milrinone | 0.3 | 0.19 | –0.11 | 0.0 |
Morphine | N/Ad | N/A | N/A | 0.1 |
Total parenteral nutrition | 1.4 | 1.4 | 0.0 | 1.3 |
Vasopressin | N/A | N/A | N/A | 2.3 |
aWFBMC: Wake Forest Baptist Medical Center.
bNICU: neonatal intensive care unit.
cCCHMC: Cincinnati Children’s Hospital Medical Center.
dN/A: not applicable. In 2018, no orders for continuous morphine or vasopressin were placed in the WFBMC NICU.
The ability to effectively implement the MED.Safe package at a second site is the first critical step toward creating a scalable and impactful solution for detecting and mitigating medication errors. This study investigated the feasibility and success of implementation for MED.Safe at a second site distinct from the origin of the software. The system outputs, such as descriptive statistics from local EHR data and discrepancy rates, served as a means to understand the institutional clinical workflows and prescribing patterns, assess the system generalizability, and help develop site-specific customizations. It is our hope that this study will serve as a guide for future institutions to efficiently assess the applicability of MED.Safe and lead to its implementation in an effort that maximizes medication safety in clinical settings.
Consideration of the clinical policies and workflows surrounding medication ordering, auditing, and MARs was vital in determining the feasibility of MED.Safe implementation at WFBMC. We observed that the NICU, PICU, and adult medical ICU were fundamentally different in their prescribing and auditing patterns (
In phase 2, we executed the discrepancy detection algorithms of the software and analyzed the output discrepancy rates at WFBMC (
WFBMC uses dosing ranges to allow for bedside adjustment of a medication so long as the dosing is in range of the order and follows ancillary instructions, protocols, or policies. Such practice is common in adult medication prescribing, particularly in the administration of insulin, where dosing might shift within a given range depending on the trend of blood glucose values or intake of food. However, the algorithms were not equipped to deal with ordering ranges because at CCHMC site-specific practices required that an order dose/rate should be determinate and an audit (modification) be documented each time a dose/rate was changed. Consequently, WFBMC had comparatively fewer audits and more discrepancies for values within the acceptable dosing range. This difference in site-specific practices resulted in high discrepancy rates for insulin (59.3% at WFBMC NICU versus 4.3% at CCHMC NICU). A quick glance at the descriptive data and discrepancy rates generated by the algorithms will cue future customizations as to the cause of the high rates and shortcut much of the time spent in exploration and validation.
Second, the investigative team (all study authors) determined that the institutional EHR was overwriting the original order values with each new audit. The overwriting resulted in a notable amount of false-positive discrepancies on the first MARs. We were able to overcome this EHR-derived technical limitation with a software update that recovered the original order dose/rate by reasoning through from the sequences of order-audit data.
Lastly, a portion of discrepancies originated from dose/rate changes with delayed order documentation. This often occurs in emergency settings where verbal orders are first placed, while electronic orderings are documented after the care is delivered. The “grace period” for entering the electronic orders varies between institutions based on the site-specific clinical practices. Operating under verbal orders without proper documentation and procedure is high risk, and it creates a blind spot for errors that may have occurred but lacked the appropriate data for the system to detect them. The inability to identify medication errors during this elapsed time might lead to perpetuation of similar errors for an extended period, ultimately lessening the value of the system in identifying errors efficiently. A change in policy to eliminate the practice of verbal ordering is one potential solution, but this does not fit with the reality of clinical practice. Another solution is to adapt the system to the “grace period” that complies with local policies surrounding verbal ordering. For instance, the MED.Safe algorithms adopted a period of 30 minutes given the institutional expectations at CCHMC, which could be extended to 45-60 minutes to comply with WFBMC’s verbal ordering policies. In our quick sensitivity analysis we found that an extension to a 60-minute window, however, did not greatly reduce the discrepancy rate. This effect appears to be site specific as we have seen this change decrease rates to a greater degree at other sites. In the future, we will add this customizable feature to the software so that the grace period can be adjusted depending on the care setting and local policy. This will also allow an automated version of the sensitivity analysis. Ultimately, the system could be more flexible and customizable to fit each institution and even department that varies in health care policy and procedures surrounding the medication use life cycle.
In phase 4, we addressed the order/audit value overwriting issue through a software update. It reduced false-positive discrepancies output by the system for most of the studied medications. The remaining 2 medications (dobutamine and insulin) with discrepancy rates notably higher than baseline CCHMC rates are largely due to the range-based dosing issue. Further reduction in false-positive discrepancies can therefore be obtained by addressing the other 2 issues, range-based dosing and verbal ordering practices. Efforts to do so are planned for future work.
Our study suggested that it was feasible to implement MED.Safe in a setting external to the development environment. However, the software package did not account for all the differences in medication administration practices at the implementation site, with a resultant impact on its performance. The identified barriers to proper fitting of the system can be overcome through both clinical practice change/policy reform and the addition of algorithm customizations where appropriate. We were able to identify targets for algorithm customization to account for these practices and to address one of those issues efficiently. These efforts have greatly advanced our knowledge of the portability of the MED.Safe and have shown us what work is left to do in order to further improve its generalizability.
The implementation of the MED.Safe system at a second site was a feasible and efficient way to track medical administration discrepancies. Analysis of medication use data and discrepancy rates output by the system revealed local medication prescribing patterns, and comparison against implementation at the original site suggested areas of both good and poor fit. Overall fit was enhanced through the implementation of a software update. To maximize efficiency in accurately detecting and correcting medication errors, modifications must be made to both the MED.Safe software package and suboptimal clinical practices. Such modifications should increase the system’s customizability to the local clinical workflows and policies, ultimately improving its accuracy and generalization for external use.
A comparison of medication ordering, auditing, and MAR data by drug generated by MED.Safe at Wake Forest Baptist Medical Center. MAR: medication administration record.
A comparison of medication ordering, auditing, and MAR data by generated by MED.Safe at Wake Forest Baptist Medical Center. MAR: medication administration record.
Descriptive statistics of orders, audits, and medication administration record data at all Wake Forest Baptist Health ICUs during the study period. ICU: intensive care unit.
Descriptive statistics of orders, audits, and medication administration record data in the Wake Forest Baptist Health NICU during the study period. NICU: neonatal intensive care unit.
Descriptive statistics of orders, audits, and medication administration record data in the Cincinnati Children’s Hospital NICU during the study period. NICU: neonatal intensive care unit.
Cincinnati Children’s Hospital Medical Center
electronic health record
intensive care unit
intravenous
medication administration error
medication administration record
neonatal intensive care unit
pediatric intensive care unit
total parenteral nutrition
Wake Forest Baptist Medical Center
The investigative team thanks Meredith Hollinger, Wake Forest Pharmacy Shared Services, for her assistance in providing data for the study. In addition, Lara Kanbar from the CCHMC Division of Biomedical Informatics provided a critical review of the near-final manuscript. This research was supported by the National Institute of Health under award number R01LM012230 (sponsored by the National Library of Medicine).
None declared.