Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, December 24 through Wednesday, December 26 inclusive. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 07.02.19 in Vol 7, No 1 (2019): Jan-Mar

Preprints (earlier versions) of this paper are available at, first published Oct 30, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Use of Electronic Health Record Access and Audit Logs to Identify Physician Actions Following Noninterruptive Alert Opening: Descriptive Study

    1Meyers Primary Care Institute, Worcester, MA, United States

    2University of Massachusetts Medical School, Worcester, MA, United States

    3Reliant Medical Group, Worcester, MA, United States

    4University of Massachusetts Memorial Health Care, Worcester, MA, United States

    5Edith Nourse Rogers Memorial Veterans Hospital, Veterans Health Administration, Bedford, MA, United States

    Corresponding Author:

    Azraa Amroze, BS

    Meyers Primary Care Institute

    385 Grove Street

    Worcester, MA, 01605

    United States

    Phone: 1 774 275 6030

    Fax:1 508 453 8193



    Background: Electronic health record (EHR) access and audit logs record behaviors of providers as they navigate the EHR. These data can be used to better understand provider responses to EHR–based clinical decision support (CDS), shedding light on whether and why CDS is effective.

    Objective: This study aimed to determine the feasibility of using EHR access and audit logs to track primary care physicians’ (PCPs’) opening of and response to noninterruptive alerts delivered to EHR InBaskets.

    Methods: We conducted a descriptive study to assess the use of EHR log data to track provider behavior. We analyzed data recorded following opening of 799 noninterruptive alerts sent to 75 PCPs’ InBaskets through a prior randomized controlled trial. Three types of alerts highlighted new medication concerns for older patients’ posthospital discharge: information only (n=593), medication recommendations (n=37), and test recommendations (n=169). We sought log data to identify the person opening the alert and the timing and type of PCPs’ follow-up EHR actions (immediate vs by the end of the following day). We performed multivariate analyses examining associations between alert type, patient characteristics, provider characteristics, and contextual factors and likelihood of immediate or subsequent PCP action (general, medication-specific, or laboratory-specific actions). We describe challenges and strategies for log data use.

    Results: We successfully identified the required data in EHR access and audit logs. More than three-quarters of alerts (78.5%, 627/799) were opened by the PCP to whom they were directed, allowing us to assess immediate PCP action; of these, 208 alerts were followed by immediate action. Expanding on our analyses to include alerts opened by staff or covering physicians, we found that an additional 330 of the 799 alerts demonstrated PCP action by the end of the following day. The remaining 261 alerts showed no PCP action. Compared to information-only alerts, the odds ratio (OR) of immediate action was 4.03 (95% CI 1.67-9.72) for medication-recommendation and 2.14 (95% CI 1.38-3.32) for test-recommendation alerts. Compared to information-only alerts, ORs of medication-specific action by end of the following day were significantly greater for medication recommendations (5.59; 95% CI 2.42-12.94) and test recommendations (1.71; 95% CI 1.09-2.68). We found a similar pattern for OR of laboratory-specific action. We encountered 2 main challenges: (1) Capturing a historical snapshot of EHR status (number of InBasket messages at time of alert delivery) required incorporation of data generated many months prior with longitudinal follow-up. (2) Accurately interpreting data elements required iterative work by a physician/data manager team taking action within the EHR and then examining audit logs to identify corresponding documentation.

    Conclusions: EHR log data could inform future efforts and provide valuable information during development and refinement of CDS interventions. To address challenges, use of these data should be planned before implementing an EHR–based study.

    JMIR Med Inform 2019;7(1):e12650




    Audit and access logs in the electronic health record (EHR) have primarily been used for security purposes [1-3], but recent studies [4-7] indicate that a broad range of additional research and clinical questions may be answered using this relatively untapped data source. Access logs contain time-stamped recordings of who accessed the EHR and what part was accessed [5]. Audit logs record chronological activity in the EHR, tracking actions such as what data were created, reviewed, or changed by the user [3]. These log data, though potentially complex to retrieve and interpret, can be used by researchers and care teams to identify targets for clinical interventions as well as care quality and safety-improvement efforts [4,8].

    Data from access and audit logs can be used to better understand how various forms of clinical decision support (CDS) impact physician behavior. This is especially relevant for noninterruptive EHR alerts, where assessments of the effectiveness of these alerts have found mixed results [9-12]. In contrast to alerts that “pop-up” and interrupt workflow when a specific EHR action is taken, noninterruptive alerts deliver information to an EHR InBasket, requiring the receiver to open and review them. This information may include warnings about out-of-range test results [9,13,14], abnormal findings on diagnostic imaging and screening [10,15-17], important changes in patient health [18], and safety concerns during transition of care [19]. However, these alerts must compete for the attention of physicians who are often overwhelmed by the number of electronic notifications they receive, read, and respond to while managing direct patient care [4,20-23].

    Access logs can provide valuable insight into the precise time of alert opening, even if the alert is opened many days postdelivery; logs can also help pinpoint actions following opening that might otherwise be difficult to link back to a particular CDS intervention. Although causality of EHR actions can be difficult to prove, there is value to demonstrating close temporal proximity of an EHR action after alert opening. Such knowledge can shed light on the mechanisms by which an EHR–based intervention is (or is not) effective.

    The effectiveness of a noninterruptive alerting system was tested by our team in a previous randomized trial (trial registration: NCT00611091) [19]. Locally developed alerts regarding older adults’ posthospital discharge were sent to their primary care physicians (PCPs) through the EHR [24,25]. These alerts highlighted what our team believed to be actionable medication safety concerns. The trial assessed whether these alerts increased outpatient visits and reduced rehospitalization [19]. The alerts did not improve either of these measures. To better understand this outcome, we conducted this descriptive study using audit and access log data to track actions taken by physicians in the EHR following alert delivery. In this paper, we present findings from this analysis of physician action. We also describe the availability and level of detail of information in these logs and review challenges and strategies for success.



    This study was based on data from the intervention arm of a prior randomized controlled trial, as mentioned above [19]. In that trial, an EHR–based intervention was implemented using locally developed noninterruptive alerts to highlight postdischarge medication safety concerns for older patients. The inpatient facility to which the health care system admitted its patients used a different EHR than that used by the health care system. An interface engine was linked to the hospital’s admission, discharge, and transfer registration system. The hospital transmitted information including discharge dates for the health care system’s patients; these data were automatically incorporated into the health care system’s EHR. Informed by health plan data reflecting new prescriptions filled, alerts were then generated for patients. These alerts were designed to convey actionable time-sensitive concerns pertaining to high-risk drugs (known to result in more adverse drug events) and new drugs. Over the 4 months prior to implementation, two physicians from the health care system met weekly to review every alert generated, suggesting modifications to ensure that alerts would be perceived as necessary, useful, and brief. Alerts that they considered inactionable were eliminated. At the time of hospital discharge, patients were randomized to the intervention (EHR alerts sent to the patient’s PCP if medication concerns were identified) or usual follow-up care. The primary goal of the intervention was to decrease rehospitalization rates.


    This study was conducted at a large Massachusetts multispecialty medical group with 217 physicians at 15 sites throughout Central Massachusetts. All sites used an EHR from Epic Systems Corporation.

    Study Sample

    When the original trial was conducted in 2011, approximately 140,000 adults received primary care through this medical group, of which approximately 24,000 were aged ≥65 years and were members of a local health plan. Patients who were members of this local health plan with primary care providers within this medical group and were discharged to home from the primary hospital used by the medical group from August 26, 2010, to August 25, 2011, were included in the trial. A total of 1282 patients were randomized to the intervention arm. In our analysis, we included only the 799 patient discharges for which PCP medication alerts were triggered (corresponding to 713 patients). These alerts were sent to 75 PCPs.

    The Institutional Review Boards at Reliant Medical Group and the University of Massachusetts Medical School approved this study, and waiver of consent was obtained.

    Variables and Data Sources

    Data available from the original trial included details about the timing and content of alerts, the scrambled identifier of the physician to whom the alerts were sent, and additional PCP characteristics including age (<50 vs ≥ 50 years), gender, and specialty (internal medicine, family medicine, PCP without a Doctor of Medicine degree such as a nurse practitioner, and subspecialist acting as PCP). To approximate full- versus part-time status of the physicians, we categorized the total number of patient encounters for each physician during the year prior to alert delivery in quartiles (0-2326, 2327-2783, 2784-3173, and >3173) across the 75 PCPs to whom alerts were sent. We collected the following information about the relevant patients: age (65-74, 75-84, and ≥85 years), gender, Charlson comorbidity index score (categorized as 0, 1, 2, and ≥3), dates and length (≤2 days, 3 days, and ≥4 days) of the related hospitalization, number of office visits in the past year (≤6, 7-11, 12-18, and >18), patient outcomes, and scrambled patient identifiers. Our research team grouped all the alerts sent during the intervention into three categories: information about new and high-risk medications at the time of hospital discharge (“information only”), recommendation to cancel or modify doses of medications (“medication recommendation”), and recommendation to order tests—laboratory tests or, in a few cases, eye exams—to monitor the impact of high-risk medications or titrate their doses (“test recommendation”).

    For this descriptive study, we sought a range of additional data elements from the EHR access and audit logs (described below) to provide information on the opening of alerts and track specific physician actions within the EHR. We captured the timing of alert opening relative to alert delivery (≤24 hours, 24-48 hours, and >48 hours) within office hours (8 AM-5 PM Monday through Friday) as compared to outside office hours. We also identified the user opening the alert (the intended PCP vs a staff member or provider other than the intended PCP).

    We gathered data on the following variables pertaining to the PCP’s InBasket load categorized by quartile: total number of notifications in the InBasket at the time of alert delivery (≤42, 43-69, 70-157, and >157), number of unopened notifications in the InBasket at the time of alert delivery (0, 1-4, 5-9, and >9), and number of notifications delivered to the InBasket in the 7 days prior to alert delivery (≤344, 345-453, 454-546, and >546). Finally, we compared alerts delivered on Saturdays to those delivered on all other days, because those arriving on Saturday were the only alerts for which the 24 hours postdelivery did not include any weekday time.

    Our goal was to identify factors associated with the alerts that impacted physician review of patient information (eg, review of EHR information related to prescribed medications, laboratory test results, and laboratory orders) or physician orders for medications or laboratory tests.

    Identifying Primary Care Physicians’ Actions in the Electronic Health Record Through Use of Access and Audit Logs

    We analyzed EHR log data corresponding to the 2-day period following opening of 799 alerts (593 information only, 37 medication recommendations, and 169 test recommendations) sent to 75 PCPs’ InBaskets. These three types of alerts highlighted new medication concerns for older patients posthospital discharge; all were intended to prompt review of the recently hospitalized patient’s chart within the EHR by the PCP. Expected actions for test recommendations included viewing test results or ordering tests. Expected actions for medication recommendations included viewing medication lists and discontinuing or ordering medications. For information-only alerts, appropriate action was left to the judgment of the PCP, but expected actions were similar to those for medication recommendations and included performance of medication reconciliation, which would entail viewing medication lists and possibly discontinuing or ordering medications. In all cases, reviews of other parts of the patient chart could be expected in support of information gathering and decision making.

    In order to retrieve data on the 2-day period following alert opening, the data manager needed to first identify the location and format of data generated by the locally developed alert system. This work was accomplished through careful coordination between a physician with EHR access and a data manager. Simulated patients were created, and alerts were triggered for these simulated patients. Following this, a collaborating physician from the medical group opened the alerts and then opened and viewed various sections within the EHR corresponding to the simulated patient’s chart. The data manager used the collaborating physician and simulated patient identifiers together with the dates and times of triggering activities to locate these actions within the EHR’s logs and tables. This allowed her to identify the code generated by the locally developed alert system, indicating that an alert had been sent. From that base, the unique identifiers generated for each delivered alert could be extracted. Using the audit log table, we then obtained time stamps for each alert opened (hour, minute, and second) and an identifier for the person opening the alert in order to track actions. Alerts could be opened by the PCP to whom they were addressed or other members of the care team.

    Immediate Actions

    We identified the first time the alert was opened by the addressed PCP. The InBasket view of the EHR version used during the study period only displayed the alert message and several clickable buttons that served as direct links to summaries of sections of the corresponding patient’s chart; thus, navigation to other sections of the EHR was required to gain any additional information. After opening the alert, some PCPs navigated to sections of the EHR for the patient triggering the alert, whereas some moved on to a different alert or the electronic record for a different patient. Using time stamps, we captured the time spent by the PCP on the alert itself, and we captured the total time spent on the relevant patient. Total time included the duration during which the alert was displayed in the EHR plus the time the PCP spent navigating sections of the EHR corresponding to the relevant patient.

    Using the EHR access log table, we identified each action taken by the provider during the 5-minute period immediately following alert opening. Data elements from each action provided information on the exact time (hour, minute, seconds) of the action as well as codes indicating the specific section of the EHR opened and the patient to whom it related. Codes indicating the component of the EHR accessed by the PCP were complex, requiring additional physician-data manager collaboration to categorize. Actions that we considered as viewing relevant patient information included opening a section of the electronic medical record (medications, laboratory, orders, results, encounters, demographic information, other clinical information, nonclinical information, and information entry) or choosing one of several options on the alert that served as direct links to summaries of components of that patient’s record. Actions that we considered as not viewing relevant patient information included opening a notification related to a different patient, opening a section of a different patient’s medical record, or doing nothing further in the EHR for 5 minutes.

    Subsequent Relevant Actions

    Considering that PCPs might be opening alerts briefly in between patient visits and might return to the patient record later to address issues raised in the alert, we sought to broaden the window for tracking PCP’s actions. For our analysis of subsequent relevant actions taken by the PCP, the time window studied included the day of alert opening and the following day. Extracted data included a timestamp as well as codes indicating the EHR component accessed (categorized in the same manner as with immediate action analyses described above). In addition to the information extracted from the access log table, we captured evidence of orders placed for medications (new medication, change, or discontinuation) and laboratory tests. We focused only on actions taken in the EHR for the patient of interest (eg, the patient for whom the alert was triggered).

    We categorized PCP actions under the broad heading of general action (includes viewing any patient data, creating documentation, and placing any orders). Within this broad group, we defined subcategories: medication-specific action (viewing the patient’s medication list or ordering a medication) and laboratory-specific action (viewing or ordering laboratory tests).

    Contextual Factors

    We also examined contextual factors potentially relevant to alert opening and subsequent EHR action. These included the number of messages and unopened messages in the physician’s InBasket as well as the flow of messages during the prior week. Capturing a snapshot of the EHR status at a historical point in time (eg, number of InBasket messages at the time of alert delivery) presented challenges. To recreate the InBasket at the moment of alert delivery, we began by assembling all the messages delivered to a specific physician’s InBasket during the 1-year period prior to alert delivery. We then eliminated any messages for which the record indicated that the message was “completed” (eg, deleted from the InBasket) prior to the date of alert delivery. This provided a count of the number of messages remaining in the InBasket. To calculate the number of messages that remained unopened at the time of alert delivery, we identified those for which there was no log entry indicating opening on or prior to the alert delivery date. We also calculated the total number of notifications that arrived in the InBasket during the week prior to alert delivery.

    Data Analysis

    Immediate Actions

    For assessment of the immediate response during the 5 minutes after alert opening, we focused only on those cases in which the physician to whom the alert was sent was the first person to open the alert. We categorized the immediate next action by the physician as one related or not related to the relevant patient. We used bivariate analyses to assess the relationship between performing an immediate relevant action and alert type, timing of alert opening relative to alert delivery, provider and patient characteristics, and contextual factors. We performed multivariate analyses to obtain odds ratios (OR) in the presence of more than one variable. Since some PCPs received multiple alerts over the 1-year period, we used generalized estimating equations with a logit link and a binomial distribution to account for clustering of measures within PCPs. To estimate the total time attributable to the alert in the 5 minutes following opening, we calculated the length of time spent by the PCP with the alert open (ie, displayed on the EHR computer screen) and combined this with the total time (immediately following alert opening) spent viewing sections of the EHR for that patient.

    Subsequent Relevant Actions

    When assessing factors related to relevant subsequent actions, we included all first openings of each alert irrespective of whether the addressed physician or a covering physician or staff member opened the alert, but tracked behavior of the addressed physician. Similar to the process for the immediate actions, we performed bivariate and multivariate analyses using generalized estimating equation models to assess the relationship between performing subsequent relevant actions and the covariates mentioned above. We also assessed the effect of alert opening by the addressed physician on the likelihood of subsequent actions.



    In the overall group of 799 alerts opened, 627 (78.5%) were opened by the addressed PCP. Of this subgroup, we were able to track actions immediately following PCP alert opening for 616 alerts (11 alerts had no available data). The analysis was then expanded to include the remaining 172 (21.5%) alerts opened by staff or a provider other than the PCP to track actions over the day of alert opening and the following day. Alerts that did not show evidence of action within this timeframe were classified as alerts not followed by timely PCP action (Figure 1).

    Immediate Actions

    Among the 616 tracked alerts, 208 (33.8%) were immediately followed by viewing of the relevant patient’s EHR (Table 1). Of 445 information-only alerts, 28.1% (125/445) were followed by immediate viewing of the patient’s EHR, as were 54.8% (17/31) of the medication-recommendation alerts and 47.1% (66/140) of the test-recommendation alerts.

    In multivariate analyses predicting immediate viewing of the patient’s electronic information, the only factor that reached statistical significance was the type of alert. As compared to information-only alerts (reference group), the odds ratio (OR) of immediate action for medication-recommendation alerts was 4.03 (95% CI 1.67-9.72); the OR for test-recommendation alerts was 2.14 (95% CI 1.38-3.32). In the 5 minutes postopening, the mean time PCPs spent on the patient of interest—with the alert on display and while navigating relevant patient’s EHR—was 106 seconds (median, 64 seconds). Alerts not immediately followed by viewing of the relevant patient’s EHR were kept on display for a mean time of 26 seconds (median, 15 seconds; Table 2).

    Figure 1. Overview of alert opening. Asterisk indicates that 11 alerts had no data available on immediate action. PCP: primary care physician.
    View this figure
    Table 1. Immediate electronic health record actions taken by primary care providers after opening noninterruptive alerts. We identified the first time the alert was opened by the addressed primary care physician and extracted from the electronic health record access log table data on the first action taken by the provider during the 5-minute period immediately following alert opening.
    View this table
    Table 2. Time spent by the primary care physician in electronic charts of patients triggering an alert during the 5 minutes postalert opening.
    View this table

    Subsequent Relevant Actions

    In the overall group of 799 alerts opened, 538 (67.3%) were followed by PCP-related actions during the day of opening or the next day (Table 3). Among information-only alerts, 64.4% (382/593) were followed by a general action in the EHR; 16.0% (95/593), by a medication-specific action; and 16.9% (100/593), by a laboratory-specific action. Among medication-recommendation alerts, 78.4% (29/37) were followed by a general action; 48.6% (18/37), by a medication-specific action; and 54.1% (20/37), by a laboratory-specific action. Among test-recommendation alerts, 75.1% (127/169) were followed by a general action, 24.3% (41/169) were followed by a medication-specific action, and 35.5% (60/169) were followed by a laboratory-specific action.

    On multivariate analysis (Table 3), alerts containing specific instructions for the PCP (medication recommendations or test recommendations) were significantly more likely to be associated with subsequent action by the PCP compared to information-only alerts. Compared to information-only alerts, the ORs of medication-specific action were significantly greater for medication recommendations (OR: 5.59; 95% CI 2.42-12.94) and test recommendations (OR: 1.71; 95% CI 1.09-2.68). Likewise, the ORs of laboratory-specific action were significantly greater for medication recommendations (OR: 7.37; 95% CI 3.64-14.97) and test recommendations (OR: 2.75; 95% CI 1.73-4.38).

    There was no significant association between the timing of alert opening and responsive action. As expected, subsequent relevant action by the PCP was more likely for alerts opened by the addressed PCP as compared to alerts opened by staff or another provider, although only the association with medication-specific action reached statistical significance (general action, OR: 1.41; 95% CI 0.81-2.45; medication-specific action, OR: 2.13; 95% CI 1.35-3.37; laboratory-specific action, OR: 1.65; 95% CI 0.99-2.73).

    Table 3. Multivariate analysis results for factors associated with primary care providers’ subsequent action following opening of noninterruptive alerts. Alerts were opened by the addressed primary care provider, staff, or provider other than the addressed primary care provider. The primary care provider’s actions in the electronic health record related to the relevant patient were tracked over the day of alert opening and the following day.
    View this table

    Contextual factors showed no consistent patterns. There was some suggestion of a stronger likelihood for responsive action if the relevant patient had more than 11 office visits in the past year. Responsive actions were also more likely taken by physicians with more patient encounters and less likely by physicians with total number of InBasket notifications at the time of alert delivery in the top quartile (>157 alerts).

    Electronic Health Record Access and Audit Logs


    We were able to identify all the data that were required to track provider behavior in the EHR access and audit logs; however, a substantial effort was required to identify and interpret these data. We encountered two main challenges in data retrieval and interpretation. As detailed below, these challenges pertained to capturing a historical snapshot (eg, number of InBasket messages at a given time in the past) and interpreting data elements from access and audit logs in the absence of clear data documentation.

    Capturing a Historical Snapshot of Electronic Health Record Status

    Our access logs do not keep successive records of evolving EHR elements (examples include InBasket message lists or medication lists, both of which have elements added and deleted over time). Thus, in order to capture the status of a provider’s InBasket retrospectively, we had to retrieve data generated many months prior and couple this with longitudinal follow-up data to determine the status at the time of interest. Specifically, to recreate a list of InBasket messages present at the time of alert delivery, we obtained an estimate of the date on which the earliest arriving messages were generated (we went back 1 year) and then captured all the messages generated from that point forward. We honed this large list using status data reflecting whether the message was opened and whether it had been “completed” (eg, deleted) prior to the date of alert delivery. Messages that were marked as completed (and those postponed to return at a future date) were removed; the remaining were concluded to be messages present in the InBasket at the time of alert delivery.

    Interpretation of Data Elements Required an Iterative Approach With Physician-Data Manager Collaboration

    We encountered challenges in deciphering the variable names for data elements drawn from the audit logs. These data had not previously been used for research purposes, and therefore, we lacked clear documentation for the interpretation of many variables. Accurately interpreting data elements required iterative work by the physician/data manager team. Using a simulated patient record in the EHR, our physician collaborator performed actions through the physician user interface and then worked with the data manager to compare results captured simultaneously through the audit log. Both a physician’s perspective (providing insight into routine actions taken by physicians within an EHR) and the data manager’s informatics knowledge (and access to data tables) were essential for this effort.


    Our descriptive study assessed the use of EHR log data to track provider behavior following opening of noninterruptive medication alerts. More than three-quarters of alerts (78.5%; 627/799) were opened by the PCP to whom they were directed, allowing us to assess immediate PCP action; of these, 208 alerts were followed by immediate action. Expanding our analysis to include alerts opened by staff or covering physicians, we found that an additional 330/799 alerts showed evidence of action by the end of the following day. The remaining 261 alerts showed no evidence of PCP action. Compared to information-only alerts, alerts containing specific instructions (including medication and test recommendations) were significantly more likely to be followed by EHR action; these alerts were also more likely to prompt medication-specific action such as medication ordering, changing, and discontinuing or medication list viewing.

    Not all alerts were followed by provider action, indicating that other factors may have played a role in providers’ decision to act on information or recommendations provided in the alerts.

    Data from access and audit logs provide a rare “behind-the-scenes” glimpse into how health care team members spend their time and where they direct their attention. These log data have been used successfully to understand questions of the health care team’s communication [26], trainee skill [27], and clinical workflow [6,7]. As we demonstrate in this paper, EHR logs can also provide valuable implementation data that can be used to improve future CDS interventions. Log data can help a research team understand whether a CDS message was opened by the addressed recipient, how long it was viewed for, and what kind of actions followed the viewing.

    There are numerous ways a research team can benefit from this knowledge. Our finding that few physician, patient, and contextual aspects impacted the opening of and action on alerts is reassuring. It does not appear that subgroups of PCPs or patients would require varying approaches for alert-based communication. Tracking rates of opening for noninterruptive messages can identify a need for improved safety measures (eg, escalation of unopened time-sensitive messages). Understanding who opens these messages lends insight into a care team’s triage processes, which might alter the design of the intervention (eg, physician-directed CDS sent to a team where all messages are opened initially by nurses might prompt the team to modify message content or change the method of communication) [4]. Realizing that providers spend an extremely short period of time with alerts open might prompt those designing these alerts to adapt messages accordingly (eg, include fewer words and optimize visibility of key text). For the intervention discussed in this paper, we preceded implementation by having two of the group practice primary care physicians review all messages generated by the system for four months. Of the tested alert types, the reviewers selected those included in the final intervention with the goal that all alerts would be necessary and actionable. For the medication information-only alerts, which focused specifically on new or high-risk medications, the reviewers were certain that recipients would review the full medication list for these patients. Alerts that were categorized as information-only may have been perceived by PCPs as not requiring further action, since they did not include specific recommendations. The brief viewing time and low level of activity after viewing information-only alerts suggests that this type of alert should be reconsidered before including them in future interventions. Tracking patterns of EHR behavior after alert opening might suggest useful shortcuts to make the alert more user-friendly (eg, if users routinely navigate to the medication list after opening the alert, providing a link directly to this destination might enhance the user experience). In sum, to understand why an EHR–based CDS intervention is (or is not) effective, researchers should be asking whether that intervention was delivered as planned and what subsequent steps were taken by recipients after intervention delivery.

    In this descriptive study, we focused on use of access and audit logs for understanding the impact of noninterruptive alerts. This approach is particularly necessary for studying noninterruptive alerts, which are usually opened outside of a patient encounter and thus present challenges for those seeking to pinpoint the timing of downstream actions. Access and audit log data can also be applied to a variety of questions pertaining to transmission of EHR–based information. Examples include tracking of information viewing (eg, understanding when a care team member opens and reads a clinic note, laboratory or test report, or other updated EHR information) as well as tracking the timing of documentation and ordering.

    Access and audit logs could also be helpful in trying to reduce the InBasket burden for physicians. Log data can reveal which types of alerts and notifications physicians act upon; those that are commonly left unopened or are not generally followed by any action could be candidates for elimination or modification. These data can also inform enhancements to the structure of the InBasket. Unopened or unacted upon messages can be moved to and organized within separate folders to which the providers can return at a later time.

    In addition to the two challenges identified here (difficulties in capturing a historical snapshot from the EHR and difficulty in interpretation of data elements in the absence of clear documentation), there are additional issues to consider when using access and audit log data. Site-to-site variability in the use of certain variables may introduce differences in interpretation of audit logs across sites. Data managers may not have privileges granted to retrieve access or audit log data. The large volume of data generated through access and audit logs may necessitate policies for limitations in what a site stores (and for how long). Understanding the local situation at an early stage of research planning may facilitate use (for instance, planning to download log data before they are deleted at a site that stores them for only a limited period). Planning ahead and setting up log data to provide real-time information as an intervention is implemented can make the process much simpler.

    There are limitations to this study. We may not have captured all PCP actions after alert opening relevant to the postdischarge patient, such as actions taken outside of the EHR. The age of the data (from a trial conducted in 2011) may not accurately reflect current provider workflow in upgraded versions of the EHR with respect to opening of alerts and responsive actions. Direct causality (EHR action due to alert opening) is not proven simply by demonstrating close temporal proximity. Our study is descriptive and could be supported by future qualitative work examining the physicians’ own perspectives on their EHR actions following alert opening.

    An EHR–based alert system intervention alone does not improve clinical patient outcomes in high-risk populations. Nonetheless, the availability of information in the EHR can benefit similar studies trying to understand the link between provider behavior in the EHR and patient care and outcomes. Although analysis of EHR logs presents many challenges, data from these logs can provide researchers with insight into designing impactful EHR–based CDS and alert interventions.


    This study was funded by a grant from the Agency for Healthcare Research and Quality (AHRQ) under award number 5R21HS023661. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health (NIH).

    The views expressed in this article are those of the authors and do not represent the views of the US Department of Veterans Affairs or the United States Government.

    Conflicts of Interest

    None declared.


    1. Boxwala AA, Kim J, Grillo JM, Ohno-Machado L. Using statistical and machine learning to help institutions detect suspicious access to electronic health records. J Am Med Inform Assoc 2011;18(4):498-505. [Medline]
    2. Menon AK, Jiang X, Kim J, Vaidya J, Ohno-Machado L. Detecting Inappropriate Access to Electronic Health Records Using Collaborative Filtering. Mach Learn 2014 Apr 01;95(1):87-101 [FREE Full text] [CrossRef] [Medline]
    3. Freeman M. Miller Weisbrod. Dallas, TX: Miller Weisbrod, LLP Electronic Records and Audit Trails   URL: [accessed 2019-01-30] [WebCite Cache]
    4. Cutrona SL, Fouayzi H, Burns L, Sadasivam RS, Mazor KM, Gurwitz JH, et al. Primary Care Providers' Opening of Time-Sensitive Alerts Sent to Commercial Electronic Health Record InBaskets. J Gen Intern Med 2017 Nov;32(11):1210-1219. [CrossRef] [Medline]
    5. Tai-Seale M, Olson CW, Li J, Chan AS, Morikawa C, Durbin M, et al. Electronic Health Record Logs Indicate That Physicians Split Time Evenly Between Seeing Patients And Desktop Medicine. Health Aff (Millwood) 2017 Dec 01;36(4):655-662 [FREE Full text] [CrossRef] [Medline]
    6. Hirsch AG, Jones JB, Lerch VR, Tang X, Berger A, Clark DN, et al. The electronic health record audit file: the patient is waiting. J Am Med Inform Assoc 2017 Apr 01;24(e1):e28-e34. [CrossRef] [Medline]
    7. Hribar MR, Read-Brown S, Reznick L, Lombardi L, Parikh M, Yackel TR, et al. Secondary Use of EHR Timestamp data: Validation and Application for Workflow Optimization. AMIA Annu Symp Proc 2015;2015:1909-1917 [FREE Full text] [Medline]
    8. Bowes WA. Measuring use of electronic health record functionality using system audit information. Stud Health Technol Inform 2010;160(Pt 1):86-90. [Medline]
    9. Lo HG, Matheny ME, Seger DL, Bates DW, Gandhi TK. Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc 2009;16(1):66-71 [FREE Full text] [CrossRef] [Medline]
    10. Singh H, Thomas EJ, Mani S, Sittig D, Arora H, Espadas D, et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med 2009 Sep 28;169(17):1578-1586 [FREE Full text] [CrossRef] [Medline]
    11. Singh H, Arora HS, Vij MS, Rao R, Khan MM, Petersen LA. Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007;14(4):459-466 [FREE Full text] [CrossRef] [Medline]
    12. Singh H, Thomas EJ, Sittig DF, Wilson L, Espadas D, Khan MM, et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? Am J Med 2010 Mar;123(3):238-244 [FREE Full text] [CrossRef] [Medline]
    13. Slovis BH, Nahass TA, Salmasian H, Kuperman G, Vawdrey DK. Asynchronous automated electronic laboratory result notifications: a systematic review. J Am Med Inform Assoc 2017 Nov 01;24(6):1173-1183. [CrossRef] [Medline]
    14. Dalal AK, Roy CL, Poon EG, Williams DH, Nolido N, Yoon C, et al. Impact of an automated email notification system for results of tests pending at discharge: a cluster-randomized controlled trial. J Am Med Inform Assoc 2014;21(3):473-480 [FREE Full text] [CrossRef] [Medline]
    15. O'Connor SD, Dalal AK, Sahni VA, Lacson R, Khorasani R. Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? J Am Med Inform Assoc 2016 Mar;23(2):333-338 [FREE Full text] [CrossRef] [Medline]
    16. Lacson R, O'Connor SD, Andriole KP, Prevedello LM, Khorasani R. Automated critical test result notification system: architecture, design, and assessment of provider satisfaction. AJR Am J Roentgenol 2014 Nov;203(5):W491-W496 [FREE Full text] [CrossRef] [Medline]
    17. Abujudeh HH, Kaewlai R, Choy G, Whelton DG, Rosenthal DI. Important imaging finding e-mail alert system: experience after 3 years of implementation. Radiology 2009 Sep;252(3):747-753. [CrossRef] [Medline]
    18. Benthin C, Pannu S, Khan A, Gong M, NHLBI Prevention Early Treatment of Acute Lung Injury (PETAL) Network. The Nature and Variability of Automated Practice Alerts Derived from Electronic Health Records in a U.S. Nationwide Critical Care Research Network. Ann Am Thorac Soc 2016 Dec;13(10):1784-1788 [FREE Full text] [CrossRef] [Medline]
    19. Gurwitz JH, Field TS, Ogarek J, Tjia J, Cutrona SL, Harrold LR, et al. An electronic health record-based intervention to increase follow-up office visits and decrease rehospitalization in older adults. J Am Geriatr Soc 2014 May;62(5):865-871 [FREE Full text] [CrossRef] [Medline]
    20. Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med 2012 Feb;125(2):209.e1-209.e7. [CrossRef] [Medline]
    21. Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record-based settings. JAMA Intern Med 2013 Apr 22;173(8):702-704 [FREE Full text] [CrossRef] [Medline]
    22. Murphy DR, Meyer AND, Russo E, Sittig DF, Wei L, Singh H. The Burden of Inbox Notifications in Commercial Electronic Health Records. JAMA Intern Med 2016 Apr;176(4):559-560 [FREE Full text] [CrossRef] [Medline]
    23. Gregory ME, Russo E, Singh H. Electronic Health Record Alert-Related Workload as a Predictor of Burnout in Primary Care Providers. Appl Clin Inform 2017 Jul 05;8(3):686-697 [FREE Full text] [CrossRef] [Medline]
    24. Field TS, Garber L, Gagne SJ, Tjia J, Preusse P, Donovan JL, et al. Technological resources and personnel costs required to implement an automated alert system for ambulatory physicians when patients are discharged from hospitals to home. Inform Prim Care 2012;20(2):87-93 [FREE Full text] [Medline]
    25. Tjia J, Field TS, Garber LD, Donovan JL, Kanaan AO, Raebel MA, et al. Development and pilot testing of guidelines to monitor high-risk medications in the ambulatory setting. Am J Manag Care 2010 Jul;16(7):489-496 [FREE Full text] [Medline]
    26. Gray JE, Feldman H, Reti S, Markson L, Lu X, Davis RB, et al. Using Digital Crumbs from an Electronic Health Record to identify, study and improve health care teams. AMIA Annu Symp Proc 2011;2011:491-500 [FREE Full text] [Medline]
    27. Chen ES, Cimino JJ. Automated discovery of patient-specific clinician information needs using clinical information system log files. AMIA Annu Symp Proc 2003:145-149 [FREE Full text] [Medline]


    AOR: adjusted odds ratio
    CDS: clinical decision support
    EHR: electronic health record
    N/A: not applicable
    OR: odds ratio
    PCP: primary care physician

    Edited by G Eysenbach; submitted 30.10.18; peer-reviewed by D Carvalho, D Murphy; comments to author 21.11.18; revised version received 14.01.19; accepted 20.01.19; published 07.02.19

    ©Azraa Amroze, Terry S Field, Hassan Fouayzi, Devi Sundaresan, Laura Burns, Lawrence Garber, Rajani S Sadasivam, Kathleen M Mazor, Jerry H Gurwitz, Sarah L Cutrona. Originally published in JMIR Medical Informatics (, 07.02.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.