We developed and tested multiple search strategies that incorporated subject heading and keyword terms for “interoperable electronic health record” (including “iEHR”), “health information exchange” or “HIE”; “interoperability”; “adoption” or “use”; and “effectiveness,” “impact,” or “value.” We consulted and sought feedback on the search strategies with health informatics experts, including from academic and government agencies each focused on health informatics, and a specialist librarian who performed an abbreviated Peer Review of Electronic Search Strategies (PRESS) assessment of a preliminary MEDLINE search strategy [6]. The health informatics experts identified five articles that reflected the targeted aims of the review, which we used to test the effectiveness of candidate search strategies to identify relevant articles. Ultimately, a final search strategy was selected and translated for use in several traditional databases—MEDLINE, Embase, and the Cumulative Index to Nursing and Allied Health Literature (CINAHL)—with supplemental searches conducted in Google Scholar and grey literature sources (ie, Google, Grey Literature Report, and OpenGrey); see Multimedia Appendix 1 for database-specific search strategies. All searches were conducted in January and February 2017. Additionally, reference lists of all included articles were reviewed to identify additional articles that the search strategy missed.

Articles were eligible for inclusion in the review if they were published in English; published between 2006 and 2017, although, only a small number of articles published in 2017 were available at the time of our review; and were either an original research study, inclusive of both quantitative and qualitative study types, or a literature review. For inclusion in the review, articles needed to focus on iEHRs and HIEs, excluding electronic medical records, across two or more health care settings; they also needed to focus on the impact or effectiveness of iEHR or HIE adoption or use.

Two reviewers (JPB and one of three research assistants) independently screened all titles and abstracts or, in the case of Google and Google Scholar search results, titles and excerpts. Discrepancies were resolved through discussion and/or review by a third reviewer (MJD). The same process was used for review of full-text articles.

Data from all articles identified for inclusion were extracted independently by two reviewers (JPB and one of three research assistants) using a predeveloped data extraction template that included the year of publication, article type, approach, methodology, jurisdiction, and health care setting. Given the anticipated heterogeneity of study types and the intention to capture the breadth of iEHR evaluation activity, we prioritized evaluation relevance over quality and, therefore, did not use available quality criteria to exclude primary studies [7-9].

For all primary studies, two reviewers (JPB and one of three research assistants) independently extracted distinct measurement items and measurement outcomes verbatim. Measurement items were then coded and recoded inductively over three iterations into thematic categories by one reviewer (JPB) in discussion with the review team. All measurement outcomes were categorized as positive, negative, or mixed or neutral by one reviewer (JPB) and reviewed by the review team. All thematic measurement item categories were classified as one of the eight evaluation dimensions of benefit—system quality, service quality, information quality, user satisfaction, use, productivity, quality, and access—based on the Infoway benefits evaluation framework (see Figure 1). This framework, based on the Delone and MacLean Information Systems Success Model [10], details the measurement item and outcome categories and has been used extensively across Canada and internationally since it was first published in 2007 [11]. This classification approach has also been used in a relevant recent review of project evaluations from electronic health record (EHR) implementations across Canada [12].

We employed a separate analysis approach for the included literature review articles, conducting a descriptive analysis of each review article that assessed the (1) main focus of the review, (2) main findings, and (3) recommendations for future research.

Our search of seven data sources identified 3851 records. After deduplication; title, abstract, and excerpt screening; and full-text review, 130 articles were included in the synthesis; Figure 2 presents the PRISMA flow diagram for the review. These 130 articles included 113 journal articles (86.9%), 11 reports (8.5%), and 6 documents of other types (4.6%). Of the 130 articles, 113 were primary studies (86.9%) and 17 were various types of reviews (13.1%). The vast majority of the articles focused on the United States (88/130, 67.7%); 6 (4.6%) focused on Israel; 3 (2.3%) each focused on Canada, Finland, and the United Kingdom; 2 (1.5%) focused on South Korea; 7 (5.4%) focused on another single country, including Australia, Austria, Brazil, Greece, Kenya, the Netherlands, and Switzerland; and another 18 (13.8%) had a multi-country focus. Of the 130 articles, 95 (73.1%) were published in the 6-year period from 2012 to 2017.

For the 113 primary studies, the majority employed quantitative methodologies exclusively (71/113, 62.8%) or in combination with qualitative methods (ie, mixed-method approaches) (20/113, 17.7%). The primary studies focused on a wide range of health care settings; the three most prevalent settings studied included acute care (59/113, 52.2%), primary care (44/113, 38.9%), and emergency departments (34/113, 30.1%). Other settings included laboratories (18/113, 15.9%), ambulatory care (17/113, 15.0%), pharmacies (13/113, 11.5%), public health departments (14/113, 12.4%), long-term care (9/113, 8.0%), payer or purchaser organizations (9/113, 8.0%), and home and community care (3/113, 2.7%).

The primary studies focused on six general dimensions. We identified 29 distinct measurement items, representing evaluation themes, in the 113 primary studies that were linked to 522 specific measurement outcomes (see Table 1). Productivity and quality were the two evaluation dimensions that received the most attention in the articles we reviewed, accounting for 14 of 29 (48%) measurement items identified and 306 of 522 (58.6%) measurement outcomes identified. Of the six evaluation dimensions assessed, service quality and system quality received the least attention, accounting for only 5 of 29 (17%) measurement items and 79 of 522 (15.1%) measurement outcomes documented from the articles reviewed. Measurement items were not assigned to either the use or access evaluation dimensions.

Overall, the majority of the 522 measurement outcomes were positive (298/522, 57.1%), with the remaining measurement outcomes reported as neutral or mixed results (107/522, 20.5%) or negative findings (117/522, 22.4%). When examining each of the 29 measurement items separately, the majority (22/29, 76%) had more positive than negative measurement outcomes, with the most frequently studied measurement items having a larger proportion of positive over negative outcomes. The 5 measurement items (5/29, 17%) with more negative than positive measurement outcomes were (1) stakeholder engagement, (2) performance and reliability, (3) security and privacy, (4) overall quality of information, and (5) ease of use; 2 measurement items (2/29, 7%) had equal positive and negative measurement outcomes: (1) layout and format and (2) community-based care, public or population health, or preventive services.

To provide more details on the findings, we consider each of the six dimensions separately. A total of 2 measurement items (2/29, 7%) were aligned with the service quality dimension. These included stakeholder engagement, which has mostly negative results, and training and support, which has mostly positive results. For the system quality dimension, 3 of 29 (10%) measurement items applied, including performance and reliability, security and privacy, and assessment and planning. Of these 3, only the latter had positive measurement outcomes. The information quality dimension had 6 of 29 (21%) measurement items, with the 2 most frequently measured items—data accuracy and completeness, as well as information availability—mostly positive, while the 4 less frequently measured items each revealed equivocal results. A total of 4 of 29 measurement items (14%) aligned with the user satisfaction dimension, with 1 showing mostly positive measurement outcomes—perceived usefulness or value and trust or confidence in the system—while the remaining 3 measurement items showed measurement outcomes that were either equivocal or negative. The productivity and quality dimensions each had 7 of 29 (24%) measurement items. All 7 measurement items for productivity had positive measurement outcomes, while 6 of the 7 measurement items for quality also had positive measurement outcomes. As noted above, the productivity and quality dimensions have received the majority of focus for measurement and have yielded mostly positive outcomes.

When looking at the results from a setting-specific perspective, where sufficient volumes existed, there were some notable differences from the overall results. Acute care settings were assessed by 59 out of 113 studies (52.2%) and represented 270 out of 522 (51.7%) distinct measurement outcomes. Of these, there was focus on each of the six dimensions, with considerable attention on service quality, system quality, and some aspects of productivity. Primary care settings were assessed by 44 out of 113 (38.9%) studies and represented 183 out of 522 (35.1%) distinct measurement outcomes that covered most of the six dimensions, with attention directed predominantly to productivity measures while service, system, and information quality received less focus. Emergency department settings were assessed by 34 out of 133 (30.1%) studies and represented 112 out of 522 (21.5%) distinct measurement outcomes. There was a lack of measurement outcomes for most of the six dimensions, with the exception of one type of productivity item and one type of quality item.

We identified 17 reviews on iEHR use or impact (see Table 2). Of these reviews, 6 (35%) focused on barriers and facilitators to adoption or implementation, and 11 (65%) focused on benefits or impacts. A total of 10 of the 17 reviews (59%) were published between 2013 and 2016 and, with the exception of 3 reviews (18%) that were limited in scope to clinical research [13], chronic disease [14], or ambulatory primary care [15], most reviews (14/17, 82%) examined general benefits or impacts of iEHRs or HIEs.

The more recent reviews (ie, published since 2015) found little generalizable evidence of benefit or impact. The reviews highlight some less methodologically robust research that focused on resource use and perception of outcomes; these reviews found that iEHRs or HIEs increase productivity (eg, reduction in duplicate testing, emergency department costs, or hospital admissions [16,17]) and are valued by patient and physician stakeholders [17], all of which is consistent with the findings from our review. However, authors of one of the recent reviews [18] cautioned on overinterpreting the generalizability of this work given the methodological limitations of the primary studies and the developing state of iEHR or HIE evaluative work overall. The recent reviews on the benefit or impact of iEHRs or HIE identify three main areas for future research, including the following: (1) focus on how the setting in which iEHRs or HIEs are used affects specific health care outcomes [18]; (2) use of more rigorous, coordinated, and systematic approaches to evaluate the relationship between iEHRs or HIEs and health care outcomes [16]; and (3) need for better understanding of the organizational factors that affect iEHR or HIE contributions to improved clinical care [17].

Consideration of the review results against the benefits evaluation framework provides a lens to assess where evaluative work has been targeted and where there may be gaps where future evaluative efforts should focus. A total of 57.1% (298/522) of all measurement outcomes were positive. Quality of care (88/136, 64.7%) and productivity (109/170, 64.1%) were the dimensions with the highest percentage of positive measurement outcomes. Prominent themes in the quality of care category were around coordination of care and clinical decision support. For the productivity dimension, efficiency in clinical processes, time savings, and costs were the prominent themes. The left side of the benefits evaluation framework (see Figure 1), including system, service, and information quality, as well as user satisfaction, had relatively lower proportions of positive measurement outcomes ranging from 39%-53%. Many of the factors critical to achieving quality and productivity benefits require concentrated efforts on the left side of the framework. Change management efforts and other studies evaluating the benefits of iEHRs and other information systems suggest that user satisfaction increases when users have access to high-performing technology that is well integrated into their workflow, interoperable with existing systems, and is able to provide them with the information they need when they need it [2,12,30,31]. In addition, appropriate levels of support and training are necessary to ensure use of information systems.

Overall, our findings suggest that positive results tend to attract more evaluation, which may be explained by efforts to use progressively more rigorous methodologies, but also may reflect inefficient allocation of limited evaluation resources that could be applied to less-studied aspects of iEHRs and HIEs. The findings also point to several broader evaluation dimensions and several specific measurement items that require more attention going forward, including use and access, for which we did not identify any measurement items, and the service quality, system quality, user satisfaction, and information quality dimensions, which had fewer measurement items than productivity and quality dimensions. It is important to note that we did not perform quality appraisals of the primary studies; therefore, these review results need to be interpreted cautiously, which is a general theme of the reviews we assessed. While promising work exists, there is a clear need for more rigorous and comprehensive evaluation, with priority to support methodologies that can produce high-quality evidence. Overall, the review findings highlight the need to support more robust and comprehensive evaluative work across Canada on the impact of connected health information, covering more disease domains, health care settings, and populations.

This review identified a large number of studies that address the use and impact of connected health information through iEHRs and HIEs. The majority of the studies have been published within the last 5 years, which reflects a developing rather than mature evidence base. Given that the bulk of this evidence base is current, concerns regarding potential temporal biases that might not accurately reflect the quickly evolving developments of iEHRs and HIEs should be limited. However, our analysis did not assess whether systematic temporal differences in dimension-specific evaluations were present (eg, service and quality evaluated sooner after system launch vs productivity evaluated at more mature stages following launch). Consistent with the developing nature of the evidence base, it is notable that a sizable proportion of the articles identified in this review (12/130, 9.2%) came from grey literature sources, highlighting broader contributions to iEHR and HIE evaluation. Beyond the United States, which is the focus of the vast majority of the primary studies identified, we found few other primary studies globally, with Israel a distant second in terms of evaluative work on iEHRs and HIEs, followed by Canada, Finland, and the United Kingdom. Given fundamental differences in the organization of health systems and services and the dearth of evaluations in non-US settings, there are limits on how generalizable these assessments across jurisdictions will be.

In conclusion, this review captures evaluative work from an evolving and active field focused on the use and impact of iEHRs and HIEs. While the overall findings suggest many positive impacts of iEHRs and HIEs, the quality of the primary studies were not evaluated systematically. When broken down by specific measurement items, some measurement outcomes consistently presented positive outcomes, while others were mostly negative or equivocal, highlighting areas for more attention. Setting-specific findings provide further insight on where more evaluative attention is needed.