This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.
Electronic health (eHealth) interventions may improve the quality of care by providing timely, accessible information about one patient or an entire population. Electronic patient care information forms the nucleus of computerized health information systems. However, interoperability among systems depends on the adoption of information standards. Additionally, investing in technology systems requires cost-effectiveness studies to ensure the sustainability of processes for stakeholders.
The objective of this study was to assess cost-effectiveness of the use of electronically available inpatient data systems, health information exchange, or standards to support interoperability among systems.
An overview of systematic reviews was conducted, assessing the MEDLINE, Cochrane Library, LILACS, and IEEE Library databases to identify relevant studies published through February 2016. The search was supplemented by citations from the selected papers. The primary outcome sought the cost-effectiveness, and the secondary outcome was the impact on quality of care. Independent reviewers selected studies, and disagreement was resolved by consensus. The quality of the included studies was evaluated using a measurement tool to assess systematic reviews (AMSTAR).
The primary search identified 286 papers, and two papers were manually included. A total of 211 were systematic reviews. From the 20 studies that were selected after screening the title and abstract, 14 were deemed ineligible, and six met the inclusion criteria. The interventions did not show a measurable effect on cost-effectiveness. Despite the limited number of studies, the heterogeneity of electronic systems reported, and the types of intervention in hospital routines, it was possible to identify some preliminary benefits in quality of care. Hospital information systems, along with information sharing, had the potential to improve clinical practice by reducing staff errors or incidents, improving automated harm detection, monitoring infections more effectively, and enhancing the continuity of care during physician handoffs.
This review identified some benefits in the quality of care but did not provide evidence that the implementation of eHealth interventions had a measurable impact on cost-effectiveness in hospital settings. However, further evidence is needed to infer the impact of standards adoption or interoperability in cost benefits of health care; this in turn requires further research.
Information technology (IT) applied to health care, or electronic health (eHealth) [
Improvements in health and economic indicators are relevant metrics to justify IT investments. Indeed, planning and investing in IT is necessary for the efficient use of information that not only advances health care but also holds financial, social, cultural, and ethical benefits. Comparative cost-effectiveness studies guide agencies and institutions in choosing the best option for desired clinical outcomes and costs, which is the key to ensuring the sustainability of government health systems and their welfare programs [
This review analyzes systematic reviews addressing the cost benefit and effectiveness of electronic medical records (EMR), standards adoption, or interoperability to discuss the benefits, drawbacks, and lessons learned from the implementation of actions related to eHealth and serves as a reference for government representatives and stakeholders. The assessment of the involvement of government and private health institutions in the implementation and maintenance of eHealth interventions that were tested and valuated worldwide is also of interest. The study was directed by 2 questions: What evidence exists regarding the impact of computerizing applications, standards, health information exchange, or interoperability to support the quality of care or patient outcomes in hospital settings? What critical cost-benefit evidence is published to provide a clear understanding of the value of eHealth implementations?
The terms used here to describe eHealth technologies are available in
It is also important to emphasize that interoperability is usually divided into (1) syntactic interoperability: the capability of two or more systems to communicate and exchange data through specified data formats and communication protocols, and (2) semantic interoperability: the ability for data shared by systems to be understood at the level of fully defined domain concepts [
Worldwide coordinated efforts resulted in the development of standards to define an EHR as one or more repositories of actionable information by computers. The European Committee for Standardization (CEN), health level seven (HL7), International Organization for Standardization (ISO), and openEHR Foundation are nonprofit organizations dedicated to providing frameworks and standards. Terminologies, EHR specifications, and information models are proposed by these international standards organizations that support the exchange, integration, interoperability, and retrieval of electronic health information [
To better represent the meaning of standards in the primary selected systematic reviews, we adopted the generic definition for the term as: “A document adopted by consensus by a recognized entity, that provides rules, guidelines and/or features for common use, in order to obtain an optimal level of performance in a given context…” [
Economic analysis supports health care policy and organizational decision making. However, it encounters some difficulties with eHealth systems, which are as follows: constantly changing technology, inconsistent study design to manage inadequate sample sizes, the inappropriateness of conventional techniques of economic evaluation, and the problem of placing value on health and nonhealth outcomes [
This review of systematic reviews has been conducted in accordance with the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement [
On February 22, 2016, electronic searches were conducted on the MEDLINE, Cochrane Library, LILACS, and IEEE Library databases. To identify the EHR concept, standards for interoperability, and health information and its cost benefits, the search strategy was:
((“Electronic health records”[MeSH Terms] OR “Health Information Exchange”[MeSH Terms] OR (“Health Information Management”[MeSH Terms] OR (“Medical Informatics” [MeSH Terms]) AND (“Interoperability” OR “Standard of Information”)) AND (“Cost-Benefit Analysis”[MeSh] OR “Evaluation Studies”[Publication Type] OR “Program Evaluation”[MeSh] OR impact or effectiveness)
The search was limited by language of publication (English, Spanish, French, Italian, and Portuguese), studies in humans, type of study (systematic reviews and meta-analyses), and year of publication (since 2005). Two systematic reviews that satisfied the criteria were identified manually. To better define certain eHealth technology descriptions, additional sources of evidence were considered.
The inclusion criteria were as follows:
Primary impact: EMR, standards, or interoperability on cost-benefit, or
Secondary impact: EMR, standards, or interoperability on quality of care (clinical outcomes), and
Real-life reviews about interventions in in-hospital settings.
Studies in primary or secondary care scenarios, studies without the primary or secondary impact of eHealth actions, and duplications were excluded. Titles and abstracts of retrieved papers were independently screened and evaluated by 2 investigators (ZSNR and TAM). Abstracts providing insufficient information were retrieved for independent, full-text evaluation by 2 investigators to determine study eligibility. Disagreements were resolved by consensus. Additional publications were identified using the reference lists of selected manuscripts.
ZSNR prepared electronic data with paper contents abstracted using StArt software (Systematic Review System) to organize the analysis [
Subgroup 1: eHealth systems implementation without health information exchange
Subgroup 2: eHealth systems with health information exchange functionalities
The methodological quality assessment was based on the AMSTAR (a measurement tool to assess systematic reviews) checklist [
A total of 288 papers were identified during the initial research phase, which decreased to 273 after removing 15 duplicates. After applying our criteria, only six systematic reviews were included in the final analysis and data-abstraction phase. The review process is represented in
The primary cause for excluding the 20 studies was mixed or outpatient settings for eHealth interventions (11 papers of 14 excluded, 79%). The Pan American Health Organization (PAHO) conducted a review of the implementation and effective use of standards to achieve interoperability in Latin American and Caribbean countries but without direct or indirect outcomes analysis [
Evidence of the cost-effectiveness of eHealth interventions that met the criteria was identified. Only one systematic review of the six performed a meta-analysis [
Flow of information through the different phases of the systematic review.
Quality assessment ratings and characteristics of the six included systematic reviews.
Study | AMSTARa score | Funding or |
Study design | Number of |
Control group |
Meta- |
|||
Yb | Nc | CAd | N/Ae | ||||||
Thompson et al 2015 [ |
4 | 7 | 0 | 0 | Y | RCTf, pre-post studies, descriptive studies | 45 total/Meta-analysis: 26 | Pre-post implementation (paper vs system) | Y |
Cheung et al 2015 [ |
5 | 4 | 0 | 2 | NCg | RCT, quasi-experimental studies, descriptive studies | 18 | Pre-post implementation | N |
de Bruin et al 2014 [ |
2 | 7 | 0 | 2 | NC | Quasi-experimental | 26 | True infection detection by infection control experts | N |
Mapp et al 2013 [ |
1 | 7 | 1 | 2 | NC | Observational, Pilot studies | 9 | No control | N |
Li et al 2013 [ |
4 | 5 | 0 | 2 | NC | RCT, quasi-experimental studies | 6 | Patient not reported in written notes or before system | N |
Govindan et al 2010 [ |
5 | 4 | 0 | 2 | Y | Observational: accuracy of the automated method with a gold standard method | 43 | Standard chart review | N |
aAMSTAR: a measurement tool to assess systematic reviews.
bY: yes.
cN: no.
dCA: cannot answer.
eN/A: not applicable.
fRCT: randomized controlled trial.
gNC: not commissioned.
Among the included systematic reviews, only one was classified as showing an effect on eHealth implementation without electronic health information exchange (Subgroup 1), and the other five were ranked as showing effects of systems implementation with incorporated health information exchange among other electronic data sources (Subgroup 2).
Considering eHealth systems implementation without health information exchange, the review of Thompson et al [
The selected review stated a mix of electronic interventions: EHR, EMR, computerized decision support systems (CDSS), computerized provider order-entry (CPOE) and surveillance systems used by physicians, nurses, allied health professionals, and managers of health services evaluating evidence from pre-and postsystems implementation. The analysis synthesized 46 publications about systems for diagnosis, treatment, and clinical monitoring. The study included a meta-analysis extracted from 26 publications to evaluate the effects of different types of systems regarding health IT in the inpatient of intensive care unit (ICU) setting on mortality, length of stay (LOS), and cost.
Not enough evidence showed that electronic interventions can improve quality and safety of health care. The goals for secondary outcomes were the effects of health IT in the inpatient and ICU on mortality or LOS. The quality of included studies and interventions varied significantly, which was highlighted as the major limitation. Despite this, the surveillance systems had a pooled odd ratio (OR) of 0.85 (95% CI 0.76-0.94) with moderate heterogeneity, I2 of 59%.
Costs were unable to be evaluated quantitatively because the primary studies presented mixed and inconclusive results, leaving us unable to draw a definitive conclusion about cost-effectiveness. The analysis of costs was more limited than the evidence on quality and efficiency.
EHR implementation with health information exchange is a recent worldwide trend in hospital settings. A summary of the results of the systematic reviews included in subgroup 2 is presented in
Descriptive summary of the systematic reviews included in electronic medical records (EMRs)/Interoperability review.
Study | Objective | Type of intervention/ |
eHealth intervention |
Interface/health |
Duration of follow-up |
Thompson et al 2015 [ |
To evaluate effects of health ITa in the inpatient and ICUb on mortality, LOSc,, and cost | Multiple health IT interventions on diagnosis, treatment, monitoring, cost reduction/No reference | EHRd, EMRe, CDSSf,CPOEg, Surveillance system | No reference | No reference |
Cheung et al 2015 [ |
To evaluate the effects of an information system integrated to PDMSh on organizational and clinical outcomes, in ICUi/Operating room | Integrating bedside equipment to an information system/vital signs, patient monitor, ventilator, anesthesia machine, dialysis machine, IV pump, lab values, hospital information system, admission, discharge and transfer | CDSS, PDMS, health information exchange | PDMS to an information system/no mention about direction of data exchange | 1 day to 1 week;11 months to 4 years |
de Bruin et al 2014 [ |
To evaluate recent trends in use of electronically available patient data by electronic surveillance systems for HAIsj and identify consequences for system effectiveness | HAIs that utilize EHR available in hospitals to surveillance the HAIs/Medico-administrative data procedures or discharge reports, free text reports, biochemistry, microbiology, and radiology laboratory test results, pharmacy dispensing records, radiology free-text records, vital signs, electronic discharge summary | Automated detection by HAI systems: EHR, health information exchange, using ICDk-9, ICD-10, discharge coding, ATCl code | EHR to HAI systems/no mention about direction of data exchange | No reference |
Mapp et al 2013 [ |
To examine early warning scoring systems and their effectiveness in predicting a patient's potential for deterioration and considers whether these scoring systems prevent unplanned ICU admissions and/or death | Instruments and clinical support systems available to assist health care personnel in recognizing early clinical deterioration/Vital signs, SpO2m, LOCn, UOPo, nurse/family concerns, complaints, lab values | EMR, CDSS, health information exchange based on SBARp communication | Early warning scoring systems that interface with EMRs and are supplemented with decision aides (algorithms) and clinical support systems/no mention about direction of data exchange | Seven studies: 3 to15 months/two studies: over 24 months to 8 years |
Li et al 2013 [ |
To evaluate the impact of the CHTsq on the quality of physician handoff, patient care, and physician work efficiency | Decision support/training, emergency referrals, supervision, alerts and reminders, client education, data collection, medicine dosing/Patient demographics, medications, diagnosis, problem lists, comment line, vital signs, to-do list, LOS, free daily notes, lab values | CHTs, EMR, CDSS, health information exchange. Allergy Code | Clinical information exchange using CHTs for physician handoff for hospitalized patients CHTs/mixed (no interface, unidirectional or bidirectional interface exchange) | 1 to 6 months |
Govindan et al 2010 [ |
To identify, describe, and evaluate the effectiveness of automated inpatient harm-detection methods | Automated harm detection on EMR. Gold standard: chart review | Automated detection by surveillance systems: EMR, health information exchange, using ICD-9, procedure codes, billing codes | Automated harm detection on EMR, using field-defined systems, natural language-processing/Unidirectional retrospective | No reference |
aIT: information technology.
bICU: intensive care unit.
cLOS: length of stay.
dEHR: electronic health record.
eEMR: electronic medical record.
fCDSS: computerized decision support systems.
gCPOE: computerized provider order-entry.
hPDMS: Patient data management system.
iICU: intensive care unit.
jHAIs: health care–associated infections systems.
kICD: international classification of disease.
iATC: anatomical therapeutic chemical.
mSpO2: oxygen saturation.
nLOC: level of consciousness.
oUOP: urine output.
pSBAR: situation, background, assessment, recommendation.
qCHTs: computerized physician handoff tools.
Descriptive summary of the results of systematic reviews included in electronic medical records(EMRs)/Interoperability review. Subgroup 1: electronic health (eHealth) systems implementation without health information exchange.
Study | Primary impact: |
Secondary impact: |
Main results | Potential bias | Lessons |
Thompson et al 2015 [ |
Mixed and inconclusive | Mortality: overall CPOEa systems did not show a significant effect (ORb: 0.91, 95% CI 0.75-1.10; I2c 66%), nor EHRd alone (OR: 0.96, 95% CI 0.77-1.19). CDSSe(OR 0.96, 95% CI 0.77-1.19). The surveillance systems had a pooled OR of 0.85 (95% CI 0.76-0.94) with moderate heterogeneity, I259% |
Electronic interventions were not shown to have a substantial effect on mortality, LOSf, or cost. | Selection, measurement | There is not enough evidence to confidently state that electronic interventions have the ability to achieve the goal of improving quality and safety. |
aCPOE: computerized provider order-entry.
bOR: odds ratio.
cI2: measure of heterogeneity.
dEHR: electronic health record.
eCDSS: computerized decision support systems.
fLOS: length of stay.
Descriptive summary of the results of systematic reviews included in the electronic health record (EHR)/Interoperability review. Subgroup 2: electronic health (eHealth) systems implementation with information exchange.
Study | Primary impact: |
Secondary impact: |
Main results | Potential bias | Lessons |
Cheung et al 2015 [ |
Not evaluateda | PDMSbreduced charting time, increased time spent on direct patient care and reduced the occurrence of errors (medication errors, intravenous and ventilation incidents). The effect on documentation was mixed. Improvement in clinical outcomes when PDMS was integrated with a CDSSc, but scarce literature is available. | The effect on documentation was mixed. Qualitative analysis showed a significant decrease in time spent on documentation. Clinical outcomes: inconclusive. | Selection, measurement | Improvement in clinical outcomes when PDMS was integrated with a CDSS, but there is scarce literature available. Organizational advantages included improved accuracy, legibility, data accessibility, and decision support. Such integration may improve clinical outcomes, although further studies are required for validation. |
de Bruin et al 2014 [ |
Not evaluateda | Electronic surveillance achieves equal or better sensitivity than manual surveillance. Several studies also reported time savings of 60% to 99.9% or a reduction in chart reviews of 40% to 90.5%. | Driven by the increased availability of electronic patient data, electronic HAIsdsurveillance systems use more data, making systems more sensitive yet less specific but also allow systems to be tailored to the needs of health care institutes’ surveillance programs. | Selection | HAIs detection systems use increasingly more EHReand patient data as more data sources become available. Thus, systems tend to become more sensitive and less specific. |
Mapp et al 2013 [ |
Not evaluateda | An increase occurred in the number of rapid response calls by nursing staff, a decrease in unplanned ICUfadmissions, and a decrease in hospital mortality. | Improvement in clinical outcomes when using early warning scoring systems. | Selection | Early warning scoring systems can be more effective with the integration of algorithms and clinical support systems. |
Li et al 2013 [ |
Not evaluateda | Impact on physician work efficiency (self-reported time spent on handing copying patient information; 50%) and proportionally more time to see patients. Time on each patient during rounding decreased by1.5 min. Impact on quality on physician handoff: completeness and consistency of the handoff document has improved. | Completeness and consistency of the handoff document has improved. Accuracy of information about patients during physician handoff. | Selection, measurement | CHTsgcould potentially enhance work efficiency and continuity of care during physician handoff, but the role in improving quality is less clear. The information available was often not sufficient to help on-call physicians make patient care decisions. |
Govindan et al 2010 [ |
Not evaluateda | Sensitivities of different methods ranged from 0.10 to 0.94, specificity from 0.10 to 0.94, PPVhfrom 0.03 to 0.84, and NPVifrom 0.70 to 0.96. The field-defined methods of automated harm detection will prove superior to natural language processing, particularly if information about harm is accurately documented. | Automated harm detection has the potential to positively influence clinical practice. | Selection, measurement | Automated harm detection has the potential to positively influence clinical practice. Another potential benefit is the reduction of person-hour required to harm surveillance. |
aNot evaluated in the selected study.
bPDMS: Patient data management system.
cCDSS: computerized decision support systems.
dHAIs: health care–associated infections systems.
eEHR: electronic health record.
fICU: intensive care unit.
gCHTs: computerized physician handoff tools
hPPV: positive predictive value.
iNPV: negative predictive value.
Most of the reviews use ICUs as settings for eHealth intervention analysis. However, the objectives of interventions were quite heterogeneous. Two studies reported the effect of surveillance systems on harm detection [
Among reviews focused on improving clinical practice, inconclusive results in direct patient care were reported by Cheung et al [
None of the reviews evaluated effects of eHealth interventions on costs.
This study found preliminary benefits in the use of electronically available inpatient data systems on the quality of care. Despite the limited number of studies that met the eligibility criteria, the heterogeneity of electronic systems reported, and different interventions on hospital routines, the identification of preliminary secondary benefits on patient mortality was possible [
However, no substantial review regarding the impact of electronic interventions on cost-effectiveness was identified. Among the six analyses included, only Thompson et al reported that some preliminary studies have identified decreases in cost, but the heterogeneity and the absence of information of follow-up impaired a proper analysis of cost-effectiveness [
Unfortunately, no study about interoperability, in the sense of syntactic and semantic meaning, on cost benefit was identified. Importantly, none of the studies in this review properly defined EHR concept as a longitudinal health record with entries by health care practitioners in multiple sites of care or mentioned interoperability applications among electronic systems. However, taking the antecedent step toward full interoperability, an effective information sharing between stakeholders and systems can be attained through the use of standards [
The current results should be interpreted as a whole with the study limitations. Only four major databases were searched and gray literature sources were not evaluated. Additionally, the limitation to English, Spanish, French, Italian, and Portuguese languages prevented the capture of all relevant studies. Furthermore, the quality of included studies was poor, and they varied regarding the type of eHealth interventions, follow-up time, and goals. This systematic review summarized primary and secondary outcomes from different classes of intervention from which to draw results, analysis, and conclusions. Due to the variation in scenarios and lack of numeric goals, a meta-analysis was considered inappropriate.
This review identified some benefits on the quality of care but did not provide evidence that the eHealth interventions had a measurable impact on cost-effectiveness, mortality, or LOS in hospital settings. Preliminary evidence indicates that the use of eHealth interventions with information exchange may improve clinical process outcomes. The absence of studies precludes the assessment of impact of interoperability on benefits of health care or cost, and this aspect needs further research. Technological barriers might influence eHealth solutions implementation and data exchange for systems integration or interoperable interfaces. There are also issues with the lack of standardization of most aspects of health information and misuse of terms in the scientific publications. Authors should be explicit when they are using interfacing syntactic interoperability or semantic interoperability to reduce the confusion with different health information exchange possibilities. Further research with long-term follow-up is needed to determine the actual impact of eHealth adoption on health care costs to demonstrate (1) value for money (including clinical impacts) and (2) the clinical impact of semantic and synthetic interoperability.
eHealth concepts and definitions.
Descriptive summary of the 14 full-text excluded systematic reviews.
a measurement tool to assess systematic reviews
anatomical therapeutic chemical
computerized decision support systems
European Committee for Standardization
computerized physician handoff tools
computerized provider order-entry
computer-based patient record
electronic health record
electronic medical record
health care–associated infections surveillance systems
Health level seven
measure of heterogeneity
International Classification of Disease
Intensive Care Unit
International Organization for Standardization
information technology
level of consciousness
length of stay
negative predictive value
odds ratio
Pan American Health Organization
patient data management system
positive predictive value
preferred reporting items for systematic reviews and meta-analyses
oxygen saturation
situation, background, assessment, recommendation
urine output
World Health Organization
We thank the team who worked with us on a systematic review with a large scope: Maria Beatriz Moreira Alkmim, Renato Minelli Figueira, João Antonio de Queiroz Oliveira, Gustavo Moreira Alkmim, Maria Clara Noman de Alencar, Nayara Dornela Quintino, and the undergratuate students—Edson Alexandre Silva Carvalho, Fernanda Cotrim Stefanelli, Gabriel Almeida Silqueira Rocha, Grace Kelly Matos e Silva, Marcelo Henriques de Camargos, Thales Matheus Mendonça Santos, Thiago Adriano de Deus Queiroz Santos, and Elsy Dumit Bechara.
Francisco Becerra-Posada and David Novillo-Ortiz are staff members of the PAHO/WHO and are themselves alone responsible for the views expressed in the paper, which does not necessarily represent the views, decisions, or policies of the PAHO/WHO.
None declared.